Skip to content

Commit 35dd23e

Browse files
author
Alexander Cherednikov
committed
Реализация RAG
1 parent 4d5fd9d commit 35dd23e

55 files changed

Lines changed: 10255 additions & 1347 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [main, master]
6+
pull_request:
7+
branches: [main, master]
8+
9+
jobs:
10+
test:
11+
runs-on: ubuntu-latest
12+
13+
strategy:
14+
matrix:
15+
php-version: ['8.2', '8.3']
16+
17+
services:
18+
qdrant:
19+
image: qdrant/qdrant:v1.11.0
20+
ports:
21+
- 6333:6333
22+
- 6334:6334
23+
options: --health-cmd="curl --fail http://localhost:6333/health" --health-interval=10s --health-timeout=5s --health-retries=3
24+
25+
steps:
26+
- name: Checkout code
27+
uses: actions/checkout@v4
28+
29+
- name: Setup PHP
30+
uses: shivammathur/setup-php@v2
31+
with:
32+
php-version: ${{ matrix.php-version }}
33+
extensions: mbstring, zip, curl
34+
coverage: xdebug
35+
36+
- name: Cache Composer dependencies
37+
uses: actions/cache@v4
38+
with:
39+
path: ~/.composer/cache
40+
key: php-${{ matrix.php-version }}-composer-${{ hashFiles('**/composer.json') }}
41+
restore-keys: |
42+
php-${{ matrix.php-version }}-composer-
43+
44+
- name: Install dependencies
45+
run: composer install --prefer-dist --no-progress --no-interaction
46+
47+
- name: Run PHPStan
48+
run: composer phpstan
49+
50+
- name: Check code style
51+
run: composer cs-check
52+
53+
- name: Test Qdrant connection
54+
run: |
55+
timeout 30 bash -c 'until curl --silent --fail http://localhost:6333/health; do sleep 1; done'
56+
echo "Qdrant is ready"
57+
58+
docker:
59+
runs-on: ubuntu-latest
60+
needs: test
61+
62+
steps:
63+
- name: Checkout code
64+
uses: actions/checkout@v4
65+
66+
- name: Build Docker image
67+
run: docker build -t rag-demo .
68+
69+
- name: Test Docker Compose setup
70+
run: |
71+
docker-compose up -d
72+
sleep 20
73+
curl --fail http://localhost:6333/health
74+
docker-compose logs
75+
docker-compose down

‎.gitignore‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,25 @@
88
/var/
99
/vendor/
1010
###< symfony/framework-bundle ###
11+
12+
# IDE files
13+
.idea/
14+
.vscode/
15+
*.swp
16+
*.swo
17+
18+
# Cache and temporary files
19+
.claude/
20+
.transformers-cache/
21+
.php-cs-fixer.cache
22+
23+
# OS files
24+
.DS_Store
25+
Thumbs.db
26+
27+
# Docker volumes (keep in git, but ignore if local overrides exist)
28+
# docker-compose.override.yml
29+
30+
###> friendsofphp/php-cs-fixer ###
31+
/.php-cs-fixer.cache
32+
###< friendsofphp/php-cs-fixer ###

‎.php-cs-fixer.php‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
<?php
2+
3+
$finder = PhpCsFixer\Finder::create()
4+
->in(__DIR__.'/src')
5+
->name('*.php');
6+
7+
$config = new PhpCsFixer\Config();
8+
9+
return $config->setRules([
10+
'@PSR12' => true,
11+
'@Symfony' => true,
12+
'array_syntax' => ['syntax' => 'short'],
13+
'ordered_imports' => true,
14+
'no_unused_imports' => true,
15+
])
16+
->setFinder($finder);

‎CHANGELOG.md‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Changelog
2+
3+
## [2.0.0] - 2024-09-03 - GitHub Ready Release 🚀
4+
5+
### ✨ Major Improvements
6+
- **Complete project restructuring** for GitHub publication
7+
- **Production-ready Docker setup** with Ollama + Qdrant
8+
- **Static analysis integration** with PHPStan + PHP CS Fixer
9+
- **CI/CD pipeline** with GitHub Actions
10+
- **One-command setup** via Makefile + setup script
11+
12+
### 🧹 Code Quality
13+
- **Clean architecture** with proper interfaces and separation of concerns
14+
- **PSR-12 + Symfony code standards** compliance
15+
- **Type safety** improvements across all services
16+
- **Documentation** updates with comprehensive README
17+
18+
### 🏗️ Infrastructure
19+
- **Docker Compose** setup with networking and persistent volumes
20+
- **Qdrant configuration** optimization for demo performance
21+
- **Automated setup scripts** for seamless deployment
22+
- **Health checks** and monitoring for all services
23+
24+
### 📦 New Features
25+
- **Makefile** with convenient development commands
26+
- **GitHub Actions** CI/CD pipeline
27+
- **MIT License** for open source distribution
28+
- **Comprehensive README** with examples and documentation
29+
30+
### 🗑️ Removed
31+
- Temporary presentation files and demo artifacts
32+
- Redundant test commands and development leftovers
33+
- IDE-specific configuration files
34+
- Unused dependencies and cache files
35+
36+
### 🛠️ Development Experience
37+
- **Static analysis** with level 8 PHPStan
38+
- **Code formatting** with PHP CS Fixer
39+
- **Git hooks** and CI/CD integration
40+
- **Docker development** environment
41+
42+
---
43+
44+
This version transforms the project from a presentation demo into a production-ready,
45+
open-source RAG implementation example that developers can fork and build upon.

‎CLAUDE.md‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
# RAG Vector Search with AI - Presentation Demo
2+
3+
This is a comprehensive demonstration of a modern RAG (Retrieval-Augmented Generation) system built with PHP, showcasing intelligent product search using vector embeddings and local AI models.
4+
5+
## System Overview
6+
7+
**Technologies:**
8+
- PHP 8.3 + Symfony 7.3 (MicroKernelTrait)
9+
- Qdrant Vector Database
10+
- Transformers PHP (local embeddings)
11+
- Ollama + Llama 3.2 (local AI model)
12+
- Docker for services
13+
14+
**Key Features:**
15+
- 🔍 **Semantic Search** - Vector similarity search in Russian/English
16+
- 🤖 **AI Query Analysis** - Natural language understanding with Llama 3.2
17+
- 💬 **Interactive Chat** - Conversational product discovery
18+
- 🚀 **Memory Optimized** - Generator-based processing for large datasets
19+
- 💰 **Cost Efficient** - 100% local, no API costs
20+
21+
## Demo Commands
22+
23+
### Basic Setup
24+
```bash
25+
docker-compose up -d # Start Qdrant
26+
ollama pull llama3.2:1b # Download AI model
27+
php bin/console products:vectorize # Index products (one time)
28+
```
29+
30+
### Search Demonstrations
31+
```bash
32+
# Traditional search (English only)
33+
php bin/console products:search "AMD processor"
34+
35+
# Russian search with translation
36+
php bin/console products:search-ru "процессор AMD"
37+
38+
# AI-powered natural language search
39+
php bin/console products:search-ai "ищу мощный процессор для игр"
40+
41+
# Interactive AI chat (main demo!)
42+
php bin/console products:chat
43+
44+
# 🆕 NEW! Improved RAG Architecture Demo
45+
php bin/console rag:demo --query "процессор AMD для игр" # Single query with detailed steps
46+
php bin/console rag:demo --interactive # Interactive RAG chat
47+
```
48+
49+
## Architecture Highlights
50+
51+
### Traditional Pipeline
52+
1. **Vector Embeddings**: Product descriptions → 384-dim vectors via all-MiniLM-L6-v2
53+
2. **AI Query Processing**: Russian queries → optimized English search terms via Llama 3.2
54+
3. **Semantic Retrieval**: Cosine similarity search in Qdrant with configurable thresholds
55+
4. **AI Response Generation**: Personalized recommendations based on search results
56+
57+
### 🆕 Improved RAG Architecture (Recommended by Curator)
58+
**Three Clear Stages:**
59+
60+
1. **📝 Query Processing (Авторизация запроса)**
61+
- Natural language analysis with Llama 3.2
62+
- Query optimization for better search results
63+
- Vector embedding generation (384-dimensional)
64+
65+
2. **🔍 Retrieval (Поиск в векторной БД)**
66+
- Semantic search in Qdrant vector database
67+
- Cosine similarity matching with relevance threshold
68+
- Context preparation for LLM
69+
70+
3. **✨ Generation (Ограниченная генерация)**
71+
- LLM works ONLY with retrieved documents
72+
- No "hallucination" - strictly based on found products
73+
- Constrained response generation with improved prompts
74+
75+
**Key Improvements**:
76+
- LLM never invents products or adds information "from itself" - it only works with the actual search results from the vector database
77+
- **Critical**: All vectorization (both indexing and search) uses identical `Task::Embeddings` with `all-MiniLM-L6-v2` model for perfect semantic consistency
78+
- Proper embedding consistency ensures accurate similarity matching between queries and stored products
79+
80+
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
81+
82+
## Development Commands
83+
84+
### Dependency Management
85+
- `composer install` - Install PHP dependencies
86+
- `composer update` - Update dependencies
87+
88+
### Symfony Console
89+
- `php bin/console` - Access Symfony console commands
90+
- `php bin/console cache:clear` - Clear application cache
91+
- `php bin/console debug:router` - Show all routes
92+
- `php bin/console debug:container` - Debug service container
93+
94+
### Development Server
95+
- `symfony serve` or `php -S localhost:8000 -t public/` - Start development server
96+
97+
### Docker Services
98+
- `docker-compose up -d` - Start Qdrant vector database
99+
- `docker-compose down` - Stop all services
100+
- `docker-compose logs qdrant` - View Qdrant logs
101+
102+
### Qdrant Vector Database
103+
- HTTP API: `http://localhost:6333`
104+
- gRPC API: `localhost:6334`
105+
- Web UI: `http://localhost:6333/dashboard`
106+
107+
## Architecture Overview
108+
109+
This is a Symfony 7.3 application using the MicroKernelTrait for minimal setup.
110+
111+
### Key Components
112+
- **Kernel**: Uses `MicroKernelTrait` for streamlined configuration
113+
- **Controllers**: Located in `src/Controller/` with attribute-based routing
114+
- **Services**: Auto-configured in `config/services.yaml` with autowiring enabled
115+
- **Configuration**: YAML-based configuration in `config/` directory
116+
117+
### Directory Structure
118+
- `src/` - Application source code (PSR-4 autoloaded as `App\` namespace)
119+
- `config/` - Application configuration files
120+
- `public/` - Web root with front controller (`index.php`)
121+
- `bin/console` - Symfony console entry point
122+
- `var/cache/` - Application cache files
123+
- `vendor/` - Composer dependencies
124+
125+
### Configuration Notes
126+
- Services use autowiring and autoconfiguration by default
127+
- Controllers are automatically registered via attribute routing
128+
- Framework configuration is split across files in `config/packages/`
129+
- Routes are configured in `config/routes.yaml` and via controller attributes

‎Dockerfile‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
FROM php:8.3-cli-alpine
2+
3+
# Install system dependencies
4+
RUN apk add --no-cache \
5+
curl \
6+
zip \
7+
unzip \
8+
git \
9+
oniguruma-dev \
10+
libzip-dev \
11+
&& docker-php-ext-install \
12+
zip \
13+
mbstring
14+
15+
# Install Composer
16+
RUN curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
17+
18+
# Set working directory
19+
WORKDIR /app
20+
21+
# Copy composer files first for better caching
22+
COPY composer.json composer.lock ./
23+
24+
# Install dependencies
25+
RUN composer install --no-dev --optimize-autoloader --no-scripts
26+
27+
# Copy application code
28+
COPY . .
29+
30+
# Create cache directory
31+
RUN mkdir -p var/cache && chmod -R 775 var/
32+
33+
# Install Transformers models
34+
RUN php bin/console transformers:download
35+
36+
EXPOSE 8000
37+
38+
CMD ["php", "-S", "0.0.0.0:8000", "-t", "public/"]

‎LICENSE‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2024 RAG Vector Search Demo
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

0 commit comments

Comments
 (0)