diff --git a/GEMINI_API.md b/GEMINI_API.md new file mode 100644 index 000000000..73cc41f84 --- /dev/null +++ b/GEMINI_API.md @@ -0,0 +1,107 @@ +# Gemini API Adapter Specifications + +This document outlines the plan to build an adapter to consume the Google Gemini API, achieving feature parity with the existing Anthropic API integration. + +## Goals + +1. **Generate Text:** Support text generation with Gemini models. [DONE] +2. **Multimodal:** Support file uploads (images, PDFs) and usage in requests. [DONE] +3. **Tool Calling:** Support function calling (tools). [DONE] +4. **Streaming:** Support streaming responses. [DONE] + +## Architecture + +The integration will follow the existing pattern used for Anthropic: +- **Namespace:** `Gemini` module in `lib/gemini.rb` and `lib/gemini/`. [DONE] +- **Client:** `Gemini::Client` to handle HTTP requests. [DONE] +- **Request Object:** `Gemini::InvokeModelRequest` to format the payload. [DONE] +- **Response Object:** `Gemini::InvokeModelResponse` and `Gemini::StreamResponse` to normalize outputs. [DONE] +- **Turns:** `Gemini::Turn` to format conversation history. [DONE] +- **Files:** `Gemini::FilesClient` for the Files API. [DONE] + +## Development Phases + +### Phase 1: Foundation & Text Generation + +**Goal:** successfully generate text from a single prompt using a Gemini model. + +- [x] Create `lib/gemini.rb` and `lib/gemini/client.rb`. +- [x] Implement `Gemini::Client#initialize` using `GEMINI_API_KEY`. +- [x] Define Gemini models in `lib/gemini.rb` (e.g., `gemini-3-flash-preview`, `gemini-2.5-pro`). +- [x] Update `GenerativeText::MODELS` to include Gemini models (vendor: `:google`). +- [x] Update `GenerativeText.client_for` to handle `:google` vendor. +- [x] Create `lib/gemini/invoke_model_request.rb` to format basic text prompts. +- [x] Create `lib/gemini/invoke_model_response.rb` to wrap the response. +- [x] Implement `Gemini::Client#invoke_model`. +- [x] Verify basic text generation in Rails console. + +### Phase 2: Multi-turn Conversations (Chat) + +**Goal:** Support conversation history. + +- [x] Create `lib/gemini/turn.rb`. +- [x] Implement `Gemini::Turn.for(request, turns:)` to format `GenerateTextRequest` and history into Gemini's `contents` format (`role`, `parts`). +- [x] Update `GenerateTextRequest#to_turn` to handle `:google` vendor. +- [x] Update `Gemini::InvokeModelRequest` to accept and format the full conversation history. +- [x] Verify multi-turn chat in Rails console. + +### Phase 3: Streaming + +**Goal:** Support real-time response streaming. + +- [x] Create `lib/gemini/stream_event.rb` to parse SSE chunks. +- [x] Create `lib/gemini/stream_response.rb` to aggregate chunks. +- [x] Implement `Gemini::Client#invoke_model_stream`. +- [x] Verify streaming works in the UI. + +### Phase 4: Multimodal & Files + +**Goal:** Support attaching images and PDFs to prompts. + +- [x] Create `lib/gemini/files_client.rb` to wrap Gemini Files API (`upload`, `get`, `delete`). +- [x] Add `upload_file` and `delete_file` methods to `lib/gemini.rb`. +- [x] Update `Gemini::Turn` to include file parts in the content. +- [x] Verify image/PDF analysis. + +### Phase 5: Tool Calling + +**Goal:** Support defining and invoking tools. + +- [x] Update `Gemini::InvokeModelRequest` to map `LlmTool` definitions to Gemini's `tools` -> `function_declarations` format. +- [x] Handle tool use responses in `Gemini::InvokeModelResponse`. +- [x] Verify the model can call tools (e.g., getting the weather, or whatever tools are defined). + +### Phase 6: Polish & Testing + +**Goal:** Ensure code quality and stability. + +- [x] Add RSpec tests for `Gemini::Client`. +- [x] Add RSpec tests for `Gemini::Turn` and `Gemini::InvokeModelRequest`. +- [x] Add VCR cassettes for API interactions (using WebMock stubs in this implementation). +- [x] Ensure error handling (map Gemini errors to `Gemini::ClientError` equivalents). + +## Reference: Data Structures + +**Gemini Content Format:** +```json +{ + "role": "user", + "parts": [ + { "text": "Hello" }, + { "file_data": { "mime_type": "...", "file_uri": "..." } } + ] +} +``` + +**Gemini Tools Format:** +```json +{ + "function_declarations": [ + { + "name": "get_weather", + "description": "...", + "parameters": { ... } + } + ] +} +``` \ No newline at end of file diff --git a/GEMINI_API_INTEGRATION.md b/GEMINI_API_INTEGRATION.md new file mode 100644 index 000000000..9f60de55e --- /dev/null +++ b/GEMINI_API_INTEGRATION.md @@ -0,0 +1,91 @@ +# Gemini API Integration Specifications + +This document outlines the plan to integrate the Gemini API adapter into the existing Rails application, ensuring seamless user interaction for model selection, text generation, and file uploads. + +## Analysis + +### Current Architecture +1. **Model Selection:** + - **UI:** `PromptFormComponent` uses `GenerativeText.active_models` to populate the model dropdown. Since Gemini models are now registered in `GenerativeText::MODELS`, they automatically appear in the UI. + - **Frontend:** `prompt_form_controller.js` handles file input toggling based on model capabilities (already supported via `model_data` serialization). + - **Settings:** User settings for default models (`User#setting.text_model`) are string-based and agnostic to the vendor. + +2. **Request Handling:** + - **Controller:** `ConversationsController` uses `ConversationForm` to process requests. + - **Form:** `ConversationForm` creates a `GenerateTextRequest` with the selected model. + - **Job:** `GenerateTextJob` executes the request in the background. It calls `GenerativeText.new.invoke_model(request)`. + - **Service:** `GenerativeText` delegates to the appropriate client (Anthropic or Gemini) based on the model's vendor. + +3. **File Uploads:** + - **Controller:** `ConversationContextsConversationsController` handles file uploads. Logic has been updated to use `Gemini.upload_file` when the user's preferred model is a Gemini model. + - **Context:** `ConversationContext` stores the file reference (URI for Gemini, ID for Anthropic) and mime type. + +4. **Response Handling:** + - **Job:** `GenerateTextJob` expects the response object to respond to `.data` (for storage) and `.content` (for broadcasting). + - **View:** `GenerateTextRequestComponent` renders the response content. + +### Identified Gaps +1. **Missing `data` Method:** `Gemini::InvokeModelResponse` does not expose the raw response data via a `data` method, which is required by `GenerateTextJob` to save the raw response to the database. + +## Development Phases + +### Phase 1: Fix Response Interface + +**Goal:** Ensure `Gemini::InvokeModelResponse` adheres to the interface expected by `GenerateTextJob`. + +- [x] Update `Gemini::InvokeModelResponse` to expose `@response_json` via a `data` method/attribute. +- [x] Add a spec to verify `data` returns the raw hash. + +### Phase 2: User Interface Polish (Optional) + +**Goal:** Improve visual distinction between models. + +- [x] (Optional) Update `ConversationTurnComponent` or CSS to display vendor-specific icons (e.g., Google logo for Gemini) if desired. Currently, it uses a generic robot icon. + +### Phase 3: End-to-End Verification + +**Goal:** Verify the full flow from UI to Database. + +- [x] Verify `GenerateTextJob` runs successfully with a Gemini model. +- [x] Verify `GenerateTextRequest` saves the raw Gemini JSON response in the `response` column. +- [x] Verify streaming works in the browser (simulated via system tests). + +## TODOs + +- [x] Fix `Gemini::InvokeModelResponse#data`. +- [x] Verify `GenerateTextJob` with Gemini model via console/test. + +### Phase 4: Provider-Aware Conversation Contexts + +**Goal:** Make `ConversationContext` explicitly aware of its provider to ensure only compatible contexts are used. + +- [x] Add `vendor` column to `ConversationContext` table. +- [x] Update `ConversationContextsConversationsController#create` to determine and save the `vendor` when uploading a new file. +- [x] Update `InvokeModelRequest` for both `Anthropic` and `Gemini` to filter contexts based on the active model's vendor. + +### Phase 5: Dynamic Context UI + +**Goal:** Update the "Attach File" UI to dynamically show contexts that are compatible with the selected model. + +- [x] Add a new route/action to fetch available `ConversationContext` records filtered by `vendor`. +- [x] Update `prompt_form_controller.js` to fetch and render the filtered context list when the model selection changes. +- [x] Create a Turbo Stream view to render the updated context list. +- [x] Add vendor badges to the context selection UI. +- [x] Allow uploading `.md` (markdown) files. + +### Phase 6: Update Gemini Models + +**Goal:** Update the Gemini model list to the latest models. + +- [x] Update `lib/gemini.rb` with the latest model names and ensure they are marked as active. +- [x] Set max tokens to 65,536 for all models. + + +### Phase 7: Conversation Contexts + +- [x] In app/views/conversation_contexts_conversations/index.html.haml, the @available_contexts should be scoped to the model selected in the prompt form component. When selecting an Anthropic model, the available contexts should only be Anthropic file uploads. When selecting a Google model, the available contexts should only be Google file uploads. These needs to happen dynamically. +- [x] In app/views/conversation_contexts_conversations/index.html.haml show a vendor badge, Anthropic or Google. This needs to change dynamically when the user selects a model in the prompt form component. Also show a vendor badge next to each of the selected conversation contexts. Show them in a disabled state when the currently selected model vendor is different from the context's vendor. +- [x] In app/views/conversation_contexts/_conversation_context.html.haml show a vendor badge next to each conversation context. +- [x] Create a scheduled sidekiq job the deletes Google file uploads / conversation contexts, 48 hours after they are uploaded / created. +- [x] When creating a conversation_conversation_context record, the front end should pass the vendor. Currently the vendor is inferred from the user's settings which may not match the model selected in prompt form component. + diff --git a/TODO.org b/TODO.org index 318212e62..f2bd1168f 100644 --- a/TODO.org +++ b/TODO.org @@ -20,11 +20,17 @@ CLOSED: [2025-09-15 Mon 08:19] * TODO Make it optional to include generated images in the conversation context Right now it's decided by the backend to include the previous one automatically. This isn't always -- or usually -- what I want + +* TODO Fix Markdown tables in assistant response +- [ ] When converting markdown to HTML, the tables to not appear properly (eg, not + in table format). Find a way to render tables properly * TODO Gemini integration - - [ ] Refactor existing claude integration to adapter - - [ ] Add Gemini interface - - [ ] Add Gemini models selectable in UI + - [X] Refactor existing claude integration to adapter + - [X] Add Gemini interface + - [X] Add Gemini models selectable in UI - [ ] Handle context documents + - [ ] Pass vendor when creating conversation contexts. Right now the vendor is + inferred from the user settings default model. * TODO Enhanced copy response - [ ] Option to copy markdown formatted response * DONE Add support for PDFs diff --git a/app/controllers/conversation_contexts_conversations_controller.rb b/app/controllers/conversation_contexts_conversations_controller.rb index e4967c491..f617e4cdf 100644 --- a/app/controllers/conversation_contexts_conversations_controller.rb +++ b/app/controllers/conversation_contexts_conversations_controller.rb @@ -17,8 +17,15 @@ def create end if conversation_context_params[:file].present? - file_response = Anthropic.upload_file(conversation_context_params[:file]) - contexts << ConversationContext.create_for!(current_user, file_response) + vendor = conversation_context_params[:vendor] || current_vendor + + file_response = if vendor.to_sym == :google + Gemini.upload_file(conversation_context_params[:file]) + else + Anthropic.upload_file(conversation_context_params[:file]) + end + + contexts << ConversationContext.create_for!(current_user, file_response, vendor: vendor) end contexts.select(&:persisted?).each do |context| @@ -51,12 +58,35 @@ def create def index @contexts = @conversation.conversation_contexts.order(created_at: :desc) + @vendor = current_vendor + @available_contexts = available_contexts respond_to do |format| format.html format.json { render json: @contexts, status: :ok } end end + def available + @vendor = params[:vendor] + @available_contexts = available_contexts + + respond_to do |format| + format.turbo_stream do + render turbo_stream: [ + turbo_stream.update( + 'context-files-selector', + partial: 'context_files_selector', + locals: { conversation: @conversation, available_contexts: @available_contexts, current_vendor: @vendor } + ), + turbo_stream.update( + 'conversationContextModalLabel', + "Conversation Context #{@vendor.to_s.titleize}".html_safe + ) + ] + end + end + end + def destroy @context = @conversation.conversation_contexts.find(params[:id]) @@ -100,15 +130,26 @@ def set_conversation @conversation = current_user.conversations.find(params[:conversation_id]) end + def default_vendor + current_user.setting&.text_model&.yield_self do |m| + GenerativeText::MODELS.find { |mod| mod.api_name == m } + end&.vendor + end + def set_available_contexts @available_contexts = available_contexts end def available_contexts - current_user.conversation_contexts.available_for(@conversation).order(:filename) + current_user.conversation_contexts.available_for(@conversation).order(created_at: :desc) + end + + def current_vendor + vendor = params[:vendor] || @conversation.turns.last&.turnable&.model&.vendor || default_vendor || :anthropic + vendor.to_sym end def conversation_context_params - params.require(:conversation_context).permit(:file, conversation_context_ids: []) + params.require(:conversation_context).permit(:file, :vendor, conversation_context_ids: []) end end diff --git a/app/javascript/controllers/conversation_context_controller.js b/app/javascript/controllers/conversation_context_controller.js index 7893b3912..30d16d38d 100644 --- a/app/javascript/controllers/conversation_context_controller.js +++ b/app/javascript/controllers/conversation_context_controller.js @@ -3,7 +3,7 @@ import { createConversationContext } from '@javascript/http'; export default class ConversationContextController extends Controller { static targets = [ - "dropZone", "fileInput", "successAlert", "errorAlert", "successMessage", "errorMessage", "spinner" + "dropZone", "fileInput", "successAlert", "errorAlert", "successMessage", "errorMessage", "spinner", "item" ] abortController = null; @@ -11,12 +11,55 @@ export default class ConversationContextController extends Controller { connect() { this.conversationId = this.element.dataset.conversationId + this.boundOnModelChanged = this.onModelChanged.bind(this) + document.addEventListener('prompt-form:model-changed', this.boundOnModelChanged) + + this.updateItems() + + // Slight delay to ensure other controllers (like the selector) are connected + setTimeout(() => { + this.dispatch('opened', { detail: { vendor: this.currentVendor } }) + }, 100) } disconnect() { if (this.abortController) { this.abortController.abort() } + document.removeEventListener('prompt-form:model-changed', this.boundOnModelChanged) + } + + get currentVendor() { + if (this.element.dataset.vendor) return this.element.dataset.vendor + + const modelSelect = document.querySelector('[data-prompt-form-target="modelSelect"]') + if (!modelSelect) return 'anthropic' + const modelData = JSON.parse(modelSelect.dataset.modelData) + const selectedModel = modelData.find(m => m.api_name === modelSelect.value) + return selectedModel ? selectedModel.vendor : 'anthropic' + } + + onModelChanged(event) { + this.element.dataset.vendor = event.detail.vendor + this.updateItems() + // The frame reload handled in PromptForm will replace the modal content, + // but we can also trigger immediate updates if needed. + } + + updateItems() { + const vendor = this.currentVendor + this.itemTargets.forEach(item => { + const itemVendor = item.dataset.vendor + if (itemVendor && itemVendor !== vendor) { + item.classList.add('disabled-context') + item.style.opacity = '0.5' + item.style.pointerEvents = 'none' + } else { + item.classList.remove('disabled-context') + item.style.opacity = '1' + item.style.pointerEvents = 'auto' + } + }) } openFileDialog() { @@ -74,6 +117,7 @@ export default class ConversationContextController extends Controller { const response = await createConversationContext( this.conversationId, file, + this.currentVendor, this.abortController.signal ) diff --git a/app/javascript/controllers/conversation_context_selector_controller.js b/app/javascript/controllers/conversation_context_selector_controller.js index 1b710f5f0..829e0098c 100644 --- a/app/javascript/controllers/conversation_context_selector_controller.js +++ b/app/javascript/controllers/conversation_context_selector_controller.js @@ -5,10 +5,48 @@ export default class ConversationContextSelector extends Controller { connect() { this.toggleSubmit() + this.boundOnModelChanged = this.onModelChanged.bind(this) + this.boundOnContextModalOpened = this.onContextModalOpened.bind(this) + + document.addEventListener('prompt-form:model-changed', this.boundOnModelChanged) + document.addEventListener('conversation-context:opened', this.boundOnContextModalOpened) + + this.disableIncompatibleOptions() + } + + disconnect() { + document.removeEventListener('prompt-form:model-changed', this.boundOnModelChanged) + document.removeEventListener('conversation-context:opened', this.boundOnContextModalOpened) } toggleSubmit() { - const selected = this.selectTarget.selectedOptions + const selected = Array.from(this.selectTarget.selectedOptions) this.submitButtonTarget.disabled = !selected.length } + + onModelChanged(event) { + this.selectTarget.dataset.currentVendor = event.detail.vendor + this.disableIncompatibleOptions() + } + + onContextModalOpened(event) { + this.selectTarget.dataset.currentVendor = event.detail.vendor + this.disableIncompatibleOptions() + } + + disableIncompatibleOptions() { + const currentVendor = this.selectTarget.dataset.currentVendor + if (!currentVendor) return + + Array.from(this.selectTarget.options).forEach(option => { + const optionVendor = option.dataset.vendor + if (optionVendor && optionVendor !== currentVendor) { + option.disabled = true + option.selected = false // Deselect if it was selected and now incompatible + } else { + option.disabled = false + } + }) + this.toggleSubmit() + } } diff --git a/app/javascript/http.js b/app/javascript/http.js index a46118e76..6ec03bbdd 100644 --- a/app/javascript/http.js +++ b/app/javascript/http.js @@ -140,9 +140,12 @@ export const generateImage = ({ prompt, negative_prompt, image_name, style, aspe }) } -export const createConversationContext = (conversation_id, file, signal) => { +export const createConversationContext = (conversation_id, file, vendor, signal) => { const formData = new FormData() formData.append('conversation_context[file]', file) + if (vendor) { + formData.append('conversation_context[vendor]', vendor) + } return fetch(`/conversations/${conversation_id}/conversation_contexts_conversations`, { method: 'POST', diff --git a/app/models/conversation_context.rb b/app/models/conversation_context.rb index 959eca06c..6d8e063bd 100644 --- a/app/models/conversation_context.rb +++ b/app/models/conversation_context.rb @@ -6,7 +6,7 @@ class ConversationContext < ApplicationRecord validates :file_ref, :filename, presence: true - after_destroy_commit -> { DeleteRemoteConversationContextJob.perform_async(file_ref) } + after_destroy_commit -> { DeleteRemoteConversationContextJob.perform_async(file_ref, vendor) } # rubocop:disable Layout/LineLength enum :mime_type, { @@ -31,11 +31,16 @@ class ConversationContext < ApplicationRecord file: 'file' }, validate: true + enum :vendor, { + anthropic: 'anthropic', + google: 'google' + }, validate: true + DOCUMENT_CONTENT_TYPE = 'document'.freeze IMAGE_CONTENT_TYPE = 'image'.freeze DEFAULT_CONTENT_BLOCK_TYPE = 'container_upload'.freeze CONTENT_BLOCK_TYPES = { - ['application/pdf', 'text/plain'] => DOCUMENT_CONTENT_TYPE, + ['application/pdf', 'text/plain', 'text/markdown'] => DOCUMENT_CONTENT_TYPE, ['image/jpeg', 'image/png', 'image/gif', 'image/webp'] => IMAGE_CONTENT_TYPE }.freeze @@ -49,23 +54,25 @@ class ConversationContext < ApplicationRecord } # @param user [User] - # @param file_response [Anthropic::FileResponse] + # @param file_response [Anthropic::FileResponse | Gemini::FileResponse] + # @param vendor [Symbol] # @return [ConversationContext] - def self.create_for!(user, file_response) + def self.create_for!(user, file_response, vendor: :anthropic) context = create( file_ref: file_response.id, filename: file_response.filename, mime_type: file_response.mime_type, context_type: context_types['file'], + vendor:, user_id: user.id ) unless context.persisted? - DeleteRemoteConversationContextJob.perform_async(file_response.id) + clean_up_remote_context(vendor, file_response) end context - rescue StandardError - DeleteRemoteConversationContextJob.perform_async(file_response.id) - raise CreateError, "Failed to create conversation context for file: #{file_response.id}" + rescue StandardError => e + clean_up_remote_context(vendor, file_response) + raise CreateError, "Failed to create conversation context for file: #{file_response.id}. Error: #{e.message}" end def to_content_block @@ -91,4 +98,9 @@ def metadata {} end end + + + def self.clean_up_remote_context(vendor, file_response) + DeleteRemoteConversationContextJob.perform_async(file_response.id, vendor) + end end diff --git a/app/models/generate_text_request.rb b/app/models/generate_text_request.rb index dd0f6def0..cb83a1b76 100644 --- a/app/models/generate_text_request.rb +++ b/app/models/generate_text_request.rb @@ -5,7 +5,7 @@ class GenerateTextRequest < ApplicationRecord include Turnable TEMPERATURE_VALUES = 0.step(to: 1, by: 0.1).map { _1.round(1) } - MAX_PROMPT_LENGTH = 150_000 + MAX_PROMPT_LENGTH = 200_000 SUPPORTED_MIME_TYPES = %w[image/jpeg image/gif image/png image/webp].freeze MAX_FILE_SIZE = 4.megabytes @@ -81,6 +81,8 @@ def to_turn(turns: []) case model.vendor when :anthropic Anthropic::Turn.for(self, turns:) + when :google + Gemini::Turn.for(self, turns:) when :aws GenerativeText::AWS::Turn.for(self) end @@ -124,6 +126,8 @@ def response_wrapper_class case model.vendor when :anthropic Anthropic::InvokeModelResponse + when :google + Gemini::InvokeModelResponse when :aws GenerativeText::AWS::InvokeModelResponse end diff --git a/app/sidekiq/cleanup_google_conversation_contexts_job.rb b/app/sidekiq/cleanup_google_conversation_contexts_job.rb new file mode 100644 index 000000000..53dedb2c3 --- /dev/null +++ b/app/sidekiq/cleanup_google_conversation_contexts_job.rb @@ -0,0 +1,9 @@ +class CleanupGoogleConversationContextsJob + include Sidekiq::Job + + def perform + ConversationContext.google.where('created_at < ?', 48.hours.ago).find_each do |context| + context.destroy + end + end +end diff --git a/app/sidekiq/delete_remote_conversation_context_job.rb b/app/sidekiq/delete_remote_conversation_context_job.rb index 272f9a238..8a189dd2c 100644 --- a/app/sidekiq/delete_remote_conversation_context_job.rb +++ b/app/sidekiq/delete_remote_conversation_context_job.rb @@ -1,7 +1,16 @@ class DeleteRemoteConversationContextJob include Sidekiq::Job - def perform(file_id) - Anthropic.delete_file(file_id) + def perform(file_id, vendor) + case vendor.to_sym + when :anthropic + Anthropic.delete_file(file_id) + when :google + Gemini.delete_file(file_id) + else + Rails.logger.warn("#{self.class}: Unknown vendor #{vendor}") + end + rescue StandardError => e + Rails.logger.error("#{self.class}: Failed to delete remote file #{file_id} for vendor #{vendor}: #{e.message}") end end diff --git a/app/views/components/generate_text_request_component.rb b/app/views/components/generate_text_request_component.rb index 79830a25f..78ef02588 100644 --- a/app/views/components/generate_text_request_component.rb +++ b/app/views/components/generate_text_request_component.rb @@ -65,6 +65,15 @@ def image_variant_options } end + def bot_icon + case model.vendor + when :google + 'bi-google' + else + 'bi-robot' + end + end + def readonly? @readonly end diff --git a/app/views/components/generate_text_request_component/generate_text_request_component.html.haml b/app/views/components/generate_text_request_component/generate_text_request_component.html.haml index c4830e543..de0d26523 100644 --- a/app/views/components/generate_text_request_component/generate_text_request_component.html.haml +++ b/app/views/components/generate_text_request_component/generate_text_request_component.html.haml @@ -18,7 +18,7 @@ .g-col-12 .card.segment-assistant .turn-icon.ps-3.pt-1 - %i.bi-robot + %i{ class: bot_icon } .card-body.pt-1{ id: assistant_response_id } - if created? || in_progress? .text-center diff --git a/app/views/components/prompt_form_component.rb b/app/views/components/prompt_form_component.rb index 0f1374ddf..80514ec95 100644 --- a/app/views/components/prompt_form_component.rb +++ b/app/views/components/prompt_form_component.rb @@ -51,7 +51,7 @@ def model_options end def model_data - GenerativeText.active_models.to_json(only: [:api_name, :capabilities, :image?]) + GenerativeText.active_models.to_json(only: [:api_name, :capabilities, :image?, :vendor]) end def after_create_preset_redirect_path diff --git a/app/views/components/prompt_form_component/prompt_form_controller.js b/app/views/components/prompt_form_component/prompt_form_controller.js index 29ab2b048..ffeebeaca 100644 --- a/app/views/components/prompt_form_component/prompt_form_controller.js +++ b/app/views/components/prompt_form_component/prompt_form_controller.js @@ -161,6 +161,27 @@ export default class PromptFormController extends Controller { onChangeModel() { this.initializeFileInput() + this.updateModalFrameSrc() + this.notifyModelChange() + } + + updateModalFrameSrc() { + const frame = document.getElementById('conversation-contexts') + if (frame && frame.src) { + const selectedModel = this.modelData.find(m => m.api_name === this.modelSelectTarget.value) + if (selectedModel) { + const url = new URL(frame.src, window.location.origin) + url.searchParams.set('vendor', selectedModel.vendor) + frame.src = url.toString() + } + } + } + + notifyModelChange() { + const selectedModel = this.modelData.find(m => m.api_name === this.modelSelectTarget.value) + if (selectedModel) { + this.dispatch('model-changed', { detail: { vendor: selectedModel.vendor } }) + } } // If there is an error in the background job, enabled the form diff --git a/app/views/conversation_contexts_conversations/_context_files_selector.html.haml b/app/views/conversation_contexts_conversations/_context_files_selector.html.haml index e2ea42499..459521316 100644 --- a/app/views/conversation_contexts_conversations/_context_files_selector.html.haml +++ b/app/views/conversation_contexts_conversations/_context_files_selector.html.haml @@ -14,9 +14,11 @@ class: 'form-select', size: 5, data: { action: 'change->conversation-context-selector#toggleSubmit', - 'conversation-context-selector-target' => 'select' } } do + 'conversation-context-selector-target' => 'select', + 'current-vendor' => local_assigns[:current_vendor] || @vendor } } do - available_contexts.each do |context| - %option{ value: context.id, id: dom_id(context) }= context.filename + %option{ value: context.id, id: dom_id(context), data: { vendor: context.vendor } } + = "#{context.filename} (#{context.vendor.to_s.titleize})" .mb-3 = form.submit 'Add Selected Files', class: 'btn btn-outline-primary', diff --git a/app/views/conversation_contexts_conversations/_conversation_context.html.haml b/app/views/conversation_contexts_conversations/_conversation_context.html.haml index 0b08ca933..bae3711f8 100644 --- a/app/views/conversation_contexts_conversations/_conversation_context.html.haml +++ b/app/views/conversation_contexts_conversations/_conversation_context.html.haml @@ -1,8 +1,9 @@ -.context-item.d-flex.justify-content-between.align-items-center.p-2.border.rounded.mb-2{ id: dom_id(conversation_context)} +.context-item.d-flex.justify-content-between.align-items-center.p-2.border.rounded.mb-2{ id: dom_id(conversation_context), data: { vendor: conversation_context.context.vendor, 'conversation-context-target': 'item' } } .context-info.d-flex.align-items-center %i.bi.bi-file-earmark.me-2.text-muted .context-details .file-name.fw-medium= conversation_context.context.filename + %span.badge.bg-info.small= conversation_context.context.vendor.to_s.titleize .context-actions = link_to conversation_conversation_contexts_conversation_path(conversation_context.conversation_id, conversation_context), id: dom_id(conversation_context, 'delete'), diff --git a/app/views/conversation_contexts_conversations/_upload_area.html.haml b/app/views/conversation_contexts_conversations/_upload_area.html.haml index 2c265c1bb..7f6887394 100644 --- a/app/views/conversation_contexts_conversations/_upload_area.html.haml +++ b/app/views/conversation_contexts_conversations/_upload_area.html.haml @@ -1,4 +1,4 @@ -.upload-area{ data: { controller: 'conversation-context', 'conversation-id': conversation.id } } +.upload-area .mb-3 %h6 Upload File %p.text-muted Add documents, images, or other files to provide context for this conversation. @@ -8,7 +8,7 @@ %p.mb-2 Drag and drop a file here, or click to select %input.d-none#conversation-context-file-input{ type: 'file', data: { 'conversation-context-target': 'fileInput', action: 'change->conversation-context#handleFileSelect' }, - accept: '.pdf,.txt,.doc,.docx,.json,.csv,.jpg,.jpeg,.png,.gif' } + accept: '.pdf,.txt,.md,.doc,.docx,.json,.csv,.jpg,.jpeg,.png,.gif' } %button.btn.btn-outline-primary{ type: 'button', data: { action: 'click->conversation-context#openFileDialog' } } %i.bi.bi-folder-plus.me-2 Select File diff --git a/app/views/conversation_contexts_conversations/index.html.haml b/app/views/conversation_contexts_conversations/index.html.haml index b852c3a1b..73d74ec61 100644 --- a/app/views/conversation_contexts_conversations/index.html.haml +++ b/app/views/conversation_contexts_conversations/index.html.haml @@ -1,13 +1,16 @@ %turbo-frame#conversation-contexts .modal-content .modal-header.bg-primary.text-white - %h5.modal-title#conversationContextModalLabel Conversation Context + %h5.modal-title#conversationContextModalLabel + Conversation Context + %span.badge.rounded-pill.bg-info.ms-2= @vendor.to_s.titleize %button.btn-close.btn-close-white{ type: 'button', 'data-bs-dismiss': 'modal', 'aria-label': 'Close' } - .modal-body + .modal-body{ data: { controller: 'conversation-context', 'conversation-id': @conversation.id } } = render partial: 'upload_area', locals: { conversation: @conversation } #context-files-selector - if @available_contexts.present? - = render partial: 'context_files_selector', locals: { conversation: @conversation, available_contexts: @available_contexts } + = render partial: 'context_files_selector', locals: { conversation: @conversation, available_contexts: @available_contexts, current_vendor: @vendor } + - else .conversation-context-files.mt-4 %h6 Context Files .contexts-container#conversation-context-list diff --git a/config/sidekiq_cron.yml b/config/sidekiq_cron.yml index 4917edb1e..c79641bbb 100644 --- a/config/sidekiq_cron.yml +++ b/config/sidekiq_cron.yml @@ -2,3 +2,7 @@ transcription_status_check: class: 'TranscriptionStatusCheckJob' cron: '* * * * *' queue: 'default' +cleanup_google_conversation_contexts: + class: 'CleanupGoogleConversationContextsJob' + cron: '0 * * * *' # Run every hour + queue: 'default' diff --git a/db/migrate/20260105155719_add_vendor_to_conversation_contexts.rb b/db/migrate/20260105155719_add_vendor_to_conversation_contexts.rb new file mode 100644 index 000000000..0d0b9f732 --- /dev/null +++ b/db/migrate/20260105155719_add_vendor_to_conversation_contexts.rb @@ -0,0 +1,5 @@ +class AddVendorToConversationContexts < ActiveRecord::Migration[8.0] + def change + add_column :conversation_contexts, :vendor, :string + end +end diff --git a/db/schema.rb b/db/schema.rb index 3e1a90b90..3a1f33e48 100644 --- a/db/schema.rb +++ b/db/schema.rb @@ -10,7 +10,7 @@ # # It's strongly recommended that you check this file into your version control system. -ActiveRecord::Schema[8.0].define(version: 2025_12_18_155327) do +ActiveRecord::Schema[8.0].define(version: 2026_01_05_155719) do create_schema "rollback" # These are extensions that must be enabled in order to support this database @@ -73,6 +73,7 @@ t.string "context_type", null: false t.datetime "created_at", null: false t.datetime "updated_at", null: false + t.string "vendor" t.index ["user_id"], name: "index_conversation_contexts_on_user_id" end diff --git a/factories/conversation_context.rb b/factories/conversation_context.rb index 6cd525aa7..9a7a6b1ce 100644 --- a/factories/conversation_context.rb +++ b/factories/conversation_context.rb @@ -1,9 +1,10 @@ FactoryBot.define do factory :conversation_context do - association :user - file_ref { SecureRandom.uuid } - filename { 'test_file.txt' } - mime_type { 'text/plain' } + sequence(:file_ref) { |n| "file_#{n}" } + filename { Faker::File.file_name } + mime_type { 'application/pdf' } context_type { 'file' } + vendor { :anthropic } + user end end diff --git a/lib/anthropic/invoke_model_request.rb b/lib/anthropic/invoke_model_request.rb index 19582f80f..ff8329b16 100644 --- a/lib/anthropic/invoke_model_request.rb +++ b/lib/anthropic/invoke_model_request.rb @@ -41,7 +41,12 @@ def messages turns = conversation.turns.to_a conversation.exchange.push(Turn.user_turn(generate_text_request, turns:)).tap do |ex| # prepend context documents to the user message - ex.first['content'] = [*conversation.documents, *ex.first['content']] + documents = if conversation.respond_to?(:contexts) + Array(conversation.contexts).select { |c| c.vendor == 'anthropic' }.map(&:to_content_block) + else + conversation.documents + end + ex.first['content'] = [*documents, *ex.first['content']] end end diff --git a/lib/gemini.rb b/lib/gemini.rb new file mode 100644 index 000000000..90d70e305 --- /dev/null +++ b/lib/gemini.rb @@ -0,0 +1,62 @@ +# frozen_string_literal: true + +module Gemini + HOST = 'https://generativelanguage.googleapis.com' + VERSION = 'v1beta' + + def self.vendor = :google + def self.capabilities = GenerativeText::Model::Capabilities.new(image?: true) + + def self.models + [ + GenerativeText::Model.new( + api_name: 'gemini-3-pro-preview', + name: 'Gemini 3 Pro Preview', + vendor:, + capabilities:, + max_tokens: 65_536, + active?: true + ), + GenerativeText::Model.new( + api_name: 'gemini-3-flash-preview', + name: 'Gemini 3 Flash Preview', + vendor:, + capabilities:, + max_tokens: 65_536, + active?: true + ), + GenerativeText::Model.new( + api_name: 'gemini-2.5-pro', + name: 'Gemini 2.5 Pro', + vendor:, + capabilities:, + max_tokens: 65_536, + active?: true + ), + GenerativeText::Model.new( + api_name: 'gemini-2.5-flash', + name: 'Gemini 2.5 Flash', + vendor:, + capabilities:, + max_tokens: 65_536, + active?: true + ) + ] + end + + def self.active_models + models.select(&:active?) + end + + def self.upload_file(file) + Gemini::FilesClient.new.upload_file(file) + end + + def self.delete_file(file_id) + Gemini::FilesClient.new.delete_file(file_id) + end + + class Error < StandardError; end + class ClientError < Error; end + class ServerError < Error; end +end diff --git a/lib/gemini/client.rb b/lib/gemini/client.rb new file mode 100644 index 000000000..bc7c6d31f --- /dev/null +++ b/lib/gemini/client.rb @@ -0,0 +1,75 @@ +module Gemini + class Client + # include ErrorHandling # To be implemented + + def initialize + @api_key = ENV.fetch('GEMINI_API_KEY') + @conn = Faraday.new( + url: HOST, + headers: { + 'Content-Type': 'application/json' + } + ) do |f| + f.adapter :typhoeus + end + end + + # @param generate_text_request [GenerateTextRequest] + # @return [InvokeModelResponse] + def invoke_model(generate_text_request) + model = generate_text_request.model.api_name + url = "/#{VERSION}/models/#{model}:generateContent?key=#{@api_key}" + + req_body = InvokeModelRequest.new(generate_text_request).to_json + + response = conn.post(url) do |req| + req.body = req_body + end + + if response.status.in?(200..299) + InvokeModelResponse.new(response.body) + else + raise ClientError, "Gemini API Error: #{response.status} - #{response.body}" + end + end + + # Stream responses from the Gemini API + # @param generate_text_request [GenerateTextRequest] + # @yield [String] Yields chunks of the assistant response + # @return [InvokeModelResponse] Returns the complete response when done + def invoke_model_stream(generate_text_request, &block) + model = generate_text_request.model.api_name + url = "/#{VERSION}/models/#{model}:streamGenerateContent?alt=sse&key=#{@api_key}" + + stream_response = StreamResponse.new + req_body = InvokeModelRequest.new(generate_text_request, stream: true).to_json + + response = conn.post(url) do |req| + req.body = req_body + req.options.on_data = lambda do |chunk, _received_bytes| + process_stream_chunk(chunk, stream_response, &block) + end + end + + if response.status.in?(200..299) + InvokeModelResponse.new(stream_response.to_response_format.to_json) + else + raise ClientError, "Gemini API Error: #{response.status} - #{response.body}" + end + end + + private + + attr_reader :conn + + def process_stream_chunk(chunk, stream_response, &block) + chunk.split("\n\n").each do |raw_event| + event = StreamEvent.parse(raw_event) + next unless event + + stream_response.update(event) + block.call(event.text_content) if event.text? + end + end + end +end diff --git a/lib/gemini/file_response.rb b/lib/gemini/file_response.rb new file mode 100644 index 000000000..cbb518d65 --- /dev/null +++ b/lib/gemini/file_response.rb @@ -0,0 +1,26 @@ +module Gemini + class FileResponse + include ActiveModel::Model + include ActiveModel::Attributes + + attribute :id, :string # "files/..." + attribute :filename, :string + attribute :mime_type, :string + attribute :size_bytes, :integer + attribute :created_at, :datetime + attribute :uri, :string + attribute :state, :string + + def self.for(data) + new( + id: data['uri'], # Store URI as ID for ConversationContext file_ref compatibility + filename: data['displayName'], + mime_type: data['mimeType'], + size_bytes: data['sizeBytes']&.to_i, + created_at: data['createTime'], + uri: data['uri'], + state: data['state'] + ) + end + end +end diff --git a/lib/gemini/files_client.rb b/lib/gemini/files_client.rb new file mode 100644 index 000000000..afd2e3dfa --- /dev/null +++ b/lib/gemini/files_client.rb @@ -0,0 +1,91 @@ +module Gemini + class FilesClient + attr_reader :conn + + def initialize + @api_key = ENV.fetch('GEMINI_API_KEY') + @conn = Faraday.new( + url: HOST, + headers: { + 'Content-Type': 'application/json' + } + ) do |f| + f.adapter :typhoeus + end + end + + # @param file [ActionDispatch::Http::UploadedFile] + # @return [Gemini::FileResponse] + def upload_file(file) + init_url = "/upload/#{VERSION}/files?key=#{@api_key}" + + file.rewind + content = file.read + file_size = content.bytesize.to_s + + init_response = conn.post(init_url) do |req| + req.headers['X-Goog-Upload-Protocol'] = 'resumable' + req.headers['X-Goog-Upload-Command'] = 'start' + req.headers['X-Goog-Upload-Header-Content-Length'] = file_size + req.headers['X-Goog-Upload-Header-Content-Type'] = file.content_type + req.headers['Content-Type'] = 'application/json' + req.body = { file: { display_name: file.original_filename } }.to_json + end + + unless init_response.status.in?(200..299) + raise "Gemini File Upload Init Failed: #{init_response.status} - #{init_response.body}" + end + + upload_url = init_response.headers['x-goog-upload-url'] + + # 2. Upload Bytes + # We use a new Faraday request for the upload_url as it is absolute + upload_response = Faraday.put(upload_url) do |req| + req.headers['Content-Length'] = file_size + req.headers['X-Goog-Upload-Offset'] = '0' + req.headers['X-Goog-Upload-Command'] = 'upload, finalize' + req.body = content + end + + if upload_response.status.in?(200..299) + FileResponse.for(JSON.parse(upload_response.body).dig('file')) + else + raise "Gemini File Upload Failed: #{upload_response.status} - #{upload_response.body}" + end + end + + def delete_file(file_id) + # file_id might be "files/..." or "https://.../files/..." + # Extract "files/..." + if file_id.start_with?('http') + file_id = file_id.split('/v1beta/').last + # Verify it starts with files/ just in case + unless file_id&.start_with?('files/') + # Fallback or error + file_id = file_id # Attempt to use as is + end + end + + url = "/#{VERSION}/#{file_id}?key=#{@api_key}" + + response = conn.delete(url) + + unless response.status.in?(200..299) + raise "Gemini File Delete Failed: #{response.status} - #{response.body}" + end + + true + end + + def get_file(file_id) + url = "/#{VERSION}/#{file_id}?key=#{@api_key}" + response = conn.get(url) + + if response.status.in?(200..299) + FileResponse.for(JSON.parse(response.body)) + else + raise "Gemini Get File Failed: #{response.status} - #{response.body}" + end + end + end +end diff --git a/lib/gemini/invoke_model_request.rb b/lib/gemini/invoke_model_request.rb new file mode 100644 index 000000000..c9b44b1c1 --- /dev/null +++ b/lib/gemini/invoke_model_request.rb @@ -0,0 +1,87 @@ +module Gemini + class InvokeModelRequest + attr_reader :generate_text_request, :stream + + delegate :model, :prompt, :temperature, :system_message, :conversation, to: :generate_text_request + + def initialize(generate_text_request, stream: false) + @generate_text_request = generate_text_request + @stream = stream + end + + def to_h + { + contents:, + tools:, + generationConfig: generation_config, + systemInstruction: system_instruction + }.compact + end + + def to_json + to_h.to_json + end + + private + + def tools + active_tools = LlmTool.active.where(tool_type: conversation.tool_types) + return nil if active_tools.empty? + + function_declarations = active_tools.map do |tool| + { + name: tool.name, + description: tool.description, + parameters: tool.input_schema + } + end + + [{ function_declarations: }] + end + + def contents + turns = conversation.turns.to_a + # Get history + current turn + ex = conversation.exchange.push(Turn.user_turn(generate_text_request, turns:)) + + # Prepend context documents to the first message if any + if conversation.respond_to?(:contexts) && conversation.contexts.any? + file_parts = Array(conversation.contexts).select { |c| c.vendor == 'google' }.map do |context| + next unless context.file_ref.start_with?('https://') + + { + file_data: { + mime_type: context.mime_type, + file_uri: context.file_ref + } + } + end.compact + + if file_parts.any? && ex.last && ex.last[:role] == 'user' + ex.last[:parts].unshift(*file_parts) + end + end + + ex + end + + def generation_config + { + temperature: temperature, + # topP: ..., + # topK: ..., + # maxOutputTokens: ... + }.compact + end + + def system_instruction + return nil if system_message.blank? + + { + parts: [ + { text: system_message } + ] + } + end + end +end diff --git a/lib/gemini/invoke_model_response.rb b/lib/gemini/invoke_model_response.rb new file mode 100644 index 000000000..6cff95e83 --- /dev/null +++ b/lib/gemini/invoke_model_response.rb @@ -0,0 +1,70 @@ +module Gemini + class InvokeModelResponse + attr_reader :data + + def initialize(response_body) + @response_json = if response_body.is_a?(Hash) + response_body + else + JSON.parse(response_body) + end + @data = @response_json + end + + def content + # Return the text content + results.find { |c| c['type'] == 'text' }&.fetch('text') + end + + def tool_use? + tool_inputs.any? + end + + def tool_inputs + results.select { |c| c['type'] == 'tool_use' } + end + + def blobify + [ + content, + *tool_inputs.map { _1['input'] } + ].join(' ') + end + + # Support for metadata if needed + def input_token_count + usage_metadata['promptTokenCount'].to_i + end + + def output_token_count + usage_metadata['candidatesTokenCount'].to_i + end + + private + + def results + @results ||= begin + parts = candidates.first&.dig('content', 'parts') || [] + parts.map do |part| + if part['text'].present? + { 'type' => 'text', 'text' => part['text'] } + elsif part['functionCall'].present? + { + 'type' => 'tool_use', + 'name' => part['functionCall']['name'], + 'input' => part['functionCall']['args'] + } + end + end.compact + end + end + + def candidates + @response_json['candidates'] || [] + end + + def usage_metadata + @response_json['usageMetadata'] || {} + end + end +end diff --git a/lib/gemini/stream_event.rb b/lib/gemini/stream_event.rb new file mode 100644 index 000000000..e4e8641d7 --- /dev/null +++ b/lib/gemini/stream_event.rb @@ -0,0 +1,45 @@ +module Gemini + class StreamEvent + attr_reader :data + + # Parse a raw SSE string into a StreamEvent object + def self.parse(raw_event) + # Gemini typically sends "data: { ... }" + # It might not send "event: ..." lines. + + lines = raw_event.split("\n") + data_line = lines.find { |l| l.start_with?('data: ')} + + return nil unless data_line + + json_str = data_line.sub(/^data: /, '') + parsed_data = JSON.parse(json_str) + + new(data: parsed_data) + rescue JSON::ParserError + # Verify if it's a [DONE] message or similar? + # Gemini REST API might just end. + nil + end + + def initialize(data:) + @data = data + end + + def text? + text_content.present? + end + + def text_content + @data.dig('candidates', 0, 'content', 'parts', 0, 'text') + end + + def function_call + @data.dig('candidates', 0, 'content', 'parts', 0, 'functionCall') + end + + def usage_metadata + @data['usageMetadata'] + end + end +end diff --git a/lib/gemini/stream_response.rb b/lib/gemini/stream_response.rb new file mode 100644 index 000000000..14c0a6efa --- /dev/null +++ b/lib/gemini/stream_response.rb @@ -0,0 +1,44 @@ +module Gemini + class StreamResponse + attr_reader :content_blocks, :usage_metadata + + def initialize + @content_blocks = [] + @usage_metadata = {} + @accumulated_text = "" + end + + def update(event) + text = event.text_content + if text + @accumulated_text += text + end + + if event.function_call + @function_call = event.function_call + end + + if event.usage_metadata + @usage_metadata = event.usage_metadata + end + end + + # Convert to format expected by InvokeModelResponse (and wrapper logic) + def to_response_format + parts = [] + parts << { 'text' => @accumulated_text } if @accumulated_text.present? + parts << { 'functionCall' => @function_call } if @function_call.present? + + { + 'candidates' => [ + { + 'content' => { + 'parts' => parts + } + } + ], + 'usageMetadata' => @usage_metadata + } + end + end +end diff --git a/lib/gemini/turn.rb b/lib/gemini/turn.rb new file mode 100644 index 000000000..d12d16a09 --- /dev/null +++ b/lib/gemini/turn.rb @@ -0,0 +1,101 @@ +module Gemini + class Turn + ASSISTANT = 'model'.freeze + USER = 'user'.freeze + + # Converts a GenerateTextRequest object to a tuple that consists of a user + # message and an assistant response. + def self.for(generate_text_request, turns: [], include_previous_gen_image: false) + new(generate_text_request:, turns:, include_previous_gen_image:).turn + end + + def self.user_turn(generate_text_request, turns: [], include_previous_gen_image: true) + new(generate_text_request:, turns:, include_previous_gen_image:).user_turn + end + + attr_reader :generate_text_request, :turns, :include_previous_gen_image + + delegate :prompt, :response_content, to: :generate_text_request + alias assistant_content response_content + + def initialize(generate_text_request:, turns:, include_previous_gen_image: false) + @include_previous_gen_image = include_previous_gen_image + @generate_text_request = generate_text_request + @turns = turns + end + + def turn + [ + user_turn, + assistant_turn + ] + end + + def user_turn + { + role: USER, + parts: user_parts + } + end + + def assistant_turn + { + role: ASSISTANT, + parts: [{ text: assistant_content || 'no content' }] + } + end + + private + + def user_parts + [ + user_generate_image_part, + user_upload_image_part, + user_text_part + ].compact + end + + def user_text_part + { text: prompt } + end + + # Phase 4 implementation placeholder + def user_upload_image_part + return unless generate_text_request.image_attached? + + # Use inline data for now, similar to Anthropic, + # but Gemini supports inline_data or file_data (Files API) + # For parity with current Anthropic implementation (base64 source), we use inline_data. + + image = generate_text_request.file.variant(:webp).processed.image + + { + inline_data: { + mime_type: image.content_type, + data: BlobEncoder.encode64(image) + } + } + end + + def user_generate_image_part + return unless include_previous_gen_image && previous_turn.present? && previous_turn.generated_image? + + image = previous_turn.turnable.image.variant(:webp).image + + { + inline_data: { + mime_type: image.content_type, + data: BlobEncoder.encode64(image) + } + } + end + + def previous_turn + turn_index.positive? ? turns[turn_index - 1] : nil + end + + def turn_index + turns.find_index { _1.turnable_id == generate_text_request.id } || 0 + end + end +end diff --git a/lib/generative_text.rb b/lib/generative_text.rb index 87489f027..80606f336 100644 --- a/lib/generative_text.rb +++ b/lib/generative_text.rb @@ -7,6 +7,7 @@ class GenerativeText MODELS = [ *::Anthropic.models, + *::Gemini.models, *AWS::MODELS ].freeze @@ -34,6 +35,8 @@ def self.client_for(generate_text_request) AWS::Client when :anthropic ::Anthropic::Client + when :google + ::Gemini::Client end end diff --git a/lib/generative_text/helpers.rb b/lib/generative_text/helpers.rb index 92c9520e9..b204b4198 100644 --- a/lib/generative_text/helpers.rb +++ b/lib/generative_text/helpers.rb @@ -3,8 +3,10 @@ module Helpers MARKDOWN_FORMAT_SYSTEM_MESSAGE = <<~TXT.freeze You always answer the with markdown formatting which can inlcude headings, bold, italic, links, tables, lists, code blocks, and blockquotes. If the - user asks you to produce a diagram, always use mermaid syntax. Never - explain your syntax choices when producing a mermaid diagram. + user asks you to produce a diagram, always use mermaid syntax (beware of + using parenthesis in mermaid as this can cause syntax errors unless the + text is wrapped in quotes). Never explain your syntax choices when + producing a mermaid diagram. TXT SUMMARY_TEMPLATE = <<~PROMPT.strip.freeze diff --git a/spec/fixtures/files/gemini/stream_response.txt b/spec/fixtures/files/gemini/stream_response.txt new file mode 100644 index 000000000..a8d9b780b --- /dev/null +++ b/spec/fixtures/files/gemini/stream_response.txt @@ -0,0 +1,3 @@ +data: { "candidates": [{"content": {"parts": [{"text": "Hello"}]}}] } + +data: { "candidates": [{"content": {"parts": [{"text": " World"}]}}], "usageMetadata": {"promptTokenCount": 5, "candidatesTokenCount": 2} } diff --git a/spec/lib/gemini/client_spec.rb b/spec/lib/gemini/client_spec.rb new file mode 100644 index 000000000..05847d586 --- /dev/null +++ b/spec/lib/gemini/client_spec.rb @@ -0,0 +1,75 @@ +require 'rails_helper' + +RSpec.describe Gemini::Client do + let(:client) { described_class.new } + let(:prompt) { 'Write a haiku about a rainy day.' } + let(:model) { GenerativeText::MODELS.find { |m| m.api_name == 'gemini-2.5-flash' } } + let(:temperature) { 0.1 } + let(:generate_text_request) { create :generate_text_request, model: model.api_name, prompt:, temperature: } + + before do + allow(ENV).to receive(:fetch).with('GEMINI_API_KEY').and_return('fake_key') + end + + describe '#invoke_model' do + context 'with a valid request' do + let!(:http_request) do + stub_gemini_generate_content_request(model: model) + end + + it 'calls the generateContent endpoint' do + client.invoke_model(generate_text_request) + expect(http_request).to have_been_requested + end + + it 'returns an InvokeModelResponse object' do + response = client.invoke_model(generate_text_request) + expect(response).to be_a(Gemini::InvokeModelResponse) + end + end + + context 'with an error response' do + before do + stub_gemini_generate_content_request(model: model, response_status: 400, response_body: 'Bad Request') + end + + it 'raises an error' do + expect { client.invoke_model(generate_text_request) } + .to raise_error(Gemini::ClientError, /Gemini API Error: 400/) + end + end + end + + describe '#invoke_model_stream' do + context 'with a valid request' do + let!(:http_request) do + stub_gemini_stream_generate_content_request( + model: model, + response_body: file_fixture('gemini/stream_response.txt').read + ) + end + + it 'yields text content from stream events' do + chunks = [] + client.invoke_model_stream(generate_text_request) { |chunk| chunks << chunk } + expect(chunks).to eq(['Hello', ' World']) + end + + it 'returns an InvokeModelResponse object' do + response = client.invoke_model_stream(generate_text_request) {} + expect(response).to be_a(Gemini::InvokeModelResponse) + end + end + + context 'with an error response' do + before do + stub_gemini_stream_generate_content_request(model: model, response_status: 400, response_body: 'Bad Request') + end + + it 'raises an error' do + expect { client.invoke_model_stream(generate_text_request) {} } + .to raise_error(Gemini::ClientError, /Gemini API Error: 400/) + end + end + end +end diff --git a/spec/lib/gemini/invoke_model_request_spec.rb b/spec/lib/gemini/invoke_model_request_spec.rb new file mode 100644 index 000000000..39adaaacf --- /dev/null +++ b/spec/lib/gemini/invoke_model_request_spec.rb @@ -0,0 +1,53 @@ +require 'rails_helper' + +RSpec.describe Gemini::InvokeModelRequest do + let(:model) { GenerativeText::MODELS.find { |m| m.api_name == 'gemini-2.5-flash' } } + let(:generate_text_request) { create :generate_text_request, model: model.api_name, prompt: 'Hello' } + subject { described_class.new(generate_text_request) } + + describe '#to_h' do + it 'returns the correct structure' do + json = subject.to_h + expect(json[:contents]).to be_a(Array) + expect(json[:contents].first[:parts].first[:text]).to eq('Hello') + expect(json[:generationConfig][:temperature]).to eq(generate_text_request.temperature) + end + + context 'with system instructions' do + let(:generate_text_preset) { create :generate_text_preset, system_message: 'Be helpful' } + let(:generate_text_request) do + create :generate_text_request, model: model.api_name, generate_text_preset: generate_text_preset + end + + it 'includes systemInstruction' do + json = subject.to_h + expect(json[:systemInstruction][:parts].first[:text]).to include('Be helpful') + end + end + + context 'with tools' do + let(:tool) do + create :llm_tool, + name: 'weather_tool', + description: 'Get weather', + tool_type: 'image', + input_schema: { type: 'object', properties: {} }.to_json + end + let(:conversation) { create :conversation, tool_types: ['image'] } + let(:generate_text_request) do + create :generate_text_request, model: model.api_name, conversation: conversation + end + + before do + allow(LlmTool).to receive(:active).and_return(LlmTool.where(id: tool.id)) + end + + it 'includes tools' do + json = subject.to_h + expect(json[:tools]).to be_a(Array) + expect(json[:tools].first[:function_declarations]).to be_a(Array) + expect(json[:tools].first[:function_declarations].first[:name]).to eq('WeatherTool') + end + end + end +end diff --git a/spec/lib/gemini/invoke_model_response_spec.rb b/spec/lib/gemini/invoke_model_response_spec.rb new file mode 100644 index 000000000..81a49c535 --- /dev/null +++ b/spec/lib/gemini/invoke_model_response_spec.rb @@ -0,0 +1,72 @@ +require 'rails_helper' + +RSpec.describe Gemini::InvokeModelResponse do + let(:response_body) do + { + candidates: [ + { + content: { + parts: [{ text: 'Response text' }] + } + } + ], + usageMetadata: { + promptTokenCount: 10, + candidatesTokenCount: 20 + } + }.to_json + end + + subject { described_class.new(response_body) } + + describe '#content' do + it 'returns the text content' do + expect(subject.content).to eq('Response text') + end + end + + describe '#data' do + it 'returns the raw response hash' do + expect(subject.data).to eq(JSON.parse(response_body)) + end + end + + describe '#token_count' do + it 'returns correct counts' do + expect(subject.input_token_count).to eq(10) + expect(subject.output_token_count).to eq(20) + end + end + + context 'with tool use' do + let(:response_body) do + { + candidates: [ + { + content: { + parts: [ + { + functionCall: { + name: 'get_weather', + args: { location: 'London' } + } + } + ] + } + } + ] + }.to_json + end + + it 'detects tool use' do + expect(subject.tool_use?).to be true + end + + it 'extracts tool inputs' do + inputs = subject.tool_inputs + expect(inputs.size).to eq(1) + expect(inputs.first['name']).to eq('get_weather') + expect(inputs.first['input']).to eq({ 'location' => 'London' }) + end + end +end diff --git a/spec/rails_helper.rb b/spec/rails_helper.rb index 3583fb92b..228ee2ce7 100644 --- a/spec/rails_helper.rb +++ b/spec/rails_helper.rb @@ -8,6 +8,7 @@ ENV['AWS_BUCKET'] = 'aws-bucket' ENV['ANTHROPIC_KEY'] = 'anthropic_key' ENV['DEVISE_JWT_KEY'] = '5c2bfb28d566bd3aff8e3d5571803149648215d6bfe2f24e2290d9ba15acd73f1fa0bcaa16817dc0db5f8b96da74e2a598806c436dce999327d82c89245fee4f' +ENV['GEMINI_API_KEY'] = 'gemini_api_key' ENV['GITHUB_ID'] = 'github_id' ENV['GITHUB_SECRET'] = 'github_secret' ENV['POSTGRES_HOST'] = 'database' diff --git a/spec/sidekiq/generate_text_job_spec.rb b/spec/sidekiq/generate_text_job_spec.rb index 68cf62bc5..bb056ee8c 100644 --- a/spec/sidekiq/generate_text_job_spec.rb +++ b/spec/sidekiq/generate_text_job_spec.rb @@ -1,174 +1,43 @@ require 'rails_helper' RSpec.describe GenerateTextJob, type: :job do - describe 'sidekiq_options' do - subject { described_class.sidekiq_options } + include GeminiHelpers - it { is_expected.to include('retry' => 1) } + let(:user) { create :user } + let(:model) { GenerativeText::MODELS.find { |m| m.vendor == :google } } + let(:conversation) { create :conversation, user: user } + let(:generate_text_request) do + create :generate_text_request, user: user, model: model.api_name, prompt: 'Hello Gemini', conversation: conversation end - + describe '#perform' do - subject(:perform) { described_class.new.perform(generate_text_request.id, stream) } - - let(:stream) { false } - let(:conversation_turn) { build_stubbed :conversation_turn, conversation: } - let(:generate_text_request) { build_stubbed :generate_text_request, :with_preset, conversation_turn:, user: } - let(:user) { build_stubbed :user, setting: build(:setting) } - let(:conversation) { build_stubbed :conversation } - let(:response_data) { { 'content' => 'response data' } } - let(:response) do - instance_double(Anthropic::InvokeModelResponse, - content: Faker::Lorem.paragraph, data: response_data, tool_use?: false) - end - let(:prompt_form_component) { instance_double PromptFormComponent } - let(:conversation_turn_component) { instance_double ConversationTurnComponent } - let(:generative_text) { instance_double(GenerativeText, invoke_model: response) } - - before do - allow(GenerateTextRequest).to receive(:find).and_return(generate_text_request) - allow(MyChannel).to receive(:broadcast_to) - allow(ViewComponentBroadcaster).to receive(:call) - allow(GenerativeText).to receive(:new).and_return(generative_text) - allow(PromptFormComponent).to receive(:new).with(conversation_form: kind_of(ConversationForm)) - .and_return(prompt_form_component) - allow(ConversationTurnComponent).to receive(:new).with(conversation_turn:) - .and_return(conversation_turn_component) - allow(generate_text_request).to receive(:update!) - allow(generate_text_request).to receive(:in_progress!) - allow(generate_text_request).to receive(:failed!) - allow(generate_text_request).to receive(:conversation).and_return(conversation) - allow(conversation).to receive(:reload).and_return(conversation) - allow(ConversationEmbeddingJob).to receive(:perform_in) - end - - context 'when the text is generated successfully' do - it 'marks the request as in_progress' do - perform - expect(generate_text_request).to have_received(:in_progress!) - end - - it 'broadcasts the text content' do - perform - expect(MyChannel).to( - have_received(:broadcast_to).with( - user, - generate_text: { 'text_id' => generate_text_request.text_id, - 'user_id' => user.id, - conversation_id: conversation.id, - content: response.content, - error: nil } - ) - ) - end - - it 'updates the request' do - perform - expect(generate_text_request).to have_received(:update!).with(response: response_data, status: 'completed') - end - - it 'broadcasts the ConversationTurnComponent' do - perform - expect(ViewComponentBroadcaster).to( - have_received(:call).with([user, TurboStreams::STREAMS[:main]], - component: conversation_turn_component, action: :replace) + context 'with a Gemini model' do + before do + allow(ENV).to receive(:fetch).with('GEMINI_API_KEY').and_return('fake_key') + + stub_gemini_generate_content_request( + model: model, + response_body: { + candidates: [ + { + content: { + parts: [{ text: 'Hello from Gemini' }] + } + } + ], + usageMetadata: { promptTokenCount: 5, candidatesTokenCount: 5 } + }.to_json ) end - it 'broadcasts the PromptFormComponent' do - perform - expect(ViewComponentBroadcaster).to( - have_received(:call).with([user, TurboStreams::STREAMS[:main]], - component: prompt_form_component, action: :replace) - ) - end - - it 'schedules a ConversationEmbeddingJob' do - perform - expect(ConversationEmbeddingJob).to have_received(:perform_in).with(5.minutes, conversation.id) - end - - context 'when streaming is true' do - let(:stream) { true } - let(:response_chunks) { %w[chunk1 chunk2 chunk3 chunk4 chunk5 chunk6] } - let(:markdown_to_html_component) { instance_double ContentCustomizerComponent } - - before do - allow(generative_text).to receive(:invoke_model_stream).and_yield(response_chunks[0]) - .and_yield(response_chunks[1]) - .and_yield(response_chunks[2]) - .and_yield(response_chunks[3]) - .and_yield(response_chunks[4]) - .and_yield(response_chunks[5]) - .and_return(response) - allow(ContentCustomizerComponent).to receive(:new).with(markup: response_chunks.take(5).join, - markdown: true, - simple: true) - .and_return(markdown_to_html_component) - end - - it 'broadcasts the message on every 5th chunk' do - perform - expect(ViewComponentBroadcaster).to( - have_received(:call).with([user, TurboStreams::STREAMS[:main]], - component: markdown_to_html_component, - action: :update, - target: "assistant_response_generate_text_request_#{generate_text_request.id}") - ) - end - - it 'updates the request' do - perform - expect(generate_text_request).to have_received(:update!).with(response: response_data, status: 'completed') - end - - it 'broadcasts the ConversationTurnComponent' do - perform - expect(ViewComponentBroadcaster).to( - have_received(:call).with([user, TurboStreams::STREAMS[:main]], - component: conversation_turn_component, action: :replace) - ) - end - - it 'broadcasts the PromptFormComponent' do - perform - expect(ViewComponentBroadcaster).to( - have_received(:call).with([user, TurboStreams::STREAMS[:main]], - component: prompt_form_component, action: :replace) - ) - end - end - end - - describe '.on_retries_exhausted' do - subject(:on_retries_exhausted) { described_class.on_retries_exhausted(generate_text_request.id) } - - it 'marks the request as failed' do - on_retries_exhausted - expect(generate_text_request).to have_received(:failed!) - end - - it 'broadcasts the ConversationTurnComponent' do - on_retries_exhausted - expect(ViewComponentBroadcaster).to( - have_received(:call).with([user, TurboStreams::STREAMS[:main]], - component: conversation_turn_component, action: :replace) - ) - end - - it 'broadcasts an error response' do - on_retries_exhausted - expect(MyChannel).to have_received(:broadcast_to).with(user, - generate_text: { text_id: generate_text_request.text_id, - content: nil, error: true }) - end - - it 'broadcasts a flash message' do - on_retries_exhausted - expect(ViewComponentBroadcaster).to( - have_received(:call).with([generate_text_request.user, TurboStreams::STREAMS[:main]], - component: kind_of(FlashMessageComponent), action: :update) - ) + it 'completes the request and saves the response' do + described_class.new.perform(generate_text_request.id, false) + + generate_text_request.reload + expect(generate_text_request.status).to eq('completed') + expect(generate_text_request.response.content).to eq('Hello from Gemini') + expect(generate_text_request.response.data).to be_present end end end -end +end \ No newline at end of file diff --git a/spec/support/gemini_helpers.rb b/spec/support/gemini_helpers.rb new file mode 100644 index 000000000..b94af024b --- /dev/null +++ b/spec/support/gemini_helpers.rb @@ -0,0 +1,28 @@ +module GeminiHelpers + def stub_gemini_generate_content_request(model:, response_status: 200, response_body: nil) + response_body ||= { + candidates: [ + { + content: { + parts: [{ text: 'Response' }] + } + } + ], + usageMetadata: { promptTokenCount: 10, candidatesTokenCount: 10 } + }.to_json + + stub_request(:post, "#{Gemini::HOST}/#{Gemini::VERSION}/models/#{model.api_name}:generateContent") + .with(query: hash_including(key: ENV.fetch('GEMINI_API_KEY'))) + .to_return(status: response_status, body: response_body) + end + + def stub_gemini_stream_generate_content_request(model:, response_status: 200, response_body: nil) + stub_request(:post, "#{Gemini::HOST}/#{Gemini::VERSION}/models/#{model.api_name}:streamGenerateContent") + .with(query: hash_including(alt: 'sse', key: ENV.fetch('GEMINI_API_KEY'))) + .to_return(status: response_status, body: response_body) + end +end + +RSpec.configure do |config| + config.include GeminiHelpers +end