Any explanation on how RAG works?

[bilogic]

Hi!

I'm trying to figure out this RAG and can't connect a few things together, let me explain

The 3 key things I know of are

• embeddings provider, seems you can use embeddinggemma locally
• vector database, I think it is LanceDB
• LLM, seems like you can pick any from opencode

1. It seems weird to me that an the 3 can work with embeddinggemma, I was on the impression that if an LLM doesn't have a matching embeddings provider, then it cannot be turned into a RAG, how does this actually work? Or did I misunderstand?
2. How does the remote LLM actually search the local vector database?

Thanks!

[MrDoe]

I get why this feels confusing — I had the same “who talks to whom?” moment when I first built a RAG pipeline. The simplest way I think about it: embeddings and the vector DB are the search engine, the LLM is the writer. You create vector representations of your documents (indexing) and store them in a DB like LanceDB. At query time you turn the user question into an embedding, use that to fetch the most relevant passages, then hand those passages (as plain text) to the LLM. The LLM never needs the vectors — it only needs the retrieved text to compose an answer.

That means an LLM doesn’t have to “use the same” embedding provider to be part of RAG. What does matter is that the index and the query use compatible embeddings so the DB returns the right passages. If you index with EmbeddingGemma and also compute query embeddings with EmbeddingGemma (same model, same preprocessing), retrieval quality will be much better. If you mix very different embedding models, nearest‑neighbor search degrades and you’ll get less relevant context to feed the LLM.

[MrDoe]

To answer the second question: This is a typical RAG pipeline:

Prerequisite - Index all files

1. Split the files in chunks
2. Use embedding LLM to calculate the vector of each chunk
3. Store each chunk with vector and clear text in vector database.

Semantic Search

1. User prompt is sent to the embedding LLM.
   -> Embedding vector is generated.
2. Find nearest neighbors of this vector in vector space to get the most relevant chunks.
3. Pass the chunks (as text) to the LLM.

[MrDoe]

I've set up "Discussions" for questions in the repo now.

[bilogic]

Thanks! Can this issue be converted to a discusssion, I have seen it happen on other repos.

For clarity, I would like to name these:

• embeddings provider, seems you can use embeddinggemma locally

• vector database, I think it is LanceDB

• LLM, seems like you can pick any from opencode, let's use big pickle from opencode

  Prerequisite - Index all files

  ...
  2. Use embedding LLM to calculate the vector of each chunk
  ...

  Semantic Search

  1. User prompt is sent to the embedding LLM.
     ...

1. I see a new term embedding LLM in Prerequisite and Semantic Search
2. Do they both refer to embeddinggemma or one of the LLM in opencode, e.g. big pickle?
3. Are you able to point me to the code where the User prompt is sent to opencode in return for an embedding vector?