feat(vector-store): Implement pgvector adapter for self-hosted RAG (#1211)

* Add conditional migration for vector_store_chunks table

Creates the pgvector-backed chunks table when VECTOR_STORE_PROVIDER=pgvector.
Enables the vector extension, adds store_id/file_id indexes, and uses
vector(1024) column type for embeddings.

* Add VectorStore::Embeddable concern for text extraction and embedding

Shared concern providing extract_text (PDF via pdf-reader, plain-text as-is),
paragraph-boundary chunking (~2000 chars, ~200 overlap), and embed/embed_batch
via OpenAI-compatible /v1/embeddings endpoint using Faraday. Configurable via
EMBEDDING_MODEL, EMBEDDING_URI_BASE, with fallback to OPENAI_* env vars.

* Implement VectorStore::Pgvector adapter with raw SQL

Replaces the stub with a full implementation using
ActiveRecord::Base.connection with parameterized binds. Supports
create_store, delete_store, upload_file (extract+chunk+embed+insert),
remove_file, and cosine-similarity search via the <=> operator.

* Add registry test for pgvector adapter selection

* Configure pgvector in compose.example.ai.yml

Switch db image to pgvector/pgvector:pg16, add VECTOR_STORE_PROVIDER,
EMBEDDING_MODEL, and EMBEDDING_DIMENSIONS env vars, and include
nomic-embed-text in Ollama's pre-loaded models.

* Update pgvector docs from scaffolded to ready

Document env vars, embedding model setup, pgvector Docker image
requirement, and Ollama pull instructions.

* Address PR review feedback

- Migration: remove env guard, use pgvector_available? check so it runs
  on plain Postgres (CI) but creates the table on pgvector-capable servers.
  Add NOT NULL constraints on content/embedding/metadata, unique index on
  (store_id, file_id, chunk_index).
- Pgvector adapter: wrap chunk inserts in a DB transaction to prevent
  partial file writes. Override supported_extensions to match formats
  that extract_text can actually parse.
- Embeddable: add hard_split fallback for paragraphs exceeding CHUNK_SIZE
  to avoid overflowing embedding model token limits.

* Bump schema version to include vector_store_chunks migration

CI uses db:schema:load which checks the version — without this bump,
the migration is detected as pending and tests fail to start.

* Update 20260316120000_create_vector_store_chunks.rb

---------

Co-authored-by: sokiee <sokysrm@gmail.com>
This commit is contained in:
Dream
2026-03-20 12:01:31 -04:00
committed by GitHub
parent 2cdddd28d7
commit 6d22514c01
9 changed files with 672 additions and 59 deletions

View File

@@ -1140,7 +1140,7 @@ Sure's AI assistant can search documents that have been uploaded to a family's v
| Backend | Status | Best For | Requirements |
|---------|--------|----------|--------------|
| **OpenAI** (default) | ready | Cloud deployments, zero setup | `OPENAI_ACCESS_TOKEN` |
| **Pgvector** | scaffolded | Self-hosted, full data privacy | PostgreSQL with `pgvector` extension |
| **Pgvector** | ready | Self-hosted, full data privacy | PostgreSQL with `pgvector` extension + embedding model |
| **Qdrant** | scaffolded | Self-hosted, dedicated vector DB | Running Qdrant instance |
#### Configuration
@@ -1156,16 +1156,29 @@ OPENAI_ACCESS_TOKEN=sk-proj-...
##### Pgvector (Self-Hosted)
> [!CAUTION]
> Only `OpenAI` has been implemented!
Use PostgreSQL's pgvector extension for fully local document search. All data stays on your infrastructure.
Use PostgreSQL's pgvector extension for fully local document search:
**Requirements:**
- Use the `pgvector/pgvector:pg16` Docker image instead of `postgres:16` (drop-in replacement)
- An embedding model served via an OpenAI-compatible `/v1/embeddings` endpoint (e.g. Ollama with `nomic-embed-text`)
- Run the migration with `VECTOR_STORE_PROVIDER=pgvector` to create the `vector_store_chunks` table
```bash
# Required
VECTOR_STORE_PROVIDER=pgvector
# Embedding model configuration
EMBEDDING_MODEL=nomic-embed-text # Default: nomic-embed-text
EMBEDDING_DIMENSIONS=1024 # Default: 1024 (must match your model)
EMBEDDING_URI_BASE=http://ollama:11434/v1 # Falls back to OPENAI_URI_BASE if not set
EMBEDDING_ACCESS_TOKEN= # Falls back to OPENAI_ACCESS_TOKEN if not set
```
> **Note:** The pgvector adapter is currently a skeleton. A future release will add full support including embedding model configuration.
If you are using Ollama (as in `compose.example.ai.yml`), pull the embedding model:
```bash
docker compose exec ollama ollama pull nomic-embed-text
```
##### Qdrant (Self-Hosted)