mirror of
https://github.com/we-promise/sure.git
synced 2026-04-18 19:44:09 +00:00
feat(vector-store): Implement pgvector adapter for self-hosted RAG (#1211)
* Add conditional migration for vector_store_chunks table Creates the pgvector-backed chunks table when VECTOR_STORE_PROVIDER=pgvector. Enables the vector extension, adds store_id/file_id indexes, and uses vector(1024) column type for embeddings. * Add VectorStore::Embeddable concern for text extraction and embedding Shared concern providing extract_text (PDF via pdf-reader, plain-text as-is), paragraph-boundary chunking (~2000 chars, ~200 overlap), and embed/embed_batch via OpenAI-compatible /v1/embeddings endpoint using Faraday. Configurable via EMBEDDING_MODEL, EMBEDDING_URI_BASE, with fallback to OPENAI_* env vars. * Implement VectorStore::Pgvector adapter with raw SQL Replaces the stub with a full implementation using ActiveRecord::Base.connection with parameterized binds. Supports create_store, delete_store, upload_file (extract+chunk+embed+insert), remove_file, and cosine-similarity search via the <=> operator. * Add registry test for pgvector adapter selection * Configure pgvector in compose.example.ai.yml Switch db image to pgvector/pgvector:pg16, add VECTOR_STORE_PROVIDER, EMBEDDING_MODEL, and EMBEDDING_DIMENSIONS env vars, and include nomic-embed-text in Ollama's pre-loaded models. * Update pgvector docs from scaffolded to ready Document env vars, embedding model setup, pgvector Docker image requirement, and Ollama pull instructions. * Address PR review feedback - Migration: remove env guard, use pgvector_available? check so it runs on plain Postgres (CI) but creates the table on pgvector-capable servers. Add NOT NULL constraints on content/embedding/metadata, unique index on (store_id, file_id, chunk_index). - Pgvector adapter: wrap chunk inserts in a DB transaction to prevent partial file writes. Override supported_extensions to match formats that extract_text can actually parse. - Embeddable: add hard_split fallback for paragraphs exceeding CHUNK_SIZE to avoid overflowing embedding model token limits. * Bump schema version to include vector_store_chunks migration CI uses db:schema:load which checks the version — without this bump, the migration is detected as pending and tests fail to start. * Update 20260316120000_create_vector_store_chunks.rb --------- Co-authored-by: sokiee <sokysrm@gmail.com>
This commit is contained in:
@@ -1140,7 +1140,7 @@ Sure's AI assistant can search documents that have been uploaded to a family's v
|
||||
| Backend | Status | Best For | Requirements |
|
||||
|---------|--------|----------|--------------|
|
||||
| **OpenAI** (default) | ready | Cloud deployments, zero setup | `OPENAI_ACCESS_TOKEN` |
|
||||
| **Pgvector** | scaffolded | Self-hosted, full data privacy | PostgreSQL with `pgvector` extension |
|
||||
| **Pgvector** | ready | Self-hosted, full data privacy | PostgreSQL with `pgvector` extension + embedding model |
|
||||
| **Qdrant** | scaffolded | Self-hosted, dedicated vector DB | Running Qdrant instance |
|
||||
|
||||
#### Configuration
|
||||
@@ -1156,16 +1156,29 @@ OPENAI_ACCESS_TOKEN=sk-proj-...
|
||||
|
||||
##### Pgvector (Self-Hosted)
|
||||
|
||||
> [!CAUTION]
|
||||
> Only `OpenAI` has been implemented!
|
||||
Use PostgreSQL's pgvector extension for fully local document search. All data stays on your infrastructure.
|
||||
|
||||
Use PostgreSQL's pgvector extension for fully local document search:
|
||||
**Requirements:**
|
||||
- Use the `pgvector/pgvector:pg16` Docker image instead of `postgres:16` (drop-in replacement)
|
||||
- An embedding model served via an OpenAI-compatible `/v1/embeddings` endpoint (e.g. Ollama with `nomic-embed-text`)
|
||||
- Run the migration with `VECTOR_STORE_PROVIDER=pgvector` to create the `vector_store_chunks` table
|
||||
|
||||
```bash
|
||||
# Required
|
||||
VECTOR_STORE_PROVIDER=pgvector
|
||||
|
||||
# Embedding model configuration
|
||||
EMBEDDING_MODEL=nomic-embed-text # Default: nomic-embed-text
|
||||
EMBEDDING_DIMENSIONS=1024 # Default: 1024 (must match your model)
|
||||
EMBEDDING_URI_BASE=http://ollama:11434/v1 # Falls back to OPENAI_URI_BASE if not set
|
||||
EMBEDDING_ACCESS_TOKEN= # Falls back to OPENAI_ACCESS_TOKEN if not set
|
||||
```
|
||||
|
||||
> **Note:** The pgvector adapter is currently a skeleton. A future release will add full support including embedding model configuration.
|
||||
If you are using Ollama (as in `compose.example.ai.yml`), pull the embedding model:
|
||||
|
||||
```bash
|
||||
docker compose exec ollama ollama pull nomic-embed-text
|
||||
```
|
||||
|
||||
##### Qdrant (Self-Hosted)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user