mirror of
https://github.com/we-promise/sure.git
synced 2026-09-05 06:41:08 +00:00
Add live AI checks to system health (#3155)
* Add live AI checks to system health Give super admins a dedicated AI status view with bounded liveness probes for LLMs, vector stores, pgvector, and embedding endpoints. Record sanitized failures in both the system debug log and Rails logger, and document the recommended local configuration.\n\nCloses #3145 * Fix AI health CI checks * Address AI health review feedback * Correct Ollama model preload guidance * Distinguish OpenAI-compatible providers * Make Ollama startup readiness explicit * Recognize Cloudflare AI endpoints
This commit is contained in:
Juan José Mata
+68
-8
@@ -1309,8 +1309,8 @@ OPENAI_ACCESS_TOKEN=sk-proj-...
|
||||
Use PostgreSQL's pgvector extension for fully local document search. All data stays on your infrastructure.
|
||||
|
||||
**Requirements:**
|
||||
- Use the `pgvector/pgvector:pg16` Docker image instead of `postgres:16` (drop-in replacement)
|
||||
- An embedding model served via an OpenAI-compatible `/v1/embeddings` endpoint (e.g. Ollama with `nomic-embed-text`)
|
||||
- Use the `pgvector/pgvector:pg16-trixie` Docker image instead of `postgres:16` (drop-in replacement)
|
||||
- An embedding model served via an OpenAI-compatible `/v1/embeddings` endpoint (e.g. Ollama with `mxbai-embed-large`)
|
||||
- Run the migration with `VECTOR_STORE_PROVIDER=pgvector` to create the `vector_store_chunks` table
|
||||
|
||||
```bash
|
||||
@@ -1318,22 +1318,55 @@ Use PostgreSQL's pgvector extension for fully local document search. All data st
|
||||
VECTOR_STORE_PROVIDER=pgvector
|
||||
|
||||
# Embedding model configuration
|
||||
EMBEDDING_MODEL=nomic-embed-text # Default: nomic-embed-text
|
||||
EMBEDDING_MODEL=mxbai-embed-large # Default: mxbai-embed-large
|
||||
EMBEDDING_DIMENSIONS=1024 # Default: 1024 (must match your model)
|
||||
EMBEDDING_URI_BASE=http://ollama:11434/v1 # Falls back to OPENAI_URI_BASE if not set
|
||||
EMBEDDING_ACCESS_TOKEN= # Falls back to OPENAI_ACCESS_TOKEN if not set
|
||||
```
|
||||
|
||||
Sure enables the `vector` extension when it first provisions the chunks table,
|
||||
provided the database user has permission. If the AI status page reports that
|
||||
the extension is available but not enabled and automatic provisioning cannot
|
||||
enable it, connect as the PostgreSQL superuser and run:
|
||||
|
||||
```sql
|
||||
CREATE EXTENSION vector;
|
||||
```
|
||||
|
||||
The LLM and embedding endpoints are independent. A common fully local setup is
|
||||
an OpenAI-compatible chat model through `OPENAI_URI_BASE`, pgvector for storage,
|
||||
and an embedding model through `EMBEDDING_URI_BASE`. Make sure
|
||||
`EMBEDDING_DIMENSIONS` matches the selected embedding model (for example,
|
||||
`mxbai-embed-large` uses 1024 dimensions).
|
||||
|
||||
If you are using Ollama (as in `compose.example.ai.yml`), pull the embedding model:
|
||||
|
||||
```bash
|
||||
docker compose exec ollama ollama pull nomic-embed-text
|
||||
docker compose -f compose.example.ai.yml --profile local-ai up -d --wait ollama
|
||||
docker compose exec ollama ollama pull mxbai-embed-large
|
||||
```
|
||||
|
||||
> [!WARNING]
|
||||
> Do not change `EMBEDDING_MODEL` for an existing pgvector index without
|
||||
> rebuilding it. Vectors created by different models are not comparable, even
|
||||
> when they have the same dimensions. Back up the database and the source
|
||||
> documents, then remove the existing documents from Sure. If the new model has
|
||||
> different dimensions, drop the now-empty chunks table so Sure can recreate it
|
||||
> with the new vector size:
|
||||
>
|
||||
> ```bash
|
||||
> docker compose -f compose.example.ai.yml exec web bin/rails runner \
|
||||
> 'ActiveRecord::Base.connection_pool.with_connection { |connection| connection.drop_table(VectorStore::Pgvector::TABLE_NAME, if_exists: true) }'
|
||||
> ```
|
||||
>
|
||||
> Change the embedding settings and restart Sure. Confirm that **System health
|
||||
> → AI status** reports the new model and dimensions, then upload the source
|
||||
> documents again. This recreates every embedding with only the new model.
|
||||
|
||||
##### Qdrant (Self-Hosted)
|
||||
|
||||
> [!CAUTION]
|
||||
> Only `OpenAI` has been implemented!
|
||||
> Qdrant is not implemented yet. Use OpenAI or pgvector for document search.
|
||||
|
||||
Use a dedicated Qdrant vector database:
|
||||
|
||||
@@ -1369,7 +1402,33 @@ volumes:
|
||||
|
||||
#### Verifying the Configuration
|
||||
|
||||
You can check whether a vector store is properly configured from the Rails console:
|
||||
Super admins can open **System health → AI status** at
|
||||
`/admin/system_health?tab=ai`. Opening that URL runs bounded, non-destructive
|
||||
live checks against the effective configuration:
|
||||
|
||||
- OpenAI-compatible and Anthropic providers must return the configured model
|
||||
from their models API.
|
||||
- The hosted OpenAI vector-store adapter must answer a list request without
|
||||
creating or changing a store.
|
||||
- The pgvector adapter must have its extension enabled, its chunks table
|
||||
present, and successfully execute a query.
|
||||
- A pgvector embedding endpoint must create one short test embedding, and the
|
||||
returned vector must match `EMBEDDING_DIMENSIONS`. The test vector is not
|
||||
stored.
|
||||
|
||||
When `OPENAI_URI_BASE` points outside OpenAI's hosted API, the page labels the
|
||||
selected provider **OpenAI-compatible** and identifies a known effective
|
||||
provider from the endpoint (for example Ollama, OpenRouter, Together, Kilo, or
|
||||
Cloudflare Workers AI/AI Gateway). Unrecognized services are shown as **Custom
|
||||
endpoint**.
|
||||
|
||||
Results are cached for 60 seconds by default. **Run checks again** bypasses the
|
||||
cache. Set `AI_HEALTH_PROBE_TIMEOUT` to change the default five-second request
|
||||
timeout and `AI_HEALTH_PROBE_CACHE_TTL` to change the cache duration. Failed
|
||||
checks are written as system-wide entries in **Settings → Debug logs** and to
|
||||
`Rails.logger`; endpoints are redacted and credentials are never included.
|
||||
|
||||
You can also check the adapter from the Rails console:
|
||||
|
||||
```ruby
|
||||
VectorStore.configured? # => true / false
|
||||
@@ -1386,7 +1445,8 @@ The following file extensions are supported for document upload and search:
|
||||
#### Privacy Notes
|
||||
|
||||
- **OpenAI backend:** Document content is sent to OpenAI's API for indexing and search. The same privacy considerations as the AI chat apply.
|
||||
- **Pgvector / Qdrant backends:** All data stays on your infrastructure. No external API calls are made for document search.
|
||||
- **Pgvector backend:** Stored chunks stay in PostgreSQL. Text is still sent to the configured embedding endpoint, which may be local or remote.
|
||||
- **Qdrant backend:** The adapter is currently scaffolded and cannot upload or search documents.
|
||||
|
||||
### Multi-Model Setup
|
||||
|
||||
@@ -1432,4 +1492,4 @@ For issues with AI features:
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** March 2026
|
||||
**Last Updated:** August 2026
|
||||
|
||||
Reference in New Issue
Block a user