mirror of
https://github.com/we-promise/sure.git
synced 2026-09-09 08:34:26 +00:00
* feat: verify AI configuration and provider liveness from worker processes
Closes #3169.
The AI status page (#3145, PR #3155) proves only that the `web` process
resolved a valid-looking configuration and can reach the configured
provider from its own network context. Most AI workloads -- assistant
responses, PDF processing, embeddings, auto-categorization, and merchant
detection -- actually run in Sidekiq `worker` processes, which can differ
from `web` in environment, DNS, proxy rules, network policy, or even
loaded credentials (workload-specific overrides, an updated Secret without
a pod restart, `web` recreated without `worker`). A passing web check says
nothing about whether a worker can do the same.
## What this adds
`WorkerAiHealthCheckJob`, queued on demand from a new "Verify worker
configuration" button on System health -> AI status. It runs the same
bounded, non-destructive probes `AiHealth` already runs, but from inside
whichever Sidekiq worker process dequeues it, and records the result via
`WorkerAiHealth`: process identity (hostname:pid), checked-at time, a
non-secret configuration fingerprint (effective provider, model, redacted
endpoint, vector-store adapter/embedding config), and probe outcomes.
The AI status tab lists every recorded result -- most recent first, kept
for `WorkerAiHealth::RETENTION` (15 minutes) -- each labeled with a status
pill (`Passing` / `Failing` / `Stale`, the last once older than
`STALE_AFTER`) and a configuration pill comparing it against the web
snapshot (`Matches web` / `Differs from web`), with a failure-reason list
reusing the existing failure-code translations when a probe failed.
## Implementation constraints from the issue, addressed directly
- **Cannot reuse a web-cached probe result, or vice versa.** `AiHealth.new`
gained an injectable `probe_cache:` (default `Rails.cache`, matching
today's behavior). The worker job passes a fresh
`ActiveSupport::Cache::NullStore` instead, so every worker check is a
live call that neither reads a web-cached entry nor leaves one behind.
- **A single job only verifies one worker.** Documented on the button
(`coverage_notice`) and in the docs: with multiple replicas, a passing
result names one process, not the fleet. Queuing again samples another.
- **Never persists or displays a raw credential.** `WorkerAiHealth::Snapshot`
only carries redacted endpoints (AiHealth already redacts these before
they reach the job) and provider/model/status fields -- there is no field
for a token to occupy. A structural test asserts this stays true.
- **Failures land in both places an operator already checks.** Same
destinations as `AiHealth::Probe`'s own failures: `Rails.logger` and
`DebugLogEntry` (new `ai_health_worker` category), tagged with the
process identity.
- **Results carry a clear status**, including the `pending` case implicitly
(no result yet renders an explanatory empty state) and `stale` for a
result whose process may no longer reflect current state.
- **DB-backed vs ENV-backed settings are labeled.** A new info block next
to the worker results explains which UI settings propagate automatically
(rails-settings-cached invalidates the shared cache on write) versus
which require restarting/recreating both `web` and `worker`.
## What this deliberately doesn't do
Full-fleet coverage (every process publishing a periodic fingerprint) --
the issue lists this under "Other options to consider," not the acceptance
criteria, and Sidekiq's normal dispatch doesn't target every process
without an explicit per-process coordination mechanism. This PR implements
the on-demand, single-check design the acceptance criteria actually
describes ("An administrator can request an asynchronous worker-side
check", "does not imply full-fleet coverage"); periodic fleet-wide
publishing is a natural follow-up if operators need it.
## Testing
- `WorkerAiHealth`: recording/reading, same-process replacement vs.
cross-process coexistence, MAX_RESULTS bounding, staleness, status
derivation (failure codes / component statuses / function-calling
refusal), `matches_web?` comparison, and the credential-field structural
guard.
- `WorkerAiHealthCheckJob`: records a passing/failing snapshot naming this
process, writes failures to Rails.logger + DebugLogEntry (and only on
failure), never leaks the access token, and -- the defining property --
is proven to construct `AiHealth.new` with an isolated `NullStore`
rather than the shared web-facing cache.
- `Admin::SystemHealthController`: empty state, a rendered result with
matching/mismatched configuration, a failing result's failure reason, a
stale result, the `verify_worker_ai` action enqueuing the job and
redirecting with a flash notice, and that non-super-admins and
unauthenticated requests cannot trigger it.
I could not run the Rails test suite in this environment (no working
Ruby/Bundler toolchain available locally -- Ruby 2.6 system Ruby vs. the
project's required 3.4.9, no way to install without sudo/Docker access).
Every file was checked with `ruby -c`, YAML files with `YAML.load_file`,
and the ERB view with `ERB.new(...).src`, plus careful manual tracing of
each test against the production code paths it exercises, but CI should
be treated as the first real run of this suite per the repository's own
guidance for exactly this situation.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9494Pxh4LZKnNLTGfFXPw
* fix: correct test setup bugs found by actually running the suite
Set up a working Docker-based Rails environment (this session's shell
had no compatible Ruby/Bundler) and ran the full suite against PR #3298.
Two real bugs surfaced that static checks couldn't have caught:
- WorkerAiHealth::Snapshot.new(**{...}.merge(overrides)) needs the
double-splat -- a bare Hash isn't auto-converted to keyword arguments.
Both test snapshot builders passed a positional Hash instead, which
raised "missing keywords" for every field on every call.
- assert_enqueued_with/assert_no_enqueued_jobs need `include
ActiveJob::TestHelper` explicitly in a plain ActiveSupport::TestCase --
every other model test in this codebase that uses them does the same;
I'd wrongly assumed it was available process-wide.
With both fixed: 7510 runs, 29877 assertions, 0 failures, 0 errors, 30
skips for the full suite; rubocop, erb_lint, and brakeman all clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9494Pxh4LZKnNLTGfFXPw
* fix(worker-ai-health): address feedback on health status and cache handling
- Add checks for 'not_configured' and 'unavailable' states in Snapshot#status
- Include PDF probe failure codes in failure_codes detection
- Fix cache lifetime extension by removing expires_in and filtering expired entries in recent()
Ensures unconfigured workers and missing PDF pipelines are marked as failing,
and stale cache entries don't get indefinite TTL refreshes.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9494Pxh4LZKnNLTGfFXPw
* fix(worker-ai-health): fix stub gaps causing ci/test_unit failure
stub_ai_health in WorkerAiHealthCheckJobTest omitted
pdf_text_extraction_probe/pdf_vision_processing_probe, so
WorkerAiHealthCheckJob#failure_codes raised NoMethodError on nil.
Also fix vector_store_status to :missing (no adapter configured) rather
than :not_configured (adapter configured but unusable) to match the
scenario AiHealth actually returns and Snapshot#status's semantics.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9494Pxh4LZKnNLTGfFXPw
* fix(worker-ai-health): compare request timeout, fix raw color class, document dev-mode cache caveat
- Add llm_request_timeout to WorkerAiHealth::Snapshot and compare it in
matches_web? (CodeRabbit) -- a worker with a different effective request
timeout than web (e.g. a workload-specific OPENAI_REQUEST_TIMEOUT override)
previously showed as "Matches web" despite a real configuration
difference, exactly the kind of drift this feature exists to catch.
- Replace border-alpha-black-25 with the border-primary functional token in
the worker result card (CodeRabbit nitpick).
- Document the dev-mode cache_store caveat jjmata flagged: bin/dev runs web
and worker as separate OS processes, and development.rb uses a
process-local memory_store/null_store, so a worker check queued locally
writes to a cache the web process never reads from -- "Verify worker
configuration" can appear to silently do nothing. Added a note to
docs/hosting/ai.md rather than changing behavior, since production's
shared Redis store is unaffected.
- Corrected the retention description in the same doc section (CodeRabbit,
most recent review): only the 5 most recently checked-in distinct
processes are retained (MAX_RESULTS), not "kept for RETENTION" -- a 6th
process checking in can evict an older entry before its own 15-minute
RETENTION window is up.
jjmata's four other findings (unconfigured-worker and missing-PDF-probe
states rendering as "Passing", and the cache-retention/TTL-extension issue)
were already fixed in 1a7b664d, before this pass -- verified against current
code, no changes needed there.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019yyETKmVExx3Q1rYwCynpb
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Jonathan Kaiser <jaysbeekay@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
656 lines
25 KiB
Ruby
656 lines
25 KiB
Ruby
require "test_helper"
|
|
|
|
class Admin::SystemHealthControllerTest < ActionDispatch::IntegrationTest
|
|
AI_ENVIRONMENT = %w[
|
|
OPENAI_ACCESS_TOKEN OPENAI_URI_BASE OPENAI_MODEL OPENAI_REQUEST_TIMEOUT
|
|
OPENAI_SUPPORTS_PDF_PROCESSING OPENAI_SUPPORTS_RESPONSES_ENDPOINT
|
|
ANTHROPIC_ACCESS_TOKEN ANTHROPIC_API_KEY
|
|
ANTHROPIC_BASE_URL ANTHROPIC_MODEL ANTHROPIC_REQUEST_TIMEOUT
|
|
VECTOR_STORE_PROVIDER EMBEDDING_URI_BASE EMBEDDING_MODEL
|
|
EMBEDDING_DIMENSIONS EMBEDDING_ACCESS_TOKEN QDRANT_URL QDRANT_API_KEY
|
|
AI_HEALTH_PROBE_TIMEOUT AI_HEALTH_PROBE_CACHE_TTL
|
|
].index_with(nil).freeze
|
|
|
|
setup do
|
|
Setting.stubs(:llm_provider).returns("openai")
|
|
Setting.stubs(:openai_access_token).returns(nil)
|
|
Setting.stubs(:openai_uri_base).returns(nil)
|
|
Setting.stubs(:openai_model).returns(nil)
|
|
Setting.stubs(:anthropic_access_token).returns(nil)
|
|
Setting.stubs(:anthropic_base_url).returns(nil)
|
|
Setting.stubs(:anthropic_model).returns(nil)
|
|
AiHealth::Probe.any_instance.stubs(:llm).returns(probe_result(:passing))
|
|
AiHealth::Probe.any_instance.stubs(:function_calling).returns(probe_result(:passing))
|
|
AiHealth::Probe.any_instance.stubs(:pdf_text_extraction).returns(probe_result(:passing))
|
|
AiHealth::Probe.any_instance.stubs(:pdf_vision_processing).returns(probe_result(:passing))
|
|
AiHealth::Probe.any_instance.stubs(:openai_vector_store).returns(probe_result(:passing))
|
|
AiHealth::Probe.any_instance.stubs(:pgvector).returns(probe_result(:passing))
|
|
AiHealth::Probe.any_instance.stubs(:embedding).returns(probe_result(:passing))
|
|
end
|
|
|
|
test "super admin can view the system health page" do
|
|
sign_in users(:sure_support_staff)
|
|
SidekiqHealth.any_instance.stubs(:healthy?).returns(true)
|
|
SidekiqHealth.any_instance.stubs(:processes_count).returns(1)
|
|
SidekiqHealth.any_instance.stubs(:last_heartbeat_at).returns(Time.current)
|
|
SidekiqHealth.any_instance.stubs(:max_queue_latency).returns(0.0)
|
|
SidekiqHealth.any_instance.stubs(:enqueued_count).returns(0)
|
|
SidekiqHealth.any_instance.stubs(:retry_count).returns(0)
|
|
SidekiqHealth.any_instance.stubs(:failed_count).returns(0)
|
|
SidekiqHealth.any_instance.stubs(:processed_count).returns(42)
|
|
SidekiqHealth.any_instance.stubs(:queue_breakdown).returns([ [ "default", 0, 0.0 ] ])
|
|
|
|
get admin_system_health_url
|
|
|
|
assert_response :success
|
|
assert_match(/Sidekiq status/, response.body)
|
|
assert_match(/Healthy/, response.body)
|
|
assert_select "button[role='tab']", text: "AI status"
|
|
assert_select "[data-ds--tabs-navigate-on-change-value='true']"
|
|
end
|
|
|
|
test "renders degraded state with reason when Sidekiq is unhealthy" do
|
|
sign_in users(:sure_support_staff)
|
|
SidekiqHealth.any_instance.stubs(:healthy?).returns(false)
|
|
SidekiqHealth.any_instance.stubs(:reason).returns(:no_worker_processes)
|
|
SidekiqHealth.any_instance.stubs(:processes_count).returns(0)
|
|
SidekiqHealth.any_instance.stubs(:last_heartbeat_at).returns(nil)
|
|
SidekiqHealth.any_instance.stubs(:max_queue_latency).returns(0.0)
|
|
SidekiqHealth.any_instance.stubs(:enqueued_count).returns(7)
|
|
SidekiqHealth.any_instance.stubs(:retry_count).returns(0)
|
|
SidekiqHealth.any_instance.stubs(:failed_count).returns(0)
|
|
SidekiqHealth.any_instance.stubs(:processed_count).returns(0)
|
|
SidekiqHealth.any_instance.stubs(:queue_breakdown).returns([])
|
|
|
|
get admin_system_health_url
|
|
|
|
assert_response :success
|
|
assert_match(/Degraded/, response.body)
|
|
assert_match(/No Sidekiq worker process is connected/, response.body)
|
|
end
|
|
|
|
test "non super admin is redirected away" do
|
|
sign_in users(:family_admin)
|
|
|
|
get admin_system_health_url
|
|
|
|
assert_redirected_to root_path
|
|
end
|
|
|
|
test "unauthenticated user is redirected to sign in" do
|
|
get admin_system_health_url
|
|
|
|
assert_redirected_to new_session_path
|
|
end
|
|
|
|
test "AI status reports the default OpenAI LLM and hosted vector store" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
|
|
with_ai_environment("OPENAI_ACCESS_TOKEN" => "sk-secret-openai") do
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
|
|
assert_response :success
|
|
assert_select "button[role='tab'][aria-selected='true']", text: "AI status"
|
|
assert_match(/LLM and PDF processing/, response.body)
|
|
assert_select "[data-testid='selected-llm-provider']", text: "OpenAI"
|
|
assert_select "[data-testid='effective-llm-provider']", text: "OpenAI"
|
|
assert_match(/gpt-4\.1/, response.body)
|
|
assert_match(%r{https://api\.openai\.com/v1}, response.body)
|
|
assert_match(/OpenAI hosted vector store/, response.body)
|
|
assert_match(/Live check passed/, response.body)
|
|
assert_match(/Live checks passed/, response.body)
|
|
assert_match(/PDF text-extraction path/, response.body)
|
|
assert_match(/PDF vision\/native path/, response.body)
|
|
assert_equal 2, response.body.scan(/Synthetic PDF check passed/).size
|
|
assert_no_match(/sk-secret-openai/, response.body)
|
|
end
|
|
|
|
test "background jobs tab does not run AI probes" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
AiHealth::Probe.any_instance.expects(:llm).never
|
|
AiHealth::Probe.any_instance.expects(:function_calling).never
|
|
AiHealth::Probe.any_instance.expects(:pdf_text_extraction).never
|
|
AiHealth::Probe.any_instance.expects(:pdf_vision_processing).never
|
|
AiHealth::Probe.any_instance.expects(:openai_vector_store).never
|
|
|
|
with_ai_environment("OPENAI_ACCESS_TOKEN" => "sk-secret-openai") do
|
|
get admin_system_health_url
|
|
end
|
|
|
|
assert_response :success
|
|
assert_match(/Not checked/, response.body)
|
|
assert_no_match(/sk-secret-openai/, response.body)
|
|
end
|
|
|
|
test "AI status warns when a custom OpenAI endpoint is paired with the hosted vector store" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
|
|
with_ai_environment(
|
|
"OPENAI_ACCESS_TOKEN" => "local-token",
|
|
"OPENAI_URI_BASE" => credentialed_url(
|
|
scheme: "http",
|
|
host: "ollama",
|
|
port: 11_434,
|
|
path: "/v1",
|
|
user: "operator",
|
|
password: "uri-secret",
|
|
query: "api_key=query-secret"
|
|
),
|
|
"OPENAI_MODEL" => "qwen3:8b"
|
|
) do
|
|
AiHealth::Probe.any_instance.stubs(:openai_vector_store).returns(
|
|
probe_result(:failing, failure_code: :request_failed, http_status: 404)
|
|
)
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
|
|
assert_response :success
|
|
assert_select "[data-testid='selected-llm-provider']", text: "OpenAI-compatible"
|
|
assert_select "[data-testid='effective-llm-provider']", text: "Ollama"
|
|
assert_match(/OpenAI-compatible API credentials/, response.body)
|
|
assert_match(%r{http://ollama:11434/v1}, response.body)
|
|
assert_match(%r{did not pass the /v1/vector_stores liveness check}, response.body)
|
|
assert_match(/use pgvector with a separate embeddings endpoint/, response.body)
|
|
assert_match(/Live check failed/, response.body)
|
|
assert_no_match(/local-token|uri-secret|query-secret/, response.body)
|
|
end
|
|
|
|
test "AI status names the missing function-calling support behind an unhelpful chat error" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
AiHealth::Probe.any_instance.stubs(:function_calling).returns(
|
|
probe_result(:failing, failure_code: :tools_refused, http_status: 404)
|
|
)
|
|
|
|
with_ai_environment(
|
|
"OPENAI_ACCESS_TOKEN" => "router-secret",
|
|
"OPENAI_URI_BASE" => "https://openrouter.ai/api/v1",
|
|
"OPENAI_MODEL" => "tngtech/deepseek-r1t2-chimera:free"
|
|
) do
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
|
|
assert_response :success
|
|
assert_select "[data-testid='function-calling-status']", text: /Not supported by the effective provider/
|
|
assert_match(/The model does not support function calling/, response.body)
|
|
assert_match(/Function-calling failure reason/, response.body)
|
|
assert_no_match(/router-secret/, response.body)
|
|
end
|
|
|
|
test "AI status names an unavailable configured LLM model" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
AiHealth::Probe.any_instance.stubs(:llm).returns(
|
|
probe_result(:failing, failure_code: :model_not_available)
|
|
)
|
|
|
|
with_ai_environment(
|
|
"OPENAI_ACCESS_TOKEN" => "gemini-secret",
|
|
"OPENAI_URI_BASE" => "https://generativelanguage.googleapis.com/v1beta/openai",
|
|
"OPENAI_MODEL" => "retired-gemini-model"
|
|
) do
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
|
|
assert_response :success
|
|
assert_match(/The configured AI model is not available/, response.body)
|
|
assert_match(/gemini-2\.5-flash/, response.body)
|
|
assert_no_match(/gemini-secret/, response.body)
|
|
end
|
|
|
|
test "AI status separates a model that ignores tools from one that cannot use them" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
AiHealth::Probe.any_instance.stubs(:function_calling).returns(
|
|
probe_result(:failing, failure_code: :no_tool_call)
|
|
)
|
|
|
|
with_ai_environment("OPENAI_ACCESS_TOKEN" => "sk-secret-openai") do
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
|
|
assert_response :success
|
|
assert_select "[data-testid='function-calling-status']", text: /Tools accepted, but the model called none/
|
|
assert_match(/answered without calling the tool it was asked to call/, response.body)
|
|
assert_no_match(/The model does not support function calling/, response.body)
|
|
end
|
|
|
|
test "AI status reports text and vision PDF probes separately" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
AiHealth::Probe.any_instance.stubs(:pdf_vision_processing).returns(
|
|
probe_result(:failing, failure_code: :invalid_response)
|
|
)
|
|
|
|
with_ai_environment("OPENAI_ACCESS_TOKEN" => "sk-secret-openai") do
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
|
|
assert_response :success
|
|
assert_match(/PDF text-extraction path/, response.body)
|
|
assert_match(/PDF vision\/native path/, response.body)
|
|
assert_match(/The synthetic PDF vision\/native check failed/, response.body)
|
|
assert_match(/Synthetic PDF check passed/, response.body)
|
|
assert_match(/Synthetic PDF check failed/, response.body)
|
|
assert_match(/Vision\/native failure reason/, response.body)
|
|
assert_match(/unexpected response/, response.body)
|
|
assert_no_match(/sk-secret-openai/, response.body)
|
|
end
|
|
|
|
test "AI status surfaces LLM and probe request timeouts as distinct values" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
|
|
# OPENAI_REQUEST_TIMEOUT bounds real LLM calls the app makes (chat, PDF
|
|
# import). AI_HEALTH_PROBE_TIMEOUT only bounds the admin "live checks".
|
|
with_ai_environment(
|
|
"OPENAI_ACCESS_TOKEN" => "local-token",
|
|
"OPENAI_REQUEST_TIMEOUT" => "300",
|
|
"AI_HEALTH_PROBE_TIMEOUT" => "5"
|
|
) do
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
|
|
assert_response :success
|
|
llm_label = response.body.index("LLM request timeout")
|
|
probe_label = response.body.index("Health-check probe timeout")
|
|
assert llm_label, "expected an 'LLM request timeout' label on the AI status page"
|
|
assert probe_label, "expected a 'Health-check probe timeout' label on the AI status page"
|
|
assert_operator llm_label, :<, probe_label, "LLM timeout row should appear before the probe timeout row"
|
|
assert response.body[llm_label, 400].include?("300s"), "LLM timeout value (300s) missing near its label"
|
|
assert response.body[probe_label, 400].include?("5s"), "probe timeout value (5s) missing near its label"
|
|
assert_no_match(/local-token/, response.body)
|
|
end
|
|
|
|
test "AI status does not probe PDF processing when it is explicitly disabled" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
AiHealth::Probe.any_instance.expects(:pdf_text_extraction).never
|
|
AiHealth::Probe.any_instance.expects(:pdf_vision_processing).never
|
|
|
|
with_ai_environment(
|
|
"OPENAI_ACCESS_TOKEN" => "sk-secret-openai",
|
|
"OPENAI_SUPPORTS_PDF_PROCESSING" => "false"
|
|
) do
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
|
|
assert_response :success
|
|
assert_match(/Disabled or not supported by the effective provider\/model/, response.body)
|
|
assert_no_match(/The synthetic PDF .* check failed/, response.body)
|
|
end
|
|
|
|
test "AI status reports Anthropic with an available pgvector store" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
Setting.stubs(:llm_provider).returns("anthropic")
|
|
|
|
connection = stub("connection")
|
|
connection.stubs(:table_exists?).with(VectorStore::Pgvector::TABLE_NAME).returns(true)
|
|
connection.stubs(:extension_enabled?).with("vector").returns(true)
|
|
ActiveRecord::Base.stubs(:connection).returns(connection)
|
|
VectorStore.expects(:embedding_access_token).returns("runtime-embedding-token")
|
|
AiHealth::Probe.any_instance.expects(:embedding).with(
|
|
endpoint: "http://ollama:11434/v1",
|
|
access_token: "runtime-embedding-token",
|
|
model: "mxbai-embed-large",
|
|
dimensions: 1024
|
|
).returns(probe_result(:passing))
|
|
|
|
with_ai_environment(
|
|
"ANTHROPIC_ACCESS_TOKEN" => "anthropic-secret",
|
|
"ANTHROPIC_MODEL" => "claude-sonnet-4-6",
|
|
"EMBEDDING_URI_BASE" => "http://ollama:11434/v1",
|
|
"EMBEDDING_MODEL" => "mxbai-embed-large",
|
|
"EMBEDDING_DIMENSIONS" => "1024"
|
|
) do
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
|
|
assert_response :success
|
|
assert_match(/Anthropic/, response.body)
|
|
assert_match(/pgvector/, response.body)
|
|
assert_match(/PostgreSQL vector extension/, response.body)
|
|
assert_match(/mxbai-embed-large/, response.body)
|
|
assert_match(%r{http://ollama:11434/v1}, response.body)
|
|
assert_match(/Live checks passed/, response.body)
|
|
assert_no_match(/anthropic-secret/, response.body)
|
|
end
|
|
|
|
test "AI status explains a missing pgvector table" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
Setting.stubs(:llm_provider).returns("anthropic")
|
|
VectorStore::Pgvector.stubs(:available?).returns(true)
|
|
AiHealth::Probe.any_instance.stubs(:pgvector).returns(
|
|
probe_result(:failing, failure_code: :table_not_found)
|
|
)
|
|
|
|
connection = stub("connection")
|
|
connection.stubs(:table_exists?).with(VectorStore::Pgvector::TABLE_NAME).returns(false)
|
|
connection.stubs(:extension_enabled?).with("vector").returns(true)
|
|
ActiveRecord::Base.stubs(:connection).returns(connection)
|
|
|
|
with_ai_environment(
|
|
"ANTHROPIC_ACCESS_TOKEN" => "anthropic-secret",
|
|
"ANTHROPIC_MODEL" => "claude-sonnet-4-6",
|
|
"EMBEDDING_URI_BASE" => "http://ollama:11434/v1",
|
|
"EMBEDDING_MODEL" => "mxbai-embed-large",
|
|
"EMBEDDING_DIMENSIONS" => "1024"
|
|
) do
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
|
|
assert_response :success
|
|
assert_match(/The vector_store_chunks table is missing/, response.body)
|
|
assert_match(/VECTOR_STORE_PROVIDER=pgvector/, response.body)
|
|
assert_match(/table was not found/, response.body)
|
|
assert_no_match(/anthropic-secret/, response.body)
|
|
end
|
|
|
|
test "AI status explains an embedding dimensions mismatch" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
Setting.stubs(:llm_provider).returns("anthropic")
|
|
VectorStore::Pgvector.stubs(:available?).returns(true)
|
|
AiHealth::Probe.any_instance.stubs(:embedding).returns(
|
|
probe_result(:failing, failure_code: :dimensions_mismatch)
|
|
)
|
|
|
|
connection = stub("connection")
|
|
connection.stubs(:table_exists?).with(VectorStore::Pgvector::TABLE_NAME).returns(true)
|
|
connection.stubs(:extension_enabled?).with("vector").returns(true)
|
|
ActiveRecord::Base.stubs(:connection).returns(connection)
|
|
|
|
with_ai_environment(
|
|
"ANTHROPIC_ACCESS_TOKEN" => "anthropic-secret",
|
|
"ANTHROPIC_MODEL" => "claude-sonnet-4-6",
|
|
"EMBEDDING_URI_BASE" => "https://generativelanguage.googleapis.com/v1beta/openai",
|
|
"EMBEDDING_MODEL" => "gemini-embedding-2-preview",
|
|
"EMBEDDING_DIMENSIONS" => "1024"
|
|
) do
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
|
|
assert_response :success
|
|
assert_match(/The embedding dimensions do not match/, response.body)
|
|
assert_match(/EMBEDDING_MODEL and EMBEDDING_DIMENSIONS/, response.body)
|
|
assert_match(/alter or recreate the pgvector embedding column\/table/, response.body)
|
|
assert_match(/gemini-embedding-2-preview/, response.body)
|
|
assert_no_match(/anthropic-secret/, response.body)
|
|
end
|
|
|
|
test "AI status explains embedding probe timeouts" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
Setting.stubs(:llm_provider).returns("anthropic")
|
|
VectorStore::Pgvector.stubs(:available?).returns(true)
|
|
AiHealth::Probe.any_instance.stubs(:embedding).returns(
|
|
probe_result(:failing, failure_code: :timeout)
|
|
)
|
|
|
|
connection = stub("connection")
|
|
connection.stubs(:table_exists?).with(VectorStore::Pgvector::TABLE_NAME).returns(true)
|
|
connection.stubs(:extension_enabled?).with("vector").returns(true)
|
|
ActiveRecord::Base.stubs(:connection).returns(connection)
|
|
|
|
with_ai_environment(
|
|
"ANTHROPIC_ACCESS_TOKEN" => "anthropic-secret",
|
|
"ANTHROPIC_MODEL" => "claude-sonnet-4-6",
|
|
"EMBEDDING_URI_BASE" => "https://generativelanguage.googleapis.com/v1beta/openai",
|
|
"EMBEDDING_MODEL" => "gemini-embedding-2-preview",
|
|
"EMBEDDING_DIMENSIONS" => "3072"
|
|
) do
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
|
|
assert_response :success
|
|
assert_match(/The embedding live check timed out/, response.body)
|
|
assert_match(/AI_HEALTH_PROBE_TIMEOUT/, response.body)
|
|
assert_match(/OPENAI_REQUEST_TIMEOUT/, response.body)
|
|
assert_no_match(/anthropic-secret/, response.body)
|
|
end
|
|
|
|
test "AI status explains when no vector store is configured" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
|
|
with_ai_environment do
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
|
|
assert_response :success
|
|
assert_match(/No vector store is configured/, response.body)
|
|
assert_match(/Uploaded documents cannot be indexed or searched/, response.body)
|
|
end
|
|
|
|
test "AI status marks Qdrant as scaffolded and redacts its URL" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
|
|
with_ai_environment(
|
|
"VECTOR_STORE_PROVIDER" => "qdrant",
|
|
"QDRANT_URL" => credentialed_url(
|
|
scheme: "https",
|
|
host: "qdrant.example.test",
|
|
port: 6333,
|
|
user: "admin",
|
|
password: "qdrant-secret",
|
|
query: "api_key=query-secret"
|
|
),
|
|
"QDRANT_API_KEY" => "header-secret"
|
|
) do
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
|
|
assert_response :success
|
|
assert_match(/Qdrant support is not implemented yet/, response.body)
|
|
assert_match(/Scaffolded/, response.body)
|
|
assert_match(%r{https://qdrant\.example\.test:6333}, response.body)
|
|
assert_no_match(/qdrant-secret|query-secret|header-secret/, response.body)
|
|
end
|
|
|
|
test "AI status explains when no worker has checked in yet" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
|
|
with_ai_environment("OPENAI_ACCESS_TOKEN" => "sk-secret-openai") do
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
|
|
assert_response :success
|
|
assert_match(/Worker verification/, response.body)
|
|
assert_match(/No worker has checked in yet/, response.body)
|
|
end
|
|
|
|
test "AI status renders a worker result and flags it matching the web configuration" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
|
|
with_memory_cache do
|
|
with_ai_environment("OPENAI_ACCESS_TOKEN" => "sk-secret-openai") do
|
|
WorkerAiHealth.record!(worker_snapshot(
|
|
process_identity: "worker-1:123",
|
|
effective_provider: :openai,
|
|
llm_model: "gpt-4.1",
|
|
llm_endpoint: "https://api.openai.com/v1",
|
|
vector_store_adapter: :openai
|
|
))
|
|
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
end
|
|
|
|
assert_response :success
|
|
assert_select "[data-testid='worker-process-identity']", text: "worker-1:123"
|
|
assert_match(/Matches web/, response.body)
|
|
assert_no_match(/sk-secret-openai/, response.body)
|
|
end
|
|
|
|
test "AI status flags a worker result whose configuration differs from the web process" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
|
|
with_memory_cache do
|
|
with_ai_environment("OPENAI_ACCESS_TOKEN" => "sk-secret-openai") do
|
|
WorkerAiHealth.record!(worker_snapshot(
|
|
process_identity: "worker-1:123",
|
|
effective_provider: :openai,
|
|
llm_model: "a-different-model-than-web-resolves",
|
|
llm_endpoint: "https://api.openai.com/v1"
|
|
))
|
|
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
end
|
|
|
|
assert_response :success
|
|
assert_match(/Differs from web/, response.body)
|
|
end
|
|
|
|
test "AI status shows a failing worker result with its failure reason" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
|
|
with_memory_cache do
|
|
with_ai_environment("OPENAI_ACCESS_TOKEN" => "sk-secret-openai") do
|
|
WorkerAiHealth.record!(worker_snapshot(
|
|
process_identity: "worker-1:123",
|
|
llm_status: :failing,
|
|
failure_codes: [ :model_not_available ]
|
|
))
|
|
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
end
|
|
|
|
assert_response :success
|
|
assert_match(/The configured model was not returned by the provider/, response.body)
|
|
end
|
|
|
|
test "AI status shows a stale worker result as stale rather than passing" do
|
|
sign_in users(:sure_support_staff)
|
|
stub_healthy_sidekiq
|
|
|
|
with_memory_cache do
|
|
with_ai_environment("OPENAI_ACCESS_TOKEN" => "sk-secret-openai") do
|
|
WorkerAiHealth.record!(worker_snapshot(
|
|
process_identity: "worker-1:123",
|
|
checked_at: (WorkerAiHealth::STALE_AFTER + 1.minute).ago
|
|
))
|
|
|
|
get admin_system_health_url(tab: "ai")
|
|
end
|
|
end
|
|
|
|
assert_response :success
|
|
assert_match(/Stale/, response.body)
|
|
end
|
|
|
|
test "verify_worker_ai queues an asynchronous worker check and redirects to the AI tab" do
|
|
sign_in users(:sure_support_staff)
|
|
|
|
assert_enqueued_with(job: WorkerAiHealthCheckJob) do
|
|
post verify_worker_ai_admin_system_health_url
|
|
end
|
|
|
|
assert_redirected_to admin_system_health_path(tab: "ai")
|
|
follow_redirect!
|
|
assert_match(/Worker check queued/, response.body)
|
|
end
|
|
|
|
test "non super admin cannot queue a worker check" do
|
|
sign_in users(:family_admin)
|
|
|
|
assert_no_enqueued_jobs only: WorkerAiHealthCheckJob do
|
|
post verify_worker_ai_admin_system_health_url
|
|
end
|
|
|
|
assert_redirected_to root_path
|
|
end
|
|
|
|
test "unauthenticated user cannot queue a worker check" do
|
|
assert_no_enqueued_jobs only: WorkerAiHealthCheckJob do
|
|
post verify_worker_ai_admin_system_health_url
|
|
end
|
|
|
|
assert_redirected_to new_session_path
|
|
end
|
|
|
|
private
|
|
# Stubs Rails.cache with an in-process MemoryStore for the duration of
|
|
# the block. Test env normally runs a NullStore (see config/environments/test.rb),
|
|
# under which WorkerAiHealth.record!/.recent (both cache: Rails.cache by
|
|
# default) would silently no-op -- fine for controller tests that don't
|
|
# care about worker results, but these need the round trip to actually work.
|
|
def with_memory_cache
|
|
Rails.stubs(:cache).returns(ActiveSupport::Cache::MemoryStore.new)
|
|
yield
|
|
ensure
|
|
Rails.unstub(:cache)
|
|
end
|
|
|
|
def worker_snapshot(overrides = {})
|
|
WorkerAiHealth::Snapshot.new(
|
|
**{
|
|
process_identity: "worker:1",
|
|
hostname: "worker",
|
|
pid: 1,
|
|
checked_at: Time.current,
|
|
effective_provider: :openai,
|
|
llm_model: "gpt-4.1",
|
|
llm_endpoint: "https://api.openai.com/v1",
|
|
llm_request_timeout: 60,
|
|
function_calling_status: :supported,
|
|
vector_store_adapter: nil,
|
|
embedding_model: nil,
|
|
embedding_endpoint: nil,
|
|
embedding_dimensions: nil,
|
|
llm_status: :passing,
|
|
vector_store_status: :not_configured,
|
|
failure_codes: []
|
|
}.merge(overrides)
|
|
)
|
|
end
|
|
|
|
def credentialed_url(scheme:, host:, port:, user:, password:, path: nil, query: nil)
|
|
URI::Generic.build(
|
|
scheme: scheme,
|
|
userinfo: "#{user}:#{password}",
|
|
host: host,
|
|
port: port,
|
|
path: path,
|
|
query: query
|
|
).to_s
|
|
end
|
|
|
|
def probe_result(status, failure_code: nil, http_status: nil)
|
|
AiHealth::Probe::Result.new(
|
|
status: status,
|
|
checked_at: status.in?([ :passing, :failing ]) ? Time.current : nil,
|
|
failure_code: failure_code,
|
|
http_status: http_status
|
|
)
|
|
end
|
|
|
|
def with_ai_environment(overrides = {}, &block)
|
|
ClimateControl.modify(AI_ENVIRONMENT.merge(overrides), &block)
|
|
end
|
|
|
|
def stub_healthy_sidekiq
|
|
SidekiqHealth.any_instance.stubs(:healthy?).returns(true)
|
|
SidekiqHealth.any_instance.stubs(:processes_count).returns(1)
|
|
SidekiqHealth.any_instance.stubs(:last_heartbeat_at).returns(Time.current)
|
|
SidekiqHealth.any_instance.stubs(:max_queue_latency).returns(0.0)
|
|
SidekiqHealth.any_instance.stubs(:enqueued_count).returns(0)
|
|
SidekiqHealth.any_instance.stubs(:retry_count).returns(0)
|
|
SidekiqHealth.any_instance.stubs(:failed_count).returns(0)
|
|
SidekiqHealth.any_instance.stubs(:processed_count).returns(42)
|
|
SidekiqHealth.any_instance.stubs(:queue_breakdown).returns([ [ "default", 0, 0.0 ] ])
|
|
end
|
|
end
|