mirror of
https://github.com/we-promise/sure.git
synced 2026-09-04 22:31:07 +00:00
Jonathan KaiserandClaude Sonnet 5
b9b5a7c58b
Closes #3169. The AI status page (#3145, PR #3155) proves only that the `web` process resolved a valid-looking configuration and can reach the configured provider from its own network context. Most AI workloads -- assistant responses, PDF processing, embeddings, auto-categorization, and merchant detection -- actually run in Sidekiq `worker` processes, which can differ from `web` in environment, DNS, proxy rules, network policy, or even loaded credentials (workload-specific overrides, an updated Secret without a pod restart, `web` recreated without `worker`). A passing web check says nothing about whether a worker can do the same. ## What this adds `WorkerAiHealthCheckJob`, queued on demand from a new "Verify worker configuration" button on System health -> AI status. It runs the same bounded, non-destructive probes `AiHealth` already runs, but from inside whichever Sidekiq worker process dequeues it, and records the result via `WorkerAiHealth`: process identity (hostname:pid), checked-at time, a non-secret configuration fingerprint (effective provider, model, redacted endpoint, vector-store adapter/embedding config), and probe outcomes. The AI status tab lists every recorded result -- most recent first, kept for `WorkerAiHealth::RETENTION` (15 minutes) -- each labeled with a status pill (`Passing` / `Failing` / `Stale`, the last once older than `STALE_AFTER`) and a configuration pill comparing it against the web snapshot (`Matches web` / `Differs from web`), with a failure-reason list reusing the existing failure-code translations when a probe failed. ## Implementation constraints from the issue, addressed directly - **Cannot reuse a web-cached probe result, or vice versa.** `AiHealth.new` gained an injectable `probe_cache:` (default `Rails.cache`, matching today's behavior). The worker job passes a fresh `ActiveSupport::Cache::NullStore` instead, so every worker check is a live call that neither reads a web-cached entry nor leaves one behind. - **A single job only verifies one worker.** Documented on the button (`coverage_notice`) and in the docs: with multiple replicas, a passing result names one process, not the fleet. Queuing again samples another. - **Never persists or displays a raw credential.** `WorkerAiHealth::Snapshot` only carries redacted endpoints (AiHealth already redacts these before they reach the job) and provider/model/status fields -- there is no field for a token to occupy. A structural test asserts this stays true. - **Failures land in both places an operator already checks.** Same destinations as `AiHealth::Probe`'s own failures: `Rails.logger` and `DebugLogEntry` (new `ai_health_worker` category), tagged with the process identity. - **Results carry a clear status**, including the `pending` case implicitly (no result yet renders an explanatory empty state) and `stale` for a result whose process may no longer reflect current state. - **DB-backed vs ENV-backed settings are labeled.** A new info block next to the worker results explains which UI settings propagate automatically (rails-settings-cached invalidates the shared cache on write) versus which require restarting/recreating both `web` and `worker`. ## What this deliberately doesn't do Full-fleet coverage (every process publishing a periodic fingerprint) -- the issue lists this under "Other options to consider," not the acceptance criteria, and Sidekiq's normal dispatch doesn't target every process without an explicit per-process coordination mechanism. This PR implements the on-demand, single-check design the acceptance criteria actually describes ("An administrator can request an asynchronous worker-side check", "does not imply full-fleet coverage"); periodic fleet-wide publishing is a natural follow-up if operators need it. ## Testing - `WorkerAiHealth`: recording/reading, same-process replacement vs. cross-process coexistence, MAX_RESULTS bounding, staleness, status derivation (failure codes / component statuses / function-calling refusal), `matches_web?` comparison, and the credential-field structural guard. - `WorkerAiHealthCheckJob`: records a passing/failing snapshot naming this process, writes failures to Rails.logger + DebugLogEntry (and only on failure), never leaks the access token, and -- the defining property -- is proven to construct `AiHealth.new` with an isolated `NullStore` rather than the shared web-facing cache. - `Admin::SystemHealthController`: empty state, a rendered result with matching/mismatched configuration, a failing result's failure reason, a stale result, the `verify_worker_ai` action enqueuing the job and redirecting with a flash notice, and that non-super-admins and unauthenticated requests cannot trigger it. I could not run the Rails test suite in this environment (no working Ruby/Bundler toolchain available locally -- Ruby 2.6 system Ruby vs. the project's required 3.4.9, no way to install without sudo/Docker access). Every file was checked with `ruby -c`, YAML files with `YAML.load_file`, and the ERB view with `ERB.new(...).src`, plus careful manual tracing of each test against the production code paths it exercises, but CI should be treated as the first real run of this suite per the repository's own guidance for exactly this situation. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A9494Pxh4LZKnNLTGfFXPw