fix(chat): surface and recover from undelivered assistant responses (#2436)

* fix(chat): surface and recover from undelivered assistant responses

When the background worker that runs AssistantResponseJob is down or not
polling the high_priority queue, the eager "pending" assistant message had no
safeguard: chat hung on "Thinking…" forever with no error, timeout, or retry,
and the LLM provider was never even called.

Add three layers of resilience:

- Client watchdog (chat_controller.js): a pending "Thinking…" bubble that waits
  past a threshold (default 90s) with no response asks the server to fail it.
  Keyed on the pending marker — which disappears the instant a real response
  streams — so a slow-but-working response is never falsely timed out.

- Server failure capture (Chat#handle_undelivered_response! +
  MessagesController#report_timeout): clears the dead bubble, records a friendly
  "the assistant didn't respond" error with Retry, and writes a DebugLogEntry so
  support can see it in /settings/debug.

- Worker liveness signal (BackgroundJobHealth + warning banner): reads Sidekiq's
  process/queue state directly from Redis in the web process — so a down worker
  is detectable even though the worker is the thing that's broken — and warns
  self-hosted users when no worker polls high_priority or it's badly backed up.
  Fails open so a Sidekiq/Redis blip never blocks chat.

Tests: Chat model (3), MessagesController (2), BackgroundJobHealth (5).

* fix(chat): address review — server-side timeout guard + watchdog retry safety

- Chat#handle_undelivered_response!: gate the state change behind a row lock and
  a server-side minimum age (UNDELIVERED_RESPONSE_TIMEOUT = 60s). The browser
  watchdog is untrusted, so re-read the row under lock and only fail a bubble
  that is still pending AND has genuinely waited past the timeout — never racing
  a worker that is finishing a slow response. (Codex P2 + CodeRabbit)
- chat_controller.js: only mark a report URL as reported on response.ok (fetch
  resolves on HTTP 4xx/5xx, rejecting only on network errors), and guard
  concurrent duplicate POSTs with an in-flight set, so a failed report retries
  instead of stranding the bubble. (CodeRabbit)
- _worker_health_warning: drop the unused `chat:` local + its render arg. (CodeRabbit)

app/javascript/controllers/chat_controller.js

@@ -1,17 +1,31 @@
 import { Controller } from "@hotwired/stimulus";
 
 export default class extends Controller {
-  static targets = ["messages", "form", "input", "submit"];
+  static targets = ["messages", "form", "input", "submit", "pendingResponse"];
+  static values = {
+    // How long a pending "Thinking…" bubble may wait before we assume the
+    // background worker never delivered a response. Generous so slow models or
+    // tool calls don't trip it.
+    responseTimeout: { type: Number, default: 90000 },
+    // How often to re-check pending bubbles.
+    pollInterval: { type: Number, default: 5000 },
+  };
 
   connect() {
+    this.reportedUrls = new Set();
+    this.inFlightUrls = new Set();
     this.#configureAutoScroll();
     this.#updateSubmitState();
+    this.#startUndeliveredWatchdog();
   }
 
   disconnect() {
     if (this.messagesObserver) {
       this.messagesObserver.disconnect();
     }
+    if (this.watchdogTimer) {
+      clearInterval(this.watchdogTimer);
+    }
   }
 
   autoResize() {
@@ -72,4 +86,66 @@ export default class extends Controller {
   #scrollToBottom = () => {
     this.messagesTarget.scrollTop = this.messagesTarget.scrollHeight;
   };
+
+  // Watchdog: a "Thinking…" bubble only resolves when the background worker
+  // streams a response over Turbo. If the worker is down — or the job dies
+  // before it can broadcast an error — the bubble would otherwise spin forever
+  // with no feedback. We detect a pending bubble that has waited past the
+  // threshold and ask the server to mark it failed, so the user gets an error
+  // message + Retry instead of a dead spinner.
+  //
+  // We key off the pending marker itself (it only exists while pending and
+  // disappears the instant a real response renders) rather than a status flag,
+  // so a response that starts streaming can never be falsely timed out.
+  #startUndeliveredWatchdog() {
+    this.#checkUndeliveredResponses();
+    this.watchdogTimer = setInterval(() => {
+      this.#checkUndeliveredResponses();
+    }, this.pollIntervalValue);
+  }
+
+  #checkUndeliveredResponses() {
+    if (!this.hasPendingResponseTarget) return;
+
+    const now = Date.now();
+
+    this.pendingResponseTargets.forEach((el) => {
+      const url = el.dataset.pendingResponseTimeoutUrl;
+      // Skip if already reported (succeeded) or a report is in flight.
+      if (!url || this.reportedUrls.has(url) || this.inFlightUrls.has(url)) return;
+
+      const createdAt = Date.parse(el.dataset.pendingResponseCreatedAt);
+      if (Number.isNaN(createdAt)) return;
+      if (now - createdAt < this.responseTimeoutValue) return;
+
+      this.#reportUndelivered(url);
+    });
+  }
+
+  #reportUndelivered(url) {
+    const token = document.querySelector('meta[name="csrf-token"]')?.content;
+
+    this.inFlightUrls.add(url);
+
+    fetch(url, {
+      method: "POST",
+      headers: {
+        "X-CSRF-Token": token || "",
+        Accept: "text/vnd.turbo-stream.html, text/html",
+      },
+      credentials: "same-origin",
+    })
+      .then((response) => {
+        // Only mark as reported on success. fetch resolves on HTTP 4xx/5xx
+        // (it rejects only on network errors), so without this check a failed
+        // POST would permanently suppress retries and strand the bubble.
+        if (response.ok) this.reportedUrls.add(url);
+      })
+      .catch(() => {
+        // Best-effort. Leave the URL un-reported so the next tick can retry.
+      })
+      .finally(() => {
+        this.inFlightUrls.delete(url);
+      });
+  }
 }