* fix(chat): make the assistant response timeout configurable (#2893)
Self-hosted users running a local model report the chat failing with
"assistant not available" after 90 seconds even though the model
generates a reply and tokens are billed.
Three timeouts are involved and only one was configurable:
- OPENAI_REQUEST_TIMEOUT (60s) — already settable, not the blocker.
- The browser watchdog in chat_controller.js (90s) — hardcoded, and
this is what actually fires.
- Chat::UNDELIVERED_RESPONSE_TIMEOUT (60s) — a constant, so raising
the client value alone would not have helped.
The watchdog cannot be avoided by streaming here: custom
OpenAI-compatible providers route through generic_chat_response, which
forces synchronous calls, so nothing renders until the whole generation
finishes. Time-to-last-token has to beat the deadline.
Adds AI_RESPONSE_TIMEOUT (ENV > Setting > 90s default, floored at 30s),
exposed on the Self-Hosting settings page and passed to the Stimulus
controller at all three mount points — show, index and the sidebar in
the application layout, each of which declares data-controller="chat"
independently.
The server floor is derived from the same value but kept 10s below it.
report_timeout answers 200 whether or not it acted and the client only
retries on a non-ok response, so a floor at or above the client value
would let clock skew strand a pending bubble permanently.
Also guards AssistantMessage#append_text!. The watchdog runs in the web
process while the job holds its own copy of the message, so a job
finishing after the bubble was destroyed or demoted would silently
resurrect it alongside the error the user was already shown.
* fix(chat): let the watchdog retry when report_timeout declines
`report_timeout` answered 200 whether or not `handle_undelivered_response!`
acted. The Stimulus watchdog only stops retrying a URL once it sees a 2xx, so
a declined report was treated as final.
That stranded the bubble whenever the client's clock ran more than
SERVER_TIMEOUT_GRACE ahead of the server's: the watchdog posts at its own
timeout, the server sees a message younger than its floor and no-ops, and
nothing ever retries. The bubble spins forever with no error and no Retry.
Answering 409 instead lets the next 5s tick try again, so any amount of skew
costs retries rather than a stuck chat. The grace window stays as an
optimisation to keep those retries rare, not as the correctness mechanism.
* docs(chat): correct the AI_RESPONSE_TIMEOUT ordering guidance
The docs, locale string and examples all said to keep OPENAI_REQUEST_TIMEOUT at
or above AI_RESPONSE_TIMEOUT. That is backwards.
The two limits span different things. OPENAI_REQUEST_TIMEOUT bounds each HTTP
call to the model on its own; AI_RESPONSE_TIMEOUT covers the whole turn and its
clock starts when the message is queued, so it also absorbs Sidekiq queue time
and, for a tool-using turn, two model calls plus the tool run between them.
Keeping the chat timeout the larger of the two means a slow model surfaces the
specific HTTP timeout error rather than a generic "no response", and the job
stops instead of running on after the chat has given up. The shipped 60/90
defaults already had this ordering; only the guidance was wrong.
compose.example.ai.yml gets 300/660 so the Ollama example can actually complete
a tool-using turn.
* fix(chat): claim the pending bubble atomically before appending
append_text! read the row's status and then saved, leaving a window in which
the watchdog could demote the row to `failed` between the two. The late
content would then land on a bubble the user had already been told failed,
flipping it back to `complete`.
Replaces the read with a conditional UPDATE that only succeeds while the row is
still pending, so the check and the state change cannot be separated.
Uses a conditional UPDATE rather than with_lock because append_text! is called
once per chunk on the streaming path; a row lock and transaction per chunk would
be far more expensive. The claim only runs on the first append, since later ones
are no longer pending.
* test(chat): isolate AI_RESPONSE_TIMEOUT from the environment
Chat.response_timeout reads ENV ahead of Setting, so stubbing only the Setting
left the assertions at the mercy of the environment they run in. With
AI_RESPONSE_TIMEOUT=45 exported, five of these tests failed — the default,
floor and grace assertions were all silently measuring the env value.
Adds a with_setting_timeout helper that stubs the Setting and clears the
variable together, and switches the controller tests to stub
Chat.undelivered_response_timeout directly, since what they care about is the
resolved floor rather than how it was configured.
Both files now pass with or without AI_RESPONSE_TIMEOUT set.
* docs(chat): size AI_RESPONSE_TIMEOUT for chained tool calls
The guidance assumed a tool-using turn costs two model calls. #2767 landed
after this branch was opened and made tool calls iterative: `Assistant::Responder`
now loops until `iteration > max_tool_call_iterations`, so a turn runs to
1 + ASSISTANT_MAX_TOOL_CALL_ITERATIONS calls — six by default — with tool
execution in between. At the default 60s per-call timeout that is up to 360s of
model time against a 90s watchdog.
Streaming does not rescue this either. `emit(:output_text)` only fires for a
response that carries text, and tool-call-only rounds carry none, so the bubble
stays on "Thinking…" through every round regardless of provider.
Documents ASSISTANT_MAX_TOOL_CALL_ITERATIONS as the cheaper lever: dropping it to
2 halves the worst case instead of demanding a half-hour timeout, at the cost of
failing long tool chains earlier with a clear limit error. compose.example.ai.yml
now shows that combination rather than a timeout sized for six calls it never had.
* docs(chat): state the whole-turn timeout as a sum, not a maximum
The guidance said to keep AI_RESPONSE_TIMEOUT "above" or "the larger of"
OPENAI_REQUEST_TIMEOUT. That understates it: the watchdog covers the entire turn,
so the bound is
(1 + ASSISTANT_MAX_TOOL_CALL_ITERATIONS) * OPENAI_REQUEST_TIMEOUT
+ tool execution + queue wait
Merely exceeding the per-call limit can still leave the chat reporting failure
while the worker keeps going.
One phrasing was outright wrong: "keep AI_RESPONSE_TIMEOUT the largest of the
three" compared a duration against ASSISTANT_MAX_TOOL_CALL_ITERATIONS, which is a
count, not seconds.
Resizes the examples against the formula — compose.example.ai.yml 1000 -> 1200 and
the Ollama doc example 600 -> 720, both now showing the arithmetic — and states
plainly that the 90s default is sized for typical cloud latency rather than the
worst-case bound, with the formula being what matters once per-call latency
approaches the timeout.
* docs(chat): list the AI settings fields and tag the formula fence
The Settings UI walkthrough listed three of the eight fields on the AI Provider
form. JSON Mode, the three Token Budget fields and the new Chat Response Timeout
were all missing, so the timeout was only discoverable from the troubleshooting
section. Rewrites the list to follow the form's own grouping and uses the labels
the form actually renders.
Also tags the whole-turn formula fence as `text` (markdownlint MD040).
* fix(compose): forward ASSISTANT_MAX_TOOL_CALL_ITERATIONS in standard compose
This file enumerates container environment explicitly — there is no env_file — so
a variable absent from the x-rails-env anchor never reaches web or worker.
The tool-call cap was only named in a comment here, while the docs added in
3360dbf5 tell operators to lower it to keep a turn inside AI_RESPONSE_TIMEOUT.
Following that advice on this compose file silently changed nothing: the app kept
the default of 5 while the timeout was sized for 3 calls, which lands back on the
"no response" error this branch exists to fix.
Left with an empty default so the app's own default governs, matching
OPENAI_MODEL and LLM_CONTEXT_WINDOW above. compose.example.ai.yml already
forwarded it.
* chore(helm): bump pipelock to 2.5.0 and surface 2.5 config
Bumps pipelock.image.tag from 2.2.0 to 2.5.0 and exposes the most
relevant 2.5 features as structured Helm values:
- pipelock.requestBodyScanning: scan outbound bodies and sensitive
headers for prompt-injection and DLP payloads. Disabled by default;
roll out with action=warn before flipping to block.
- pipelock.healthWatchdog: structured config for the wedge-detection
watchdog with an exposeSubsystems toggle for /health detail.
- pipelock.mcpToolPolicy.rules: structured values for rendering
mcp_tool_policy.rules including redirect-profile references.
Also fixes a latent config-validation regression: pipelock 2.x rejects
an enabled mcp_tool_policy with no rules, but the chart previously
defaulted to enabled=true with an empty rules list, which hard-fails
'pipelock check'. The default is now enabled=false; operators must
explicitly enable and provide at least one rule.
Refreshes README, CHANGELOG, docs/hosting/pipelock.md, docs/hosting/ai.md,
compose example pin comment, and pipelock.example.yaml to call out 2.5
highlights (Audit Packet v0 verifiers, SPIFFE-strict envelopes, scanner
attribution on MCP block receipts, pipelock doctor). Also fixes a stale
docs/hosting/mcp.md reference to the removed compose.example.pipelock.yml.
* chore(helm): fail helm template when mcp_tool_policy enabled with no rules
Adds a guard in asserts.tpl so an operator who sets
pipelock.mcpToolPolicy.enabled=true without populating
pipelock.mcpToolPolicy.rules gets a clear render-time error instead
of a container crash-loop with the pipelock validation message.
Per CodeRabbit feedback on #1913.
* Versions
---------
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
- Helm chart default pipelock.image.tag bumped from 2.0.0 to 2.2.0
(three minor releases behind latest)
- README: pipelock CI scan status badge added to the existing badge row
- charts/sure/README.md, docs/hosting/pipelock.md, pipelock.example.yaml:
refreshed feature notes to reference the upstream changelog rather than
pinning to a single version
- compose.example.ai.yml: pin example comment bumped to :2.2.0
- Workflow pin (@v2) unchanged — floating major tag picks up 2.2.x
* Ipv6 support
* Proper fix for containers, dev and local
* Edits similar to non-AI compose file
---------
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
* chore(helm): bump pipelock to v2.0.0 with trusted domains and redirect profiles
- Bump pipelock image tag from 1.5.0 to 2.0.0
- Add first-class Helm values for trustedDomains and mcpToolPolicy.redirectProfiles
- Update CI GitHub Action from @v1 to @v2
- Update compose example, config reference, and docs with v2.0 features
* Releasing this today in `alpha` form
---------
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
* Add conditional migration for vector_store_chunks table
Creates the pgvector-backed chunks table when VECTOR_STORE_PROVIDER=pgvector.
Enables the vector extension, adds store_id/file_id indexes, and uses
vector(1024) column type for embeddings.
* Add VectorStore::Embeddable concern for text extraction and embedding
Shared concern providing extract_text (PDF via pdf-reader, plain-text as-is),
paragraph-boundary chunking (~2000 chars, ~200 overlap), and embed/embed_batch
via OpenAI-compatible /v1/embeddings endpoint using Faraday. Configurable via
EMBEDDING_MODEL, EMBEDDING_URI_BASE, with fallback to OPENAI_* env vars.
* Implement VectorStore::Pgvector adapter with raw SQL
Replaces the stub with a full implementation using
ActiveRecord::Base.connection with parameterized binds. Supports
create_store, delete_store, upload_file (extract+chunk+embed+insert),
remove_file, and cosine-similarity search via the <=> operator.
* Add registry test for pgvector adapter selection
* Configure pgvector in compose.example.ai.yml
Switch db image to pgvector/pgvector:pg16, add VECTOR_STORE_PROVIDER,
EMBEDDING_MODEL, and EMBEDDING_DIMENSIONS env vars, and include
nomic-embed-text in Ollama's pre-loaded models.
* Update pgvector docs from scaffolded to ready
Document env vars, embedding model setup, pgvector Docker image
requirement, and Ollama pull instructions.
* Address PR review feedback
- Migration: remove env guard, use pgvector_available? check so it runs
on plain Postgres (CI) but creates the table on pgvector-capable servers.
Add NOT NULL constraints on content/embedding/metadata, unique index on
(store_id, file_id, chunk_index).
- Pgvector adapter: wrap chunk inserts in a DB transaction to prevent
partial file writes. Override supported_extensions to match formats
that extract_text can actually parse.
- Embeddable: add hard_split fallback for paragraphs exceeding CHUNK_SIZE
to avoid overflowing embedding model token limits.
* Bump schema version to include vector_store_chunks migration
CI uses db:schema:load which checks the version — without this bump,
the migration is detected as pending and tests fail to start.
* Update 20260316120000_create_vector_store_chunks.rb
---------
Co-authored-by: sokiee <sokysrm@gmail.com>
* feat(helm): add Pipelock ConfigMap, scanning config, and consolidate compose
- Add ConfigMap template rendering DLP, response scanning, MCP input/tool
scanning, and forward proxy settings from values
- Mount ConfigMap as /etc/pipelock/pipelock.yaml volume in deployment
- Add checksum/config annotation for automatic pod restart on config change
- Gate HTTPS_PROXY/HTTP_PROXY env injection on forwardProxy.enabled (skip
in MCP-only mode)
- Use hasKey for all boolean values to prevent Helm default swallowing false
- Single source of truth for ports (forwardProxy.port/mcpProxy.port)
- Pipelock-specific imagePullSecrets with fallback to app secrets
- Merge standalone compose.example.pipelock.yml into compose.example.ai.yml
- Add pipelock.example.yaml for Docker Compose users
- Add exclude-paths to CI workflow for locale file false positives
* Add external assistant support (OpenAI-compatible SSE proxy)
Allow self-hosted instances to delegate chat to an external AI agent
via an OpenAI-compatible streaming endpoint. Configurable per-family
through Settings UI or ASSISTANT_TYPE env override.
- Assistant::External::Client: SSE streaming HTTP client (no new gems)
- Settings UI with type selector, env lock indicator, config status
- Helm chart and Docker Compose env var support
- 45 tests covering client, config, routing, controller, integration
* Add session key routing, email allowlist, and config plumbing
Route to the actual OpenClaw session via x-openclaw-session-key header
instead of creating isolated sessions. Gate external assistant access
behind an email allowlist (EXTERNAL_ASSISTANT_ALLOWED_EMAILS env var).
Plumb session_key and allowedEmails through Helm chart, compose, and
env template.
* Add HTTPS_PROXY support to External::Client for Pipelock integration
Net::HTTP does not auto-read HTTPS_PROXY/HTTP_PROXY env vars (unlike
Faraday). Explicitly resolve proxy from environment in build_http so
outbound traffic to the external assistant routes through Pipelock's
forward proxy when enabled. Respects NO_PROXY for internal hosts.
* Add UI fields for external assistant config (Setting-backed with env fallback)
Follow the same pattern as OpenAI settings: database-backed Setting
fields with env var defaults. Self-hosters can now configure the
external assistant URL, token, and agent ID from the browser
(Settings > Self-Hosting > AI Assistant) instead of requiring env vars.
Fields disable when the corresponding env var is set.
* Improve external assistant UI labels and add help text
Change placeholder to generic OpenAI-compatible URL pattern. Add help
text under each field explaining where the values come from: URL from
agent provider, token for authentication, agent ID for multi-agent
routing.
* Add external assistant docs and fix URL help text
Add External AI Assistant section to docs/hosting/ai.md covering setup
(UI and env vars), how it works, Pipelock security scanning, access
control, and Docker Compose example. Drop "chat completions" jargon
from URL help text.
* Harden external assistant: retry logic, disconnect UI, error handling, and test coverage
- Add retry with backoff for transient network errors (no retry after streaming starts)
- Add disconnect button with confirmation modal in self-hosting settings
- Narrow rescue scope with fallback logging for unexpected errors
- Safe cleanup of partial responses on stream interruption
- Gate ai_available? on family assistant_type instead of OR-ing all providers
- Truncate conversation history to last 20 messages
- Proxy-aware HTTP client with NO_PROXY support
- Sanitize protocol to use generic headers (X-Agent-Id, X-Session-Key)
- Full test coverage for streaming, retries, proxy routing, config, and disconnect
* Exclude external assistant client from Pipelock scan-diff
False positive: `@token` instance variable flagged as "Credential in URL".
Temporary workaround until Pipelock supports inline suppression.
* Address review feedback: NO_PROXY boundary fix, SSE done flag, design tokens
- Fix NO_PROXY matching to require domain boundary (exact match or .suffix),
case-insensitive. Prevents badexample.com matching example.com.
- Add done flag to SSE streaming so read_body stops after [DONE]
- Move MAX_CONVERSATION_MESSAGES to class level
- Use bg-success/bg-destructive design tokens for status indicators
- Add rationale comment for pipelock scan exclusion
- Update docs last-updated date
* Address second round of review feedback
- Allowlist email comparison is now case-insensitive and nil-safe
- Cap SSE buffer at 1 MB to prevent memory blowup from malformed streams
- Don't expose upstream HTTP response body in user-facing errors (log it instead)
- Fix frozen string warning on buffer initialization
- Fix "builtin" typo in docs (should be "built-in")
* Protect completed responses from cleanup, sanitize error messages
- Don't destroy a fully streamed assistant message if post-stream
metadata update fails (only cleanup partial responses)
- Log raw connection/HTTP errors internally, show generic messages
to users to avoid leaking network/proxy details
- Update test assertions for new error message wording
* Fix SSE content guard and NO_PROXY test correctness
Use nil check instead of present? for SSE delta content to preserve
whitespace-only chunks (newlines, spaces) that can occur in code output.
Fix NO_PROXY test to use HTTP_PROXY matching the http:// client URL so
the proxy resolution and NO_PROXY bypass logic are actually exercised.
* Forward proxy credentials to Net::HTTP
Pass proxy_uri.user and proxy_uri.password to Net::HTTP.new so
authenticated proxies (http://user:pass@host:port) work correctly.
Without this, credentials parsed from the proxy URL were silently
dropped. Nil values are safe as positional args when no creds exist.
* Update pipelock integration to v0.3.1 with full scanning config
Bump Helm image tag from 0.2.7 to 0.3.1. Add missing security
sections to both the Helm ConfigMap and compose example config:
mcp_tool_policy, mcp_session_binding, and tool_chain_detection.
These protect the /mcp endpoint against tool injection, session
hijacking, and multi-step exfiltration chains.
Add version and mode fields to config files. Enable include_defaults
for DLP and response scanning to merge user patterns with the 35
built-in patterns. Remove redundant --mode CLI flag from the Helm
deployment template since mode is now in the config file.
* feat(helm): add Pipelock ConfigMap, scanning config, and consolidate compose
- Add ConfigMap template rendering DLP, response scanning, MCP input/tool
scanning, and forward proxy settings from values
- Mount ConfigMap as /etc/pipelock/pipelock.yaml volume in deployment
- Add checksum/config annotation for automatic pod restart on config change
- Gate HTTPS_PROXY/HTTP_PROXY env injection on forwardProxy.enabled (skip
in MCP-only mode)
- Use hasKey for all boolean values to prevent Helm default swallowing false
- Single source of truth for ports (forwardProxy.port/mcpProxy.port)
- Pipelock-specific imagePullSecrets with fallback to app secrets
- Merge standalone compose.example.pipelock.yml into compose.example.ai.yml
- Add pipelock.example.yaml for Docker Compose users
- Add exclude-paths to CI workflow for locale file false positives
* Add CHANGELOG entry for Pipelock security proxy integration
* Missed v0.6.8 release
---------
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
* Add backup service to Docker Compose configuration
* Add backup_data volume to Docker Compose examples
* Add backup profiles and update backup volume path in Docker Compose examples
* Linter likes those spaces in brackets
* Switch to `stable` tags for sure
---------
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
Josh