Commit Graph
65 Commits
Author SHA1 Message Date
Andrew B c9fbfd9f71 fix(chat): make the assistant response timeout configurable (#2910)
* fix(chat): make the assistant response timeout configurable (#2893)

Self-hosted users running a local model report the chat failing with
"assistant not available" after 90 seconds even though the model
generates a reply and tokens are billed.

Three timeouts are involved and only one was configurable:

  - OPENAI_REQUEST_TIMEOUT (60s) — already settable, not the blocker.
  - The browser watchdog in chat_controller.js (90s) — hardcoded, and
    this is what actually fires.
  - Chat::UNDELIVERED_RESPONSE_TIMEOUT (60s) — a constant, so raising
    the client value alone would not have helped.

The watchdog cannot be avoided by streaming here: custom
OpenAI-compatible providers route through generic_chat_response, which
forces synchronous calls, so nothing renders until the whole generation
finishes. Time-to-last-token has to beat the deadline.

Adds AI_RESPONSE_TIMEOUT (ENV > Setting > 90s default, floored at 30s),
exposed on the Self-Hosting settings page and passed to the Stimulus
controller at all three mount points — show, index and the sidebar in
the application layout, each of which declares data-controller="chat"
independently.

The server floor is derived from the same value but kept 10s below it.
report_timeout answers 200 whether or not it acted and the client only
retries on a non-ok response, so a floor at or above the client value
would let clock skew strand a pending bubble permanently.

Also guards AssistantMessage#append_text!. The watchdog runs in the web
process while the job holds its own copy of the message, so a job
finishing after the bubble was destroyed or demoted would silently
resurrect it alongside the error the user was already shown.

* fix(chat): let the watchdog retry when report_timeout declines

`report_timeout` answered 200 whether or not `handle_undelivered_response!`
acted. The Stimulus watchdog only stops retrying a URL once it sees a 2xx, so
a declined report was treated as final.

That stranded the bubble whenever the client's clock ran more than
SERVER_TIMEOUT_GRACE ahead of the server's: the watchdog posts at its own
timeout, the server sees a message younger than its floor and no-ops, and
nothing ever retries. The bubble spins forever with no error and no Retry.

Answering 409 instead lets the next 5s tick try again, so any amount of skew
costs retries rather than a stuck chat. The grace window stays as an
optimisation to keep those retries rare, not as the correctness mechanism.

* docs(chat): correct the AI_RESPONSE_TIMEOUT ordering guidance

The docs, locale string and examples all said to keep OPENAI_REQUEST_TIMEOUT at
or above AI_RESPONSE_TIMEOUT. That is backwards.

The two limits span different things. OPENAI_REQUEST_TIMEOUT bounds each HTTP
call to the model on its own; AI_RESPONSE_TIMEOUT covers the whole turn and its
clock starts when the message is queued, so it also absorbs Sidekiq queue time
and, for a tool-using turn, two model calls plus the tool run between them.

Keeping the chat timeout the larger of the two means a slow model surfaces the
specific HTTP timeout error rather than a generic "no response", and the job
stops instead of running on after the chat has given up. The shipped 60/90
defaults already had this ordering; only the guidance was wrong.

compose.example.ai.yml gets 300/660 so the Ollama example can actually complete
a tool-using turn.

* fix(chat): claim the pending bubble atomically before appending

append_text! read the row's status and then saved, leaving a window in which
the watchdog could demote the row to `failed` between the two. The late
content would then land on a bubble the user had already been told failed,
flipping it back to `complete`.

Replaces the read with a conditional UPDATE that only succeeds while the row is
still pending, so the check and the state change cannot be separated.

Uses a conditional UPDATE rather than with_lock because append_text! is called
once per chunk on the streaming path; a row lock and transaction per chunk would
be far more expensive. The claim only runs on the first append, since later ones
are no longer pending.

* test(chat): isolate AI_RESPONSE_TIMEOUT from the environment

Chat.response_timeout reads ENV ahead of Setting, so stubbing only the Setting
left the assertions at the mercy of the environment they run in. With
AI_RESPONSE_TIMEOUT=45 exported, five of these tests failed — the default,
floor and grace assertions were all silently measuring the env value.

Adds a with_setting_timeout helper that stubs the Setting and clears the
variable together, and switches the controller tests to stub
Chat.undelivered_response_timeout directly, since what they care about is the
resolved floor rather than how it was configured.

Both files now pass with or without AI_RESPONSE_TIMEOUT set.

* docs(chat): size AI_RESPONSE_TIMEOUT for chained tool calls

The guidance assumed a tool-using turn costs two model calls. #2767 landed
after this branch was opened and made tool calls iterative: `Assistant::Responder`
now loops until `iteration > max_tool_call_iterations`, so a turn runs to
1 + ASSISTANT_MAX_TOOL_CALL_ITERATIONS calls — six by default — with tool
execution in between. At the default 60s per-call timeout that is up to 360s of
model time against a 90s watchdog.

Streaming does not rescue this either. `emit(:output_text)` only fires for a
response that carries text, and tool-call-only rounds carry none, so the bubble
stays on "Thinking…" through every round regardless of provider.

Documents ASSISTANT_MAX_TOOL_CALL_ITERATIONS as the cheaper lever: dropping it to
2 halves the worst case instead of demanding a half-hour timeout, at the cost of
failing long tool chains earlier with a clear limit error. compose.example.ai.yml
now shows that combination rather than a timeout sized for six calls it never had.

* docs(chat): state the whole-turn timeout as a sum, not a maximum

The guidance said to keep AI_RESPONSE_TIMEOUT "above" or "the larger of"
OPENAI_REQUEST_TIMEOUT. That understates it: the watchdog covers the entire turn,
so the bound is

  (1 + ASSISTANT_MAX_TOOL_CALL_ITERATIONS) * OPENAI_REQUEST_TIMEOUT
    + tool execution + queue wait

Merely exceeding the per-call limit can still leave the chat reporting failure
while the worker keeps going.

One phrasing was outright wrong: "keep AI_RESPONSE_TIMEOUT the largest of the
three" compared a duration against ASSISTANT_MAX_TOOL_CALL_ITERATIONS, which is a
count, not seconds.

Resizes the examples against the formula — compose.example.ai.yml 1000 -> 1200 and
the Ollama doc example 600 -> 720, both now showing the arithmetic — and states
plainly that the 90s default is sized for typical cloud latency rather than the
worst-case bound, with the formula being what matters once per-call latency
approaches the timeout.

* docs(chat): list the AI settings fields and tag the formula fence

The Settings UI walkthrough listed three of the eight fields on the AI Provider
form. JSON Mode, the three Token Budget fields and the new Chat Response Timeout
were all missing, so the timeout was only discoverable from the troubleshooting
section. Rewrites the list to follow the form's own grouping and uses the labels
the form actually renders.

Also tags the whole-turn formula fence as `text` (markdownlint MD040).

* fix(compose): forward ASSISTANT_MAX_TOOL_CALL_ITERATIONS in standard compose

This file enumerates container environment explicitly — there is no env_file — so
a variable absent from the x-rails-env anchor never reaches web or worker.

The tool-call cap was only named in a comment here, while the docs added in
3360dbf5 tell operators to lower it to keep a turn inside AI_RESPONSE_TIMEOUT.
Following that advice on this compose file silently changed nothing: the app kept
the default of 5 while the timeout was sized for 3 calls, which lands back on the
"no response" error this branch exists to fix.

Left with an empty default so the app's own default governs, matching
OPENAI_MODEL and LLM_CONTEXT_WINDOW above. compose.example.ai.yml already
forwarded it.
2026-08-15 06:12:22 +02:00
Guillem Arias Fauste 344cf091e1 feat(auth): sign in with a passkey, without a password (#2911)
* feat(auth): sign in with a passkey, without a password

Passkeys could only ever replace the TOTP code: registration required 2FA
to already be on, and the WebAuthn ceremony was reachable only after
User.authenticate_by had succeeded. A registered passkey can now complete
sign-in on its own, from the login page.

The ceremony requests userVerification: "required", so the authenticator
has to confirm the person as well as the device. That makes a lone passkey
two independent factors, the same bar as the password plus TOTP flow it
replaces, which is why this path deliberately skips the TOTP step. A
credential that can only prove presence is rejected here and still works
as a second factor.

Sign-in is usernameless: no email is submitted, because the browser
returns the account handle with the assertion. Nothing on this path can be
probed to learn whether an account exists. Registration now asks for a
discoverable credential with residentKey: "preferred" so the key is
offered by the picker, while authenticators without a free resident-key
slot still register as a second factor.

Where conditional mediation is available, saved passkeys appear in the
email field's autofill menu; everywhere else the button covers it. The
automatic challenge request that conditional mediation makes on every page
load gets its own looser Rack::Attack budget, so ordinary page views can
no longer exhaust the limit that protects the MFA endpoints.

Set AUTH_PASSKEY_LOGIN_ENABLED=false to keep passkeys as a second factor
only. Passkey sign-in follows the same policy as local login, so it stays
closed to regular users when AUTH_LOCAL_LOGIN_ENABLED is false.

* refactor(auth): group the passkey button with the other sign-in methods

It sat directly under the password fields, so the forgot-password link
split it from the identical SSO buttons. It is an alternative to the
credential form rather than part of it.

* fix(auth): close the passkey challenge races and document the upgrade

Three review passes converged on the conditional-mediation flow. The
AbortController was created after `isConditionalMediationAvailable()`
resolved, so a button click or a Turbo disconnect landing in that window
found nothing to abort: the conditional task carried on, re-minted the
challenge, and the assertion the user was about to produce verified
against a challenge the server had already replaced.

It is created before the first await now, and held in a local, because
`abortConditionalMediation()` nulls the field. Checking that one signal
after each await covers both triggers, so no separate connected flag is
needed.

The same symptom had a second cause nobody flagged: `authenticate()` was
not re-entrant. A double-click minted a fresh challenge under an open
authenticator prompt and rejected a perfectly valid passkey, with no race
window at all — and it was live on the MFA step-up too, which shares the
method.

The conditional catch was silent for every failure, including a rejected
assertion the user had deliberately chosen from the autofill menu.
Splitting the try draws the line where it belongs: silence before the
user has been asked anything, feedback once they have picked a passkey.
Filtering on `error.name` cannot draw it, since `fetchOptions` and
`verifyCredential` both raise a plain Error.

Also documents the upgrade: passwordless is on by default and applies to
already-registered credentials, so a passkey added purely as a second
factor can now sign its owner in alone. Nothing in the schema marks a
credential discoverable — the authenticator decides — and the opt-out is
instance-wide.

The invitation test is a guard, not coverage for this change. The pending
token lives in the Rack session and `complete_sign_in` reads it right
after creating the session, so a `reset_session` dropped in between
strands the invitee in their own family, silently and with every existing
test still green.

* fix(auth): cancel the in-flight conditional options request

Aborting the conditional flow did not cancel its options request, because
`fetchOptions` never received the signal. A click landing while that POST
was in flight left it to finish, and its response could apply last.

The challenge rides in the session cookie, so "the server wrote it" only
counts if the Set-Cookie reaches the browser. Threading the signal means
an aborted request's response is discarded, which closes the window
without needing the server to hold two challenges open.

Also drops the absolute claim about which existing credentials gain
passwordless sign-in. `residentKey: "preferred"` is a request an
authenticator may decline, and nothing records what it decided, so the
honest statement is that password managers and platform authenticators
generally store discoverable credentials rather than always.
2026-08-12 20:35:13 +02:00
efb7cc3935 Tooling for the wealth + tax agent harness (#2848)
* Expose the Statement Vault to external agents over MCP

A user wants to manage patrimonial history — a document-backed record of a
family's wealth where every figure traces back to the statement it came from —
by pointing an external agent harness at Sure. That model belongs in the
harness, not in Sure: it needs numbered build deltas, golden tests and closed
periods that a mutable Postgres row cannot provide.

What Sure was missing was the seam. The Statement Vault already does most of
the work — original bytes retained, SHA-256 dedup, period detection, account
matching with a confidence score, reconciliation against ledger balances, and a
month-by-month coverage map — but it is reachable only from the web UI. An
agent could not archive a document, cite one, or check for gaps.

Adds five preview MCP tools over what already exists, plus a citation grammar
for values the agent writes:

- upload_account_statement, list_account_statements, get_account_statement,
  get_statement_coverage
- record_valuation, whose source citation is parsed rather than trusted:
  ["estimated: "] citation [" (grade: A|B|C)"]. An uncited or free-styled
  value is rejected at the write boundary instead of landing in the ledger
  looking authoritative.

link and reject are deliberately not exposed. Attaching a statement to an
account is the human's decision, and the vault UI is where it is made; the
agent reports the suggested match and stops there.

Assistant.function_classes now takes a user so preview tools stay out of the
default surface. They are hidden from tools/list and not callable by name
without the preference enabled, and the vault tools re-check the manager role
and per-account permissions, since MCP calls never pass through a controller.

Docs: the blueprint this implements, and a guide covering which side owns which
layer, the vocabulary map between the two, the monthly runbook, and the gaps
(non-user holders, non-statement documents, one value per date).

No migrations, no API endpoints, no UI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JFDp9HhXDeswadu4cxFojn

* Address review feedback on the vault MCP tools

Two non-blocking items from the review pass:

Document why get_statement_coverage reads through accessible_by rather than
writable_by. It reports which documents exist and writes nothing, so read
access is the right bar — and tightening it would hide coverage gaps from
people who can already see the figures those gaps sit behind. The comment
exists so a future refactor doesn't "fix" it.

Close the acknowledged verification gap with tests rather than a one-off
manual check. The review noted that nothing proved a real vault payload
serializes cleanly out through tools/call — vault responses are richer than
the other tools' output, with nested account hashes, decimal balances, dates
and a compacted hash. Two integration tests now drive the real /mcp endpoint
end to end against a real AccountStatement: one listing it, one uploading
bytes and reading back the SHA-256. Permanent regression coverage instead of
a smoke test someone has to remember to repeat.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JFDp9HhXDeswadu4cxFojn

* docs(llm-guides): replace patrimonial blueprint with its final revision

Swap the embedded early draft for the authoritative final revision of the
wealth + tax modelling blueprint (MIT © 2026 diegomarino):

- rename the domain vocabulary: patrimonial -> wealth, fiscal -> tax
  (tax_data/, the tax layer, tax_runner)
- add §9.5 (the intel file: shape, generation, and the capture loop)
- tighten worked examples down to placeholders
- add the MIT header; keep the in-repo NOTE block (adapted to the new
  vocabulary) and the filename untouched so cross-links don't break

* docs(llm-guides): align agent-harness guide with blueprint + fix reconcile semantics

Follow the blueprint rename (patrimonial -> wealth, fiscal -> tax,
fiscal_data/ -> tax_data/, "Phase 7 (fiscal layer)" -> "(the tax layer)")
so the two docs stop disagreeing on vocabulary.

Correct the reconciliation mapping, which conflated two different invariants:

- blueprint reconcile-or-abort (§7 pass 3) is parse-integrity (parsed parts
  == the document's own printed total); Sure's reconciliation_checks is
  ledger agreement (statement balances vs the ledger). Sure has no
  parse-integrity check and never aborts.
- opening_balance / closing_balance are user-entered, not auto-extracted, so
  over MCP reconciliation is "unavailable" until a human fills them.
- tolerance differs: blueprint 1.00/account-period vs Sure's fixed 0.01.

State in the ownership table, the invariants section, the vocabulary map and
the monthly runbook that parse-integrity and the abort belong to the harness
extractor.

* Correct the vault tools' reconciliation claims and citation parsing

Review findings from @diegomarino, all verified against the code before
changing anything.

The reconciliation claim was the serious one. get_account_statement told
agents the checks were "the trustworthy part" and returned "the balances read
off it" — but nothing reads balances off a document. MetadataDetector never
touches them and create_from_prepared_upload! never sets them; they are
user-editable fields in the Statement Vault UI. So a statement archived over
MCP always came back with an empty check list, which an agent could easily
read as "the document agrees with the ledger" when it means "nobody has
entered the figures". The description now says so, and the payload carries a
reconciliation_note spelling it out for anything reading only the JSON. Also
noted that these checks are ledger agreement, not parse integrity: nothing
here verifies a document's parts sum to its printed total.

Provenance::Citation had two patterns disagreeing about spacing. GRADE_SUFFIX
allowed "(grade:A)" but FORMAT required exactly one space, so that citation
passed the pre-check and then parsed as ungraded with the grade swallowed into
the text — silently discarding the reliability the caller supplied, which is
the one thing this parser exists to prevent.

list_account_statements downcases content_sha256 before querying. The column
is constrained to lowercase hex, so uppercase input could never match, and an
agent would read the empty result as "not archived" and upload a duplicate.
Its period filters are renamed overlapping_from / overlapping_until, since
they match on overlap and the old names claimed otherwise to anyone reading
the schema without the descriptions. has_more now explains that there is no
cursor and the way forward is a bigger limit or narrower filters.

record_valuation no longer overwrites the entry's notes. Re-recording a date
would destroy a note a person had written there. Nothing is removed now: an
identical citation is a no-op, a changed one is appended, and the trail of
what was cited when survives. Detecting "did this tool write that line?" is
not possible — almost any prose parses as a valid ungraded citation — so the
code does not guess.

Minor: accept urlsafe base64 on upload, and explain in the code why
record_valuation checks the account ACL rather than the vault manager role, so
nobody "tightens" it into the wrong permission later.

Tests cover each: the grade-spacing cases both ways, uppercase SHA lookup,
overlap window boundaries, note preservation and no-stacking, the unavailable
reconciliation note appearing and disappearing, and — per the review — that
the download URL's signed id actually expires, rather than trusting the
description's claim.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JFDp9HhXDeswadu4cxFojn

* Repair a bad merge in the MCP controller test

The merge of main spliced the incoming `tools/call executes
update_transaction` test into the middle of the upload round-trip test,
before its closing `end`. That left the file one `end` short, so it did
not parse — taking out both `ci / lint` (Lint/Syntax) and `ci / test_unit`
(the whole file failed to load).

Restores the missing `end`. Both tests are kept as their authors wrote
them; nothing else changes.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JFDp9HhXDeswadu4cxFojn

* docs: use wealth history wording (#2885)

* Stop the vault tools promising verification they don't perform

Three findings from the automated review passes, all confirmed against the
code before changing anything.

The download URL was dead on arrival for the caller it was built for. Sure
serves stored files through Active Storage controllers that
config/initializers/active_storage_authorization.rb gates on
`viewable_by?(Current.user)` — a signed-in browser session. An MCP client has
a bearer token and no session, so following the URL would have redirected to
sign-in. Removed it rather than leaving a link that cannot work, and the
description now points at search_family_files or the vault UI.

Coverage called a month `covered` when a document merely existed. An
unreconciled statement is not mismatched, so it took the `covered` branch, and
the payload carried nothing to correct the reading — the same "advertised
verification that never happened" bug fixed last round in
get_account_statement, in a second place. Months now carry their own
reconciliation_status, and the description says covered means presence, not
agreement.

Listing filtered visibility after limiting. Beyond underfilling a page, with
no cursor and a 100-row cap an accessible statement behind enough newer
invisible ones was unreachable. Visibility now lives in the query, mirroring
viewable_by? for a statement manager.

Also: rescue unexpected upload failures into a tool error instead of a raw
exception string, derive the documented size limit from MAX_FILE_SIZE, list
every coverage status in mcp.md, and cover the failed-reconciliation and
base64-normalisation branches.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JFDp9HhXDeswadu4cxFojn

* Keep storage exception detail out of the MCP response

The upload_failed message interpolated the exception text, which crosses out
to an external agent. A storage failure can carry bucket names, object keys,
paths or request details, so the agent now gets a fixed message and the
exception stays in the server log. The test asserts the absence of detail
rather than pinning the leaked string into the contract.

Also fixes a test that did not test what it claimed: the urlsafe-base64 case
used a fixture encoding to plain base64, so it exercised the padding branch
and never the "-_" translation. It now uses content whose encoding contains
both characters and asserts that up front.

Renames "rejects content that decodes to zero bytes" to "rejects blank
content", which is what it actually covers — Base64.strict_encode64("") is
"", which is blank and returns before the decoder runs, so invalid_content is
correct and empty_file is not reachable from this path.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JFDp9HhXDeswadu4cxFojn

* docs: align wealth blueprint review feedback

* Correct the harness runbook: parse before publishing

The guide told an implementer to archive each document to Sure first and
work from there. That strands them. Sure never returns a document's bytes
over MCP — Active Storage serves stored files only to a signed-in browser
session — and there is no text fallback either, because statements archived
through upload_account_statement never enter the vector store, so
search_family_files cannot see them. A statement in Sure is metadata to an
agent and nothing more.

That blocks exactly three blueprint steps, all of them operating on bank and
broker statements: the extractors, the parts-vs-printed-total check, and the
glyph decoder. Everything else it parses — tax returns, capital accounts,
annual accounts — the harness already holds locally.

So the order inverts: the harness ingests into its own vault, extracts there
with the whole file in reach, and publishes to Sure afterwards. This restores
principle 8 rather than bending it — the recurring pipeline reads from the
canonical store, and treating Sure as canonical forced a re-fetch the
architecture never sanctioned. Both sides hash the same bytes, so the SHA-256
verifies Sure holds the identical document without moving it.

Writes down the two consequences: a statement uploaded straight into Sure's
UI can be known but never parsed (reliability C or PENDING until a copy
reaches the harness), and neither vault backs up the other.

Also drops a stale tools-table row still advertising the 15-minute download
URL removed earlier, corrects get_account_statement's description where it
suggested search_family_files as a fallback it cannot be, and disambiguates
"the vault" in the MCP tool table, which is what misled me in the first place.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JFDp9HhXDeswadu4cxFojn

---------

Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: diegomarino <diegomarino@users.noreply.github.com>
Co-authored-by: Sure Admin (bot) <sure-admin@splashblot.com>
2026-08-04 23:33:01 +02:00
AtlasandJuan José Mata 53a87a7645 fix(mcp): assign OAuth clients read_write scope (#2884)
* fix(mcp): assign OAuth clients read_write scope

* fix(mcp): default registered OAuth scope

---------

Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2026-08-04 23:10:39 +02:00
Oscar 0fa1fbd650 add redbark setup guide to hosting docs (#2817)
- new docs/hosting/redbark.md covering account setup, api keys, linking and sync behaviour
- listed redbark in the onboarding guide's provider integrations
2026-07-28 06:42:57 +02:00
Juan José Mata c6eb7cdeed Revert "Refactor application workflows and update test coverage"
This reverts commit 565e049f89.
2026-07-23 13:39:46 -07:00
Juan José Mata 565e049f89 Refactor application workflows and update test coverage 2026-07-21 22:01:59 -07:00
Sure Admin (bot)coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>Juan José Mata
f62c805c69 docs: clarify local LLM context window tuning (#2661)
* docs(ai): document local LLM context window tuning

* Update compose.example.yml

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: Juan José Mata <jjmata@jjmata.com>

---------

Signed-off-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2026-07-17 07:10:29 +02:00
Guillem Arias Fauste e699f5272b feat(sidekiq): mount Web UI in production behind super-admin sessions (#2683)
/sidekiq was mounted `unless Rails.env.production?`, and the Docker
image bakes RAILS_ENV=production — so no self-hosted or managed
deployment has ever had queue tooling, while the production basic-auth
block (with default "sure"/"sure" credentials) was dead code. Worse,
any custom non-production env (e.g. staging) got the dashboard with no
authentication at all.

Now:

- Development keeps the open mount for convenience.
- Everywhere else the route only exists for a signed-in super admin,
  via a routing constraint that resolves the signed session cookie the
  same way Authentication#find_session_by_cookie does. The session's
  user is always the true user (impersonation is resolved at the
  Current level and impersonating a super admin is forbidden), and
  sessions are only created after MFA verification, so neither can
  bypass it. Fails closed — errors mean 404.
- The "sure"/"sure" default credentials are deleted. Basic auth now
  activates only when BOTH SIDEKIQ_WEB_USERNAME and
  SIDEKIQ_WEB_PASSWORD are explicitly set, as an optional second
  layer on top of the constraint.
- Documented in .env.example and docs/hosting/docker.md, including
  warnings that the dashboard is break-glass tooling: never manually
  retry SimplefinConnectionUpdateJob (single-use token), and deleting
  jobs does not update the corresponding Sure records.

The super-admin bar's Jobs link is feature-detected via
sidekiq_web_available? and lights up automatically now that the route
exists in production.
2026-07-17 07:00:27 +02:00
Sure Admin (bot) 9f253f21df docs: explain self-hosted onboarding modes (#2533) 2026-06-30 21:09:25 +02:00
Josh 60d9a70aff Refresh Pipelock integration for v2.8 receipts (#2406)
* chore(pipelock): refresh integration for v2.8 receipts

* Clarify Pipelock receipt key mounts
2026-06-19 17:16:38 +02:00
JoshandJuan José Mata ca895416a4 chore(helm): bump pipelock to 2.5.0 and surface 2.5 config (#1913)
* chore(helm): bump pipelock to 2.5.0 and surface 2.5 config

Bumps pipelock.image.tag from 2.2.0 to 2.5.0 and exposes the most
relevant 2.5 features as structured Helm values:

- pipelock.requestBodyScanning: scan outbound bodies and sensitive
  headers for prompt-injection and DLP payloads. Disabled by default;
  roll out with action=warn before flipping to block.
- pipelock.healthWatchdog: structured config for the wedge-detection
  watchdog with an exposeSubsystems toggle for /health detail.
- pipelock.mcpToolPolicy.rules: structured values for rendering
  mcp_tool_policy.rules including redirect-profile references.

Also fixes a latent config-validation regression: pipelock 2.x rejects
an enabled mcp_tool_policy with no rules, but the chart previously
defaulted to enabled=true with an empty rules list, which hard-fails
'pipelock check'. The default is now enabled=false; operators must
explicitly enable and provide at least one rule.

Refreshes README, CHANGELOG, docs/hosting/pipelock.md, docs/hosting/ai.md,
compose example pin comment, and pipelock.example.yaml to call out 2.5
highlights (Audit Packet v0 verifiers, SPIFFE-strict envelopes, scanner
attribution on MCP block receipts, pipelock doctor). Also fixes a stale
docs/hosting/mcp.md reference to the removed compose.example.pipelock.yml.

* chore(helm): fail helm template when mcp_tool_policy enabled with no rules

Adds a guard in asserts.tpl so an operator who sets
pipelock.mcpToolPolicy.enabled=true without populating
pipelock.mcpToolPolicy.rules gets a clear render-time error instead
of a container crash-loop with the pipelock validation message.

Per CodeRabbit feedback on #1913.

* Versions

---------

Co-authored-by: Juan José Mata <jjmata@jjmata.com>
2026-05-24 13:50:44 +02:00
ghost 911aa34ba9 feat(auth): add WebAuthn MFA credentials (#1628)
* feat(auth): add WebAuthn MFA credentials

* fix(auth): harden WebAuthn MFA review paths

* fix(auth): polish WebAuthn error handling

* fix(auth): handle duplicate WebAuthn credential races

* fix(auth): permit WebAuthn credential params

* fix(auth): trim WebAuthn registration controller cleanup

* fix(auth): tighten WebAuthn MFA handling

* fix(auth): pin WebAuthn relying party config
2026-05-03 22:13:28 +02:00
LPW b457514c31 chore(pipelock): bump chart default to v2.2.0, add CI scan badge (#1494)
- Helm chart default pipelock.image.tag bumped from 2.0.0 to 2.2.0
  (three minor releases behind latest)
- README: pipelock CI scan status badge added to the existing badge row
- charts/sure/README.md, docs/hosting/pipelock.md, pipelock.example.yaml:
  refreshed feature notes to reference the upstream changelog rather than
  pinning to a single version
- compose.example.ai.yml: pin example comment bumped to :2.2.0
- Workflow pin (@v2) unchanged — floating major tag picks up 2.2.x
2026-04-18 09:32:23 +02:00
soky srmandJuan José Mata 90b1308866 Ipv6 support (#1437)
* Ipv6 support

* Proper fix for containers, dev and local

* Edits similar to non-AI compose file

---------

Co-authored-by: Juan José Mata <jjmata@jjmata.com>
2026-04-13 13:44:37 +02:00
Copilotjjmatacopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
1527611239 Default production SSO provider source to YAML to avoid boot-time schema errors (#1278)
* Initial plan

* Default production SSO provider source to YAML

Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com>
Agent-Logs-Url: https://github.com/we-promise/sure/sessions/d3a36ca8-e936-4687-a466-9b4c93c19150

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com>
2026-03-25 15:08:36 +01:00
LPWandJuan José Mata 1ddc427fd5 chore(helm): bump pipelock to v2.0.0 with trusted domains and redirect profiles (#1266)
* chore(helm): bump pipelock to v2.0.0 with trusted domains and redirect profiles

- Bump pipelock image tag from 1.5.0 to 2.0.0
- Add first-class Helm values for trustedDomains and mcpToolPolicy.redirectProfiles
- Update CI GitHub Action from @v1 to @v2
- Update compose example, config reference, and docs with v2.0 features

* Releasing this today in `alpha` form

---------

Co-authored-by: Juan José Mata <jjmata@jjmata.com>
2026-03-24 09:30:54 +01:00
LPWandJuan José Mata e43a8f295d Bump pipelock image from 0.3.2 to 1.5.0 (#1238)
* Bump pipelock image from 0.3.2 to 1.5.0

* Releasing via `alpha`

---------

Co-authored-by: Juan José Mata <jjmata@jjmata.com>
2026-03-21 15:32:07 +01:00
Dreamandsokiee 6d22514c01 feat(vector-store): Implement pgvector adapter for self-hosted RAG (#1211)
* Add conditional migration for vector_store_chunks table

Creates the pgvector-backed chunks table when VECTOR_STORE_PROVIDER=pgvector.
Enables the vector extension, adds store_id/file_id indexes, and uses
vector(1024) column type for embeddings.

* Add VectorStore::Embeddable concern for text extraction and embedding

Shared concern providing extract_text (PDF via pdf-reader, plain-text as-is),
paragraph-boundary chunking (~2000 chars, ~200 overlap), and embed/embed_batch
via OpenAI-compatible /v1/embeddings endpoint using Faraday. Configurable via
EMBEDDING_MODEL, EMBEDDING_URI_BASE, with fallback to OPENAI_* env vars.

* Implement VectorStore::Pgvector adapter with raw SQL

Replaces the stub with a full implementation using
ActiveRecord::Base.connection with parameterized binds. Supports
create_store, delete_store, upload_file (extract+chunk+embed+insert),
remove_file, and cosine-similarity search via the <=> operator.

* Add registry test for pgvector adapter selection

* Configure pgvector in compose.example.ai.yml

Switch db image to pgvector/pgvector:pg16, add VECTOR_STORE_PROVIDER,
EMBEDDING_MODEL, and EMBEDDING_DIMENSIONS env vars, and include
nomic-embed-text in Ollama's pre-loaded models.

* Update pgvector docs from scaffolded to ready

Document env vars, embedding model setup, pgvector Docker image
requirement, and Ollama pull instructions.

* Address PR review feedback

- Migration: remove env guard, use pgvector_available? check so it runs
  on plain Postgres (CI) but creates the table on pgvector-capable servers.
  Add NOT NULL constraints on content/embedding/metadata, unique index on
  (store_id, file_id, chunk_index).
- Pgvector adapter: wrap chunk inserts in a DB transaction to prevent
  partial file writes. Override supported_extensions to match formats
  that extract_text can actually parse.
- Embeddable: add hard_split fallback for paragraphs exceeding CHUNK_SIZE
  to avoid overflowing embedding model token limits.

* Bump schema version to include vector_store_chunks migration

CI uses db:schema:load which checks the version — without this bump,
the migration is detected as pending and tests fail to start.

* Update 20260316120000_create_vector_store_chunks.rb

---------

Co-authored-by: sokiee <sokysrm@gmail.com>
2026-03-20 17:01:31 +01:00
Andrei Onelaskmanu[bot] <192355599+askmanu[bot]@users.noreply.github.com>Juan José Mata
a0b1029ba9 Documentation for review AI Assistant features, MCP and API additions (#1168)
* Create MCP server endpoint documentation

* Add Assistant Architecture section to AI documentation

* Add Users API documentation for account reset and delete endpoints

* Document Pipelock CI security scanning in contributing guide

* fix: correct scope and error codes in Users API documentation

* Exclude `docs/hosting/ai.md` from Pipelock scan

---------

Co-authored-by: askmanu[bot] <192355599+askmanu[bot]@users.noreply.github.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
2026-03-16 18:24:28 +01:00
LPW ca8f04040f Expand AI docs: external assistant, MCP, architecture, troubleshooting (#1115)
* Expand AI docs: architecture, MCP, external assistant setup, troubleshooting

- Add architecture overview explaining two independent AI pipelines
  (chat assistant vs auto-categorization)
- Document MCP callback endpoint (JSON-RPC 2.0, auth, available tools)
- Add OpenClaw gateway configuration example
- Add Kubernetes network policy guidance (targetPort vs servicePort)
- Add Pipelock notes (mcpToolPolicy, NO_PROXY behavior)
- Add troubleshooting for "Failed to generate response" with external assistant
- Fix stale function list (4 tools -> 7)
- Fix incorrect env-vs-UI precedence statement
- Fix em-dashes in existing content

* Fix troubleshooting curl to use pod env vars

Use sh -c so $EXTERNAL_ASSISTANT_TOKEN and $EXTERNAL_ASSISTANT_URL
expand inside the pod, not on the local shell.
2026-03-04 11:26:43 +01:00
a53a131c46 Add Pipelock operational templates, docs, and config hardening (#1102)
* feat(helm): add Pipelock ConfigMap, scanning config, and consolidate compose

- Add ConfigMap template rendering DLP, response scanning, MCP input/tool
  scanning, and forward proxy settings from values
- Mount ConfigMap as /etc/pipelock/pipelock.yaml volume in deployment
- Add checksum/config annotation for automatic pod restart on config change
- Gate HTTPS_PROXY/HTTP_PROXY env injection on forwardProxy.enabled (skip
  in MCP-only mode)
- Use hasKey for all boolean values to prevent Helm default swallowing false
- Single source of truth for ports (forwardProxy.port/mcpProxy.port)
- Pipelock-specific imagePullSecrets with fallback to app secrets
- Merge standalone compose.example.pipelock.yml into compose.example.ai.yml
- Add pipelock.example.yaml for Docker Compose users
- Add exclude-paths to CI workflow for locale file false positives

* Add external assistant support (OpenAI-compatible SSE proxy)

Allow self-hosted instances to delegate chat to an external AI agent
via an OpenAI-compatible streaming endpoint. Configurable per-family
through Settings UI or ASSISTANT_TYPE env override.

- Assistant::External::Client: SSE streaming HTTP client (no new gems)
- Settings UI with type selector, env lock indicator, config status
- Helm chart and Docker Compose env var support
- 45 tests covering client, config, routing, controller, integration

* Add session key routing, email allowlist, and config plumbing

Route to the actual OpenClaw session via x-openclaw-session-key header
instead of creating isolated sessions. Gate external assistant access
behind an email allowlist (EXTERNAL_ASSISTANT_ALLOWED_EMAILS env var).
Plumb session_key and allowedEmails through Helm chart, compose, and
env template.

* Add HTTPS_PROXY support to External::Client for Pipelock integration

Net::HTTP does not auto-read HTTPS_PROXY/HTTP_PROXY env vars (unlike
Faraday). Explicitly resolve proxy from environment in build_http so
outbound traffic to the external assistant routes through Pipelock's
forward proxy when enabled. Respects NO_PROXY for internal hosts.

* Add UI fields for external assistant config (Setting-backed with env fallback)

Follow the same pattern as OpenAI settings: database-backed Setting
fields with env var defaults. Self-hosters can now configure the
external assistant URL, token, and agent ID from the browser
(Settings > Self-Hosting > AI Assistant) instead of requiring env vars.
Fields disable when the corresponding env var is set.

* Improve external assistant UI labels and add help text

Change placeholder to generic OpenAI-compatible URL pattern. Add help
text under each field explaining where the values come from: URL from
agent provider, token for authentication, agent ID for multi-agent
routing.

* Add external assistant docs and fix URL help text

Add External AI Assistant section to docs/hosting/ai.md covering setup
(UI and env vars), how it works, Pipelock security scanning, access
control, and Docker Compose example. Drop "chat completions" jargon
from URL help text.

* Harden external assistant: retry logic, disconnect UI, error handling, and test coverage

- Add retry with backoff for transient network errors (no retry after streaming starts)
- Add disconnect button with confirmation modal in self-hosting settings
- Narrow rescue scope with fallback logging for unexpected errors
- Safe cleanup of partial responses on stream interruption
- Gate ai_available? on family assistant_type instead of OR-ing all providers
- Truncate conversation history to last 20 messages
- Proxy-aware HTTP client with NO_PROXY support
- Sanitize protocol to use generic headers (X-Agent-Id, X-Session-Key)
- Full test coverage for streaming, retries, proxy routing, config, and disconnect

* Exclude external assistant client from Pipelock scan-diff

False positive: `@token` instance variable flagged as "Credential in URL".
Temporary workaround until Pipelock supports inline suppression.

* Address review feedback: NO_PROXY boundary fix, SSE done flag, design tokens

- Fix NO_PROXY matching to require domain boundary (exact match or .suffix),
  case-insensitive. Prevents badexample.com matching example.com.
- Add done flag to SSE streaming so read_body stops after [DONE]
- Move MAX_CONVERSATION_MESSAGES to class level
- Use bg-success/bg-destructive design tokens for status indicators
- Add rationale comment for pipelock scan exclusion
- Update docs last-updated date

* Address second round of review feedback

- Allowlist email comparison is now case-insensitive and nil-safe
- Cap SSE buffer at 1 MB to prevent memory blowup from malformed streams
- Don't expose upstream HTTP response body in user-facing errors (log it instead)
- Fix frozen string warning on buffer initialization
- Fix "builtin" typo in docs (should be "built-in")

* Protect completed responses from cleanup, sanitize error messages

- Don't destroy a fully streamed assistant message if post-stream
  metadata update fails (only cleanup partial responses)
- Log raw connection/HTTP errors internally, show generic messages
  to users to avoid leaking network/proxy details
- Update test assertions for new error message wording

* Fix SSE content guard and NO_PROXY test correctness

Use nil check instead of present? for SSE delta content to preserve
whitespace-only chunks (newlines, spaces) that can occur in code output.

Fix NO_PROXY test to use HTTP_PROXY matching the http:// client URL so
the proxy resolution and NO_PROXY bypass logic are actually exercised.

* Forward proxy credentials to Net::HTTP

Pass proxy_uri.user and proxy_uri.password to Net::HTTP.new so
authenticated proxies (http://user:pass@host:port) work correctly.
Without this, credentials parsed from the proxy URL were silently
dropped. Nil values are safe as positional args when no creds exist.

* Update pipelock integration to v0.3.1 with full scanning config

Bump Helm image tag from 0.2.7 to 0.3.1. Add missing security
sections to both the Helm ConfigMap and compose example config:
mcp_tool_policy, mcp_session_binding, and tool_chain_detection.
These protect the /mcp endpoint against tool injection, session
hijacking, and multi-step exfiltration chains.

Add version and mode fields to config files. Enable include_defaults
for DLP and response scanning to merge user patterns with the 35
built-in patterns. Remove redundant --mode CLI flag from the Helm
deployment template since mode is now in the config file.

* Pipelock Helm hardening + docs for external assistant and pipelock

Helm templates:
- ServiceMonitor for Prometheus scraping on /metrics (proxy port)
- Ingress template for MCP reverse proxy (external AI agent access)
- PodDisruptionBudget with minAvailable/maxUnavailable mutual exclusion
- topologySpreadConstraints on Deployment
- Structured logging config (format, output, include_allowed/blocked)
- extraConfig escape hatch for additional pipelock.yaml sections
- requireForExternalAssistant guard (fails when assistant enabled without pipelock)
- Component label on Service metadata for ServiceMonitor targeting
- NOTES.txt pipelock section with health, access, security, metrics info
- Bump pipelock image tag 0.3.1 -> 0.3.2
- Fix: rename _asserts.tpl -> asserts.tpl (Helm skipped _ prefixed file)

Documentation:
- Helm chart README: full Pipelock section
- docs/hosting/pipelock.md: dedicated hosting guide (Docker + Kubernetes)
- docs/hosting/docker.md: AI features section (external assistant, pipelock)
- .env.example: external assistant and MCP env vars

Infra:
- Chart.lock pinning dependency versions
- .gitignore for vendored subchart tarballs

* Fix bot comments: quote ingress host, fix sidecar wording, add code block lang

* Fail fast when pipelock ingress enabled with empty hosts

* Fail fast when pipelock ingress host has empty paths

* Messed up the conflict merge

---------

Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
2026-03-03 16:32:35 +01:00
LPW 84bfe5b7ab Add external AI assistant with Pipelock security proxy (#1069)
* feat(helm): add Pipelock ConfigMap, scanning config, and consolidate compose

- Add ConfigMap template rendering DLP, response scanning, MCP input/tool
  scanning, and forward proxy settings from values
- Mount ConfigMap as /etc/pipelock/pipelock.yaml volume in deployment
- Add checksum/config annotation for automatic pod restart on config change
- Gate HTTPS_PROXY/HTTP_PROXY env injection on forwardProxy.enabled (skip
  in MCP-only mode)
- Use hasKey for all boolean values to prevent Helm default swallowing false
- Single source of truth for ports (forwardProxy.port/mcpProxy.port)
- Pipelock-specific imagePullSecrets with fallback to app secrets
- Merge standalone compose.example.pipelock.yml into compose.example.ai.yml
- Add pipelock.example.yaml for Docker Compose users
- Add exclude-paths to CI workflow for locale file false positives

* Add external assistant support (OpenAI-compatible SSE proxy)

Allow self-hosted instances to delegate chat to an external AI agent
via an OpenAI-compatible streaming endpoint. Configurable per-family
through Settings UI or ASSISTANT_TYPE env override.

- Assistant::External::Client: SSE streaming HTTP client (no new gems)
- Settings UI with type selector, env lock indicator, config status
- Helm chart and Docker Compose env var support
- 45 tests covering client, config, routing, controller, integration

* Add session key routing, email allowlist, and config plumbing

Route to the actual OpenClaw session via x-openclaw-session-key header
instead of creating isolated sessions. Gate external assistant access
behind an email allowlist (EXTERNAL_ASSISTANT_ALLOWED_EMAILS env var).
Plumb session_key and allowedEmails through Helm chart, compose, and
env template.

* Add HTTPS_PROXY support to External::Client for Pipelock integration

Net::HTTP does not auto-read HTTPS_PROXY/HTTP_PROXY env vars (unlike
Faraday). Explicitly resolve proxy from environment in build_http so
outbound traffic to the external assistant routes through Pipelock's
forward proxy when enabled. Respects NO_PROXY for internal hosts.

* Add UI fields for external assistant config (Setting-backed with env fallback)

Follow the same pattern as OpenAI settings: database-backed Setting
fields with env var defaults. Self-hosters can now configure the
external assistant URL, token, and agent ID from the browser
(Settings > Self-Hosting > AI Assistant) instead of requiring env vars.
Fields disable when the corresponding env var is set.

* Improve external assistant UI labels and add help text

Change placeholder to generic OpenAI-compatible URL pattern. Add help
text under each field explaining where the values come from: URL from
agent provider, token for authentication, agent ID for multi-agent
routing.

* Add external assistant docs and fix URL help text

Add External AI Assistant section to docs/hosting/ai.md covering setup
(UI and env vars), how it works, Pipelock security scanning, access
control, and Docker Compose example. Drop "chat completions" jargon
from URL help text.

* Harden external assistant: retry logic, disconnect UI, error handling, and test coverage

- Add retry with backoff for transient network errors (no retry after streaming starts)
- Add disconnect button with confirmation modal in self-hosting settings
- Narrow rescue scope with fallback logging for unexpected errors
- Safe cleanup of partial responses on stream interruption
- Gate ai_available? on family assistant_type instead of OR-ing all providers
- Truncate conversation history to last 20 messages
- Proxy-aware HTTP client with NO_PROXY support
- Sanitize protocol to use generic headers (X-Agent-Id, X-Session-Key)
- Full test coverage for streaming, retries, proxy routing, config, and disconnect

* Exclude external assistant client from Pipelock scan-diff

False positive: `@token` instance variable flagged as "Credential in URL".
Temporary workaround until Pipelock supports inline suppression.

* Address review feedback: NO_PROXY boundary fix, SSE done flag, design tokens

- Fix NO_PROXY matching to require domain boundary (exact match or .suffix),
  case-insensitive. Prevents badexample.com matching example.com.
- Add done flag to SSE streaming so read_body stops after [DONE]
- Move MAX_CONVERSATION_MESSAGES to class level
- Use bg-success/bg-destructive design tokens for status indicators
- Add rationale comment for pipelock scan exclusion
- Update docs last-updated date

* Address second round of review feedback

- Allowlist email comparison is now case-insensitive and nil-safe
- Cap SSE buffer at 1 MB to prevent memory blowup from malformed streams
- Don't expose upstream HTTP response body in user-facing errors (log it instead)
- Fix frozen string warning on buffer initialization
- Fix "builtin" typo in docs (should be "built-in")

* Protect completed responses from cleanup, sanitize error messages

- Don't destroy a fully streamed assistant message if post-stream
  metadata update fails (only cleanup partial responses)
- Log raw connection/HTTP errors internally, show generic messages
  to users to avoid leaking network/proxy details
- Update test assertions for new error message wording

* Fix SSE content guard and NO_PROXY test correctness

Use nil check instead of present? for SSE delta content to preserve
whitespace-only chunks (newlines, spaces) that can occur in code output.

Fix NO_PROXY test to use HTTP_PROXY matching the http:// client URL so
the proxy resolution and NO_PROXY bypass logic are actually exercised.

* Forward proxy credentials to Net::HTTP

Pass proxy_uri.user and proxy_uri.password to Net::HTTP.new so
authenticated proxies (http://user:pass@host:port) work correctly.
Without this, credentials parsed from the proxy URL were silently
dropped. Nil values are safe as positional args when no creds exist.

* Update pipelock integration to v0.3.1 with full scanning config

Bump Helm image tag from 0.2.7 to 0.3.1. Add missing security
sections to both the Helm ConfigMap and compose example config:
mcp_tool_policy, mcp_session_binding, and tool_chain_detection.
These protect the /mcp endpoint against tool injection, session
hijacking, and multi-step exfiltration chains.

Add version and mode fields to config files. Enable include_defaults
for DLP and response scanning to merge user patterns with the 35
built-in patterns. Remove redundant --mode CLI flag from the Helm
deployment template since mode is now in the config file.
2026-03-03 15:47:51 +01:00
Juan José Mata 4e4ca916a1 Update backend table with status and requirements
Clarify status of non-OpenAI vector store

Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
2026-02-11 15:59:12 +01:00
Juan José Matacoderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>Claude
9e57954a99 Add Family vector search function call / support for document vault (#961)
* Add SearchFamilyImportedFiles assistant function with vector store support

Implement per-Family document search using OpenAI vector stores, allowing
the AI assistant to search through uploaded financial documents (tax returns,
statements, contracts, etc.). The architecture is modular with a provider-
agnostic VectorStoreConcept interface so other RAG backends can be added.

Key components:
- Assistant::Function::SearchFamilyImportedFiles - tool callable from any LLM
- Provider::VectorStoreConcept - abstract vector store interface
- Provider::Openai vector store methods (create, upload, search, delete)
- Family::VectorSearchable concern with document management
- FamilyDocument model for tracking uploaded files
- Migration adding vector_store_id to families and family_documents table

https://claude.ai/code/session_01TSkKc7a9Yu2ugm1RvSf4dh

* Extract VectorStore adapter layer for swappable backends

Replace the Provider::VectorStoreConcept mixin with a standalone adapter
architecture under VectorStore::. This cleanly separates vector store
concerns from the LLM provider and makes it trivial to swap backends.

Components:
- VectorStore::Base — abstract interface (create/delete/upload/remove/search)
- VectorStore::Openai — uses ruby-openai gem's native vector_stores.search
- VectorStore::Pgvector — skeleton for local pgvector + embedding model
- VectorStore::Qdrant — skeleton for Qdrant vector DB
- VectorStore::Registry — resolves adapter from VECTOR_STORE_PROVIDER env
- VectorStore::Response — success/failure wrapper (like Provider::Response)

Consumers updated to go through VectorStore.adapter:
- Family::VectorSearchable
- Assistant::Function::SearchFamilyImportedFiles
- FamilyDocument

Removed: Provider::VectorStoreConcept, vector store methods from Provider::Openai

https://claude.ai/code/session_01TSkKc7a9Yu2ugm1RvSf4dh

* Add Vector Store configuration docs to ai.md

Documents how to configure the document search feature, covering all
three supported backends (OpenAI, pgvector, Qdrant), environment
variables, Docker Compose examples, supported file types, and privacy
considerations.

https://claude.ai/code/session_01TSkKc7a9Yu2ugm1RvSf4dh

* No need to specify `imported` in code

* Missed a couple more places

* Tiny reordering for the human OCD

* Update app/models/assistant/function/search_family_files.rb

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: Juan José Mata <jjmata@jjmata.com>

* PR comments

* More PR comments

---------

Signed-off-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2026-02-11 15:22:56 +01:00
BitToby ba6e286b41 feat: add SSL_CA_FILE and SSL_VERIFY environment variables to support… (#894)
* feat: add SSL_CA_FILE and SSL_VERIFY environment variables to support self-signed certificates in self-hosted environments

* fix: NoMethodError by defining SSL helper methods before configure block executes

* refactor: Refactor SessionsController to use shared SslConfigurable module and simplify SSL initializer redundant checks

* refactor: improve SSL configuration robustness and error detection accuracy

* fix:HTTParty SSL options, add file validation guards, prevent Tempfile GC, and redact URLs in error logs

* fix:  Fix SSL concern indentation and stub Simplefin POST correctly in tests

* fix: normalize ssl_verify to always return boolean instead of nil

* fix: solve failing SimpleFin test

* refactor:  trim unused error-handling code from SslConfigurable, replace Tempfile with fixed-path CA bundle, fix namespace pollution in initializers, and add unit tests for core SSL configuration and Langfuse CRL callback.

* fix: added require ileutils in the initializer and require ostruct in the test file.

* fix: solve autoload conflict that broke provider loading, validate all certs in PEM bundles, and add missing requires.
2026-02-06 18:04:03 +01:00
6f8858b1a6 feat/Add AI-Powered Bank Statement Import (step 1, PDF import & analysis) (#808)
* feat: Add PDF import with AI-powered document analysis

This enhances the import functionality to support PDF files with AI-powered
document analysis. When a PDF is uploaded, it is processed by AI to:
- Identify the document type (bank statement, credit card statement, etc.)
- Generate a summary of the document contents
- Extract key metadata (institution, dates, balances, transaction count)

After processing, an email is sent to the user asking for next steps.

Key changes:
- Add PdfImport model for handling PDF document imports
- Add Provider::Openai::PdfProcessor for AI document analysis
- Add ProcessPdfJob for async PDF processing
- Add PdfImportMailer for user notification emails
- Update imports controller to detect and handle PDF uploads
- Add PDF import option to the new import page
- Add i18n translations for all new strings
- Add comprehensive tests for the new functionality

* Add bank statement import with AI extraction

- Create ImportBankStatement assistant function for MCP
- Add BankStatementExtractor with chunked processing for small context windows
- Register function in assistant configurable
- Make PdfImport#pdf_file_content public for extractor access
- Increase OpenAI request timeout to 600s for slow local models
- Increase DB connection pool to 20 for concurrent operations

Tested with M-Pesa bank statement via remote Ollama (qwen3:8b):
- Successfully extracted 18 transactions
- Generated CSV and created TransactionImport
- Works with 3000 char chunks for small context windows

* Add pdf-reader gem dependency

The BankStatementExtractor uses PDF::Reader to parse bank statement
PDFs, but the gem was not properly declared in the Gemfile. This would
cause NameError in production when processing bank statements.

Added pdf-reader ~> 2.12 to Gemfile dependencies.

* Fix transaction deduplication to preserve legitimate duplicates

The previous deduplication logic removed ALL duplicate transactions based
on [date, amount, name], which would drop legitimate same-day duplicates
like multiple ATM withdrawals or card authorizations.

Changed to only deduplicate transactions that appear in consecutive chunks
(chunking artifacts) while preserving all legitimate duplicates within the
same chunk or non-adjacent chunks.

* Refactor bank statement extraction to use public provider method

Address code review feedback:
- Add public extract_bank_statement method to Provider::Openai
- Remove direct access to private client via send(:client)
- Update ImportBankStatement to use new public method
- Add require 'set' to BankStatementExtractor
- Remove PII-sensitive content from error logs
- Add defensive check for nil response.error
- Handle oversized PDF pages in chunking logic
- Remove unused process_native and process_generic methods
- Update email copy to reflect feature availability
- Add guard for nil document_type in email template
- Document pdf-reader gem rationale in Gemfile

Tested with both OpenAI (gpt-4o) and Ollama (qwen3:8b):
- OpenAI: 49 transactions extracted in 30s
- Ollama: 40 transactions extracted in 368s
- All encapsulation and error handling working correctly

* Update schema.rb with ai_summary and document_type columns

* Address PR #808 review comments

- Rename :csv_file to :import_file across controllers/views/tests
- Add PDF test fixture (sample_bank_statement.pdf)
- Add supports_pdf_processing? method for graceful degradation
- Revert unrelated database.yml pool change (600->3)
- Remove month_start_day schema bleed from other PR
- Fix PdfProcessor: use .strip instead of .strip_heredoc
- Add server-side PDF magic byte validation
- Conditionally show PDF import option when AI provider available
- Fix ProcessPdfJob: sanitize errors, handle update failure
- Move pdf_file attachment from Import to PdfImport
- Document deduplication logic limitations
- Fix ImportBankStatement: catch specific exceptions only
- Remove unnecessary require 'set'
- Remove dead json_schema method from PdfProcessor
- Reduce default OpenAI timeout from 600s to 60s
- Fix nil guard in text mailer template
- Add require 'csv' to ImportBankStatement
- Remove Gemfile pdf-reader comment

* Fix RuboCop indentation in ProcessPdfJob

* Refactor PDF import check to use model predicate method

Replace is_a?(PdfImport) type check with requires_csv_workflow? predicate
that leverages STI inheritance for cleaner controller logic.

* Fix missing 'unknown' locale key and schema version mismatch

- Add 'unknown: Unknown Document' to document_types locale
- Fix schema version to match latest migration (2026_01_24_180211)

* Document OPENAI_REQUEST_TIMEOUT env variable

Added to .env.local.example and docs/hosting/ai.md

* Rename ALLOWED_MIME_TYPES to ALLOWED_CSV_MIME_TYPES for clarity

* Add comment explaining requires_csv_workflow? predicate

* Remove redundant required_column_keys from PdfImport

Base class already returns [] by default

* Add ENV toggle to disable PDF processing for non-vision endpoints

OPENAI_SUPPORTS_PDF_PROCESSING=false can be used for OpenAI-compatible
endpoints (e.g., Ollama) that don't support vision/PDF processing.

* Wire up transaction extraction for PDF bank statements

- Add extracted_data JSONB column to imports
- Add extract_transactions method to PdfImport
- Call extraction in ProcessPdfJob for bank statements
- Store transactions in extracted_data for later review

* Fix ProcessPdfJob retry logic, sanitize and localize errors

- Allow retries after partial success (classification ok, extraction failed)
- Log sanitized error message instead of raw message to avoid data leakage
- Use i18n for user-facing error messages

* Add vision-capable model validation for PDF processing

* Fix drag-and-drop test to use correct field name csv_file

* Schema bleedover from another branch

* Fix drag-drop import form field name to match controller

* Add vision capability guard to process_pdf method

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
2026-01-30 20:44:25 +01:00
eureka928 02c71bca0a Add AI Cache Management documentation
Document the AI cache reset feature including what it does, when to use it,
how to reset via UI, and cost implications.
2026-01-26 10:41:14 +01:00
Juan José Mata 16e4c4ede4 Small QOL fix in shell samples
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
2026-01-22 08:14:43 +01:00
Nicolas PERNOT 73c816559d MERGE doc/adding_https_info INTO main (#709)
* Suggest to use .env.example

it's easier to start with a pre-done file than search for info about each configuration

* added paragraph to tell how to activate https

* fix type.

As there was 2 space, I assumed there should be a line break

* Corrects minor typos in HTTPS documentation

Addresses a couple of minor typographical errors in the HTTPS documentation section, improving clarity and readability for users setting up HTTPS.

* Clarifies .env setup in Docker documentation

Corrects grammar and improves clarity in the Docker documentation regarding obtaining the `.env.example` file for initial configuration.

* Fixes typo in HTTPS documentation

Corrects a grammatical error in the Docker HTTPS setup guide, improving clarity for users configuring SSL.
2026-01-19 19:36:14 +01:00
LPWandJosh Waldrep 320e087a22 Add support for displaying and managing legacy SSO providers (#628)
* feat: add support for displaying and managing legacy SSO providers

- Introduced UI section for environment/YAML-configured SSO providers.
- Added warnings and guidance on migrating legacy providers to database-backed configuration.
- Enhanced localization with new keys for legacy provider management.
- Updated form and toggle components for improved usability.

* Expand SSO documentation: add SAML 2.0 support, JIT provisioning settings, super-admin setup steps, audit logging, and user administration details.

* Update JIT provisioning docs: clarify role mapping behavior and add examples; note new `logout_idp` audit log event.

---------

Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
2026-01-13 09:37:19 +01:00
Josh Waldrep 238fa8e0ca Merge remote-tracking branch 'upstream/main' into sso-upgrades
# Conflicts:
#	app/views/simplefin_items/_simplefin_item.html.erb
#	db/schema.rb
2026-01-10 11:57:23 -05:00
zenaufaandJuan José Mata ae3eb0abf1 Added troubleshooting information for CSV import. (#558)
* Document CSV import processing delay issue

Added troubleshooting information for CSV import delays.

Signed-off-by: zenaufa <zenaufa@hotmail.com>

* Small edits suggested by LLM

---------

Signed-off-by: zenaufa <zenaufa@hotmail.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
2026-01-07 21:27:12 +01:00
Josh Waldrep 14993d871c feat: comprehensive SSO/OIDC upgrade with enterprise features
Multi-provider SSO support:
   - Database-backed SSO provider management with admin UI
   - Support for OpenID Connect, Google OAuth2, GitHub, and SAML 2.0
   - Flipper feature flag (db_sso_providers) for dynamic provider loading
   - ProviderLoader service for YAML or database configuration

   Admin functionality:
   - Admin::SsoProvidersController for CRUD operations
   - Admin::UsersController for super_admin role management
   - Pundit policies for authorization
   - Test connection endpoint for validating provider config

   User provisioning improvements:
   - JIT (just-in-time) account creation with configurable default role
   - Changed default JIT role from admin to member (security)
   - User attribute sync on each SSO login
   - Group/role mapping from IdP claims

   SSO identity management:
   - Settings::SsoIdentitiesController for users to manage connected accounts
   - Issuer validation for OIDC identities
   - Unlink protection when no password set

   Audit logging:
   - SsoAuditLog model tracking login, logout, link, unlink, JIT creation
   - Captures IP address, user agent, and metadata

   Advanced OIDC features:
   - Custom scopes per provider
   - Configurable prompt parameter (login, consent, select_account, none)
   - RP-initiated logout (federated logout to IdP)
   - id_token storage for logout

   SAML 2.0 support:
   - omniauth-saml gem integration
   - IdP metadata URL or manual configuration
   - Certificate and fingerprint validation
   - NameID format configuration
2026-01-03 17:56:42 -05:00
LPWandJosh Waldrep b23711ae0d Add configurable multi-provider SSO, SSO-only mode, and JIT controls via auth.yml (#441)
* Add configuration and logic for dynamic SSO provider support and stricter JIT account creation

- Introduced `config/auth.yml` for centralized auth configuration and documentation.
- Added support for multiple SSO providers, including Google, GitHub, and OpenID Connect.
- Implemented stricter JIT SSO account creation modes (`create_and_link` vs `link_only`).
- Enabled optional restriction of JIT creation by allowed email domains.
- Enhanced OmniAuth initializer for dynamic provider setup and better configurability.
- Refined login UI to handle local login disabling and emergency super-admin override.
- Updated account creation flow to respect JIT mode and domain checks.
- Added tests for SSO account creation, login form visibility, and emergency overrides.

# Conflicts:
#	app/controllers/sessions_controller.rb

* remove non-translation

* Refactor authentication views to use translation keys and update locale files

- Extracted hardcoded strings in `oidc_accounts/link.html.erb` and `sessions/new.html.erb` into translation keys for better localization support.
- Added missing translations for English and Spanish in `sessions` and `oidc_accounts` locale files.

* Enhance OmniAuth provider configuration and refine local login override logic

- Updated OmniAuth initializer to support dynamic provider configuration with `name` and scoped parameters for Google and GitHub.
- Improved local login logic to enforce stricter handling of super-admin override when local login is disabled.
- Added test for invalid super-admin override credentials.

* Document Google sign-in configuration for local development and self-hosted environments

---------

Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
2025-12-24 00:15:53 +01:00
Blaž Dular 8972cb59f0 docs: add env variable for ai debug to docs (#494) 2025-12-23 19:57:32 +01:00
Juan José Mata c47a790ad9 We tag alphas as latest now
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
2025-11-11 11:07:06 +01:00
soky srmandJuan José Mata da114b5b3d Update ai.md (#263)
* Update ai.md

Change some deprecated models

Signed-off-by: soky srm <sokysrm@gmail.com>

* Fix typo in AI model description

Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>

---------

Signed-off-by: soky srm <sokysrm@gmail.com>
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2025-10-30 23:38:14 +01:00
Juan José Mata f18c11c7ac Update AI model recommendations section
Added a caution note about model support and testing approach.

Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
2025-10-29 18:52:26 +01:00
Juan José Mata 3f4330eea8 Update AI assistant documentation with version caution
Added caution note regarding AI assistant support versions.

Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
2025-10-29 18:44:58 +01:00
Juan José Mataandsokie 768e85ce08 Add OpenID Connect login support (#77)
* Add OpenID Connect login support
* Add docs for OIDC config with Google Auth
* Use Google styles for log in
- Add support for linking existing account
- Force users to sign-in with passoword first, when linking existing accounts
- Add support to create new user when using OIDC
- Add identities to user to prevent account take-ver
- Make tests mocking instead of being integration tests
- Manage session handling correctly
- use OmniAuth.config.mock_auth instead of passing auth data via request env
* Conditionally render Oauth button

- Set a config item `configuration.x.auth.oidc_enabled`
- Hide button if disabled

---------

Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: sokie <sokysrm@gmail.com>
2025-10-24 16:07:45 +02:00
Copilotcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>jjmataJuan José Mata
a8f318c3f9 Fix "Messages is invalid" error for Ollama/custom LLM providers and add comprehensive AI documentation (#225)
* Add comprehensive AI/LLM configuration documentation
* Fix Chat.start! to use default model when model is nil or empty
* Ensure all controllers use Chat.default_model for consistency
* Move AI doc inside `hosting/`
* Probably too much error handling

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2025-10-24 12:04:19 +02:00
Juan José Mataandcoderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> 7c5ddd674d Make branding configurable (#173)
* Remove orphan function

* Add centralized branding helpers and update locales

* Remove _plus and add (proper) brand

* No longer Sure, configurable

* Consistency with compose file naming

* Missed `product_name` mapping

* Fix brand/product name in mailers

* Product name in email reset flow

* Fix i18n errors/tests

* Fix password mailer brand/product name (again)

* Missed hardcoded `Sure` in onboarding goals

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: Juan José Mata <jjmata@jjmata.com>

* PR nitpick on documentation

* Missing interpolation key for invited UI

* Orphan assets

* New logos

---------

Signed-off-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-10-22 19:14:03 +02:00
Pedro Camara Junior 3aea1513d1 Add comprehensive Hetzner Cloud deployment guide (#211)
* Add comprehensive Hetzner Cloud deployment guide

* Fix markdown linting issues and backup retention policy

- Add missing language identifiers to fenced code blocks (bash)
- Fix inconsistent backup retention policy (standardize to 7 days)
- Address CodeRabbit review feedback for PR #211
2025-10-21 15:34:44 +02:00
Juan José Mata 5706280dd7 More rebranding changes (#159)
* Replace Maybe for Sure in select code areas

* Make sure passwords are consistent

* Remove (admin|member) from demo data first name

* Database and schema names finally to `sure`

* Fix broken test

* Another (benchmarking) database name to `sure_*`

* More rebranding to Sure

* Missed this Maybe mention in the same page

* Random nitpicks and more Maybes

* Demo data accounts and more Maybes

* Test data account updates

* Impersonation test accounts

* Consistency with `compose.example.yml`
2025-09-24 00:19:51 +02:00
Phillip Boushy f8e420c5d1 Move plaid PR mention to note (#144)
* maybe#2419 was recreated as sure#1 but message wasn't removed.
 * Information in maybe#2419 gives more details that are helpful for
   users wanting to setup Plaid
2025-09-18 11:47:22 +02:00
Amr AwadandJuan José Mata 26c18427c7 Add a 'Bank Sync' page in Settings (#111)
* add initial pages for Bank Sync and Lunch Flow

* update breadcrumbs on Lunch Flow page

* update content for Lunch Flow page

* add norefeerrer to Github link

* update lunch flow url

* nest lunch_flow resource under bank_sync

* add a provider link partial

* remove trailing whitespaces

* update providers style to match merchants page

* remove separate lunch flow page

* fix hover on dark mode

* point lunch flow to custom sure landing page

* [i18n] Bank Sync label

* [i18n] API Keys

* [i18n] Self-Hosting consistency

* Security breadcrum, not "securities" default

---------

Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2025-08-15 03:07:15 +02:00
Juan José Matagemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Vincent Teo
dd0cb60b56 feat: Add Brand Fetch logo link for logos (see #43) (#99)
* feat: Add Brand Fetch logo link for logos

* docs: brand fetch integration docs

* Document CLIENT_ID location?

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Juan José Mata <jjmata@jjmata.com>

---------

Signed-off-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: Vincent Teo <vinteo@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2025-08-11 22:34:54 +02:00
Juan José Mata f6dde1a098 Add Langfuse-based LLM observability (#86)
* Add Langfuse-based LLM observability

* Document Langfuse configuration

* Don't hardcode model in use
2025-08-06 23:23:07 +02:00
Juan José Mata 099425d240 First quick Sure rebrand (#74)
* First cut of smallest rebrand, pending icons

* Leave SQL schema tokens/user names the same for now

* First batch of logos

* Release notes/what's new

* /releases missing

* redirect_uri for sureapp://

* Padded logo

* Test the correct /releases URL

* Missed a few mobile URIs

* Some icons/asssets from /website/ repo

* Seed/sample data user @sure.local now

* New screenshot

* Want to keep their legal "boilerplate" from the upstream repo
2025-08-05 23:35:01 +02:00