24 Commits

Include renames

Author	SHA1	Message	Date
Juan José Mata	fd6f4ff078	Add live AI checks to system health (#3155) ... * Add live AI checks to system health Give super admins a dedicated AI status view with bounded liveness probes for LLMs, vector stores, pgvector, and embedding endpoints. Record sanitized failures in both the system debug log and Rails logger, and document the recommended local configuration.\n\nCloses #3145 * Fix AI health CI checks * Address AI health review feedback * Correct Ollama model preload guidance * Distinguish OpenAI-compatible providers * Make Ollama startup readiness explicit * Recognize Cloudflare AI endpoints	2026-08-24 22:41:08 +02:00
Josh	2f821e2567	chore(security): update Pipelock integration to 3.4.0 (#3122) ... * chore(security): update Pipelock integration to 3.4.0 * fix(ci): validate shipped Pipelock configs * fix(security): isolate external assistant profile * fix(ci): build Helm dependencies before validation * fix(ci): strengthen Pipelock contract checks	2026-08-22 05:41:00 +02:00
Brandon	25a9011f14	feat(ai): analytical tool-set upgrade for the builtin assistant (#3064) ... * fix(assistant): survive tool failures with error and hint results instead of aborting the turn A tool exception used to raise FunctionExecutionError out of the responder loop, turning the whole turn into a generic chat error banner. An unknown tool name was worse: the rescue block itself crashed (fn.name on nil). Tool failures now come back to the model as data ({error, hint}) so the conversation survives and the model can retry once with corrected arguments. The catch-all branch logs and tells the model not to retry. FunctionExecutionError remains defined for API compatibility. * fix(assistant): strict schemas declare every property as required get_categories and get_tags declared an optional page property while inheriting strict mode, which is invalid under strict function calling (every property must be listed in required). Both now opt out of strict mode like every other paginated tool, and gain a page_size param (1..100) while being touched. A registry-walking test asserts the invariant for every current and future tool, preview tools included. * fix(assistant): HistoryTrimmer always keeps the newest turn Trimming iterates newest-first and stopped at the first group over budget. When the newest group alone exceeded the budget, everything was dropped, including the user message the model was being asked to answer, and the provider received only the system prompt. The newest group now always survives. * perf(assistant): compact AI time series and make account history opt-in get_accounts shipped a 5-year monthly series for every account on every call, roughly 60 formatted money strings per account, which dominates the tool payload for multi-account families and swamps small self-hosted context windows. The series is now opt-in (include_balance_series) and bounded by a named period (series_period, default last_365_days). to_ai_time_series states the currency once and emits numeric values instead of formatting every point; the system prompt already tells the model how to render currency. get_accounts also now returns account ids (they are what other tools accept as account_ids filters) and respects the visible scope, so hidden accounts no longer leak into responses. * refactor(assistant): id and name filters replace user-data enums in get_transactions The schema inlined every account, category, merchant, and tag name as enum values on every request. That grows without bound with family data, defeats provider prompt caching (definitions change whenever a name does), and is the pattern that made empty-enum pruning necessary in the first place. Filters are now plain string arrays documented as exact names from the sibling get_* tools, which Transaction::Search already resolves server-side, plus an account_ids UUID filter. New params: page_size (1..100), sort_by amount, types (income/expense/transfer, the way to exclude transfers), and statuses (pending/confirmed). The three now-unused enum helpers are removed from the base class; family_tag_names stays for update_tag, which still identifies tags by name. * feat(assistant): add get_merchants and get_recurring_transactions Merchants were unreachable: names appeared nowhere and update_transaction's merchant_id had no source of ids, making it unusable. get_merchants lists id, exact name, and source, scoped through available_merchants_for so merchants seen only in accounts hidden from the user never leak. Recurring transactions had a model, an Upcoming view, and no assistant reach. get_recurring_transactions lists detected and manual recurring items (status filter defaulting to active, optional upcoming_within_days window) with per-currency totals of active non-transfer items, answering subscription and upcoming-bill questions directly instead of via transaction paging. * feat(assistant): flexible periods on get_balance_sheet and trends on get_income_statement get_balance_sheet was hard-wired to five years of monthly history with no parameters, although Period supports arbitrary ranges and the chart builder takes any interval. It now accepts a named period or custom dates plus an interval, with a 400-point cap so a day-granularity request over a decade returns an error instead of a giant series. The default call is byte-compatible with the old shape. The balance sheet object is also memoized; it was being constructed four times per call. get_income_statement gains the analysis surface the assistant lacked: group_by month for a monthly income/expenses/net series (capped at 36 buckets), compare_previous_period for an equal-length prior window with absolute and percent deltas, and account_ids to scope totals to specific accounts via IncomeStatement#totals_for. Category breakdowns are family-wide by construction, so the account-filtered view omits them and says why. Unknown or inaccessible account ids come back as a soft failure naming the ids so the model can correct itself. * feat(assistant): preview reads for insights and valuations The Insights feed is generated nightly with pre-computed numbers, and the chat assistant could not read a word of it. get_insights returns the visible feed (type filter, acknowledged toggle, limit) without marking anything read; an assistant read is not the user viewing the feed. It sits in PREVIEW_FUNCTION_CLASSES because the feature itself is preview-gated, which also keeps it off the default /mcp surface. record_valuation was write-only: an agent recording provenance-cited valuations had no way to audit what it wrote or find dates already carrying a value. get_valuations lists valuation entries newest first with kind and the citation notes, scoped to accessible visible accounts. * feat(assistant): cache-stable system prompt with session context The prompt interpolated currency mid-text and the date near the end, so no two requests shared a cacheable prefix, and it told the model nothing about the family: not one account name, not a single category. Models opened most conversations blind, either wandering through tools or answering without data. The prompt is now STATIC_INSTRUCTIONS, a frozen constant that is byte-identical for every request (providers discount an exactly-repeated prefix; tool definitions are also stable now that schemas carry no user data), followed by a trailing Session context block holding everything volatile: date, date format, currency details, an account roster with balances, and category names. The static half gains a request-classification rule (CHAT / LOOKUP / ANALYSIS), a reuse-what-you-have rule with an explicit re-fetch carve-out, specific-tool preference, and the error/hint retry-once rule that pairs with the tool soft-fail contract. Context stays cheap by construction: the roster collapses to per-type counts beyond 25 accounts, categories to a count beyond 60 names, and both collapse whenever the configured context window is under 4096 (the self-hosted default is 2048), via the new Assistant::TokenBudget helper. Intro chats are untouched. * feat(assistant): raise tool-round cap to 8 with a no-tools grace turn; instructions-aware history budget Five rounds was tight for a tool surface that now supports real analysis chains, and hitting the cap raised ToolCallLimitError, which surfaced to the user as a dead chat with an error banner. The default is now eight rounds (env override unchanged), and on the final permitted round the follow-up request offers no tools, so the model must answer in text with whatever it gathered. The limit error remains as a defensive backstop. The generic-path history budget reserved a flat 256 tokens for a system prompt that already estimates well past that; the trimmer now budgets against the actual instructions when available. LLM_MAX_RESPONSE_TOKENS was reserved in budget math but never sent to the provider. It is now sent (max_tokens on chat completions, max_output_tokens on the Responses API) only when explicitly configured via ENV or a stored Setting; stock installs keep today's uncapped behavior. * test(evals): chat golden v2 exercising the real prompt and registry The eval runner scored a fiction: hardcoded instructions and four fake permissive tool schemas, so a prompt or registry regression could sail through green. It now runs STATIC_INSTRUCTIONS plus a fixed synthetic session context and builds definitions from Assistant.function_classes against a reference user (classes whose schema cannot build are skipped with a log line, never faked). chat_golden_v2 adds routing scenarios the upgrade cares about: CHAT classification must use no tools, aggregates route to get_income_statement / get_balance_sheet rather than transaction paging, and the new analytical tools are selected with sensible params. The dataset header documents the harness's single-shot limitation. * docs(ai,mcp): current tool tables, responder loop, prompt structure, timeout math Both docs listed 7 tools against a registry of 19, in three separate drift-prone copies. mcp.md now carries the canonical tables (default + preview); ai.md links to them from the MCP section, keeps one grouped functions list for the architecture chapter, and replaces its stale hardcoded registry snippet with a pointer to assistant.rb. The architecture section gains the contracts contributors need when adding a function: the responder loop (rounds vs calls, cap 8, the no-tools grace turn) and the error/hint soft-failure convention, plus the prompt's static/session-context split and its collapse gates. Timeout guidance is recomputed for the new default cap. * fix(assistant): address automated review findings Codex and CodeRabbit findings on the initial push, all verified before changing anything: - AI time series rounded every value to two decimals, which turns 0.001 BTC into 0.0; values now round to the currency's own precision (BTC 8, CLF 4, OMR 3). - get_income_statement validated account_ids against all visible accounts, but totals_for excludes hidden, excluded-from-reports and tax-advantaged accounts, so those ids produced silent zeros. Ids now validate against income_statement.eligible_accounts and the soft failure explains eligibility. - get_recurring_transactions computed totals from the displayed rows, so past the 200-row cap the value labeled a total was partial. Totals now aggregate over the full filtered scope in SQL, and the response carries total_results and a truncated flag. The upcoming_within_days window also starts at today, matching its documentation; overdue items appear in unwindowed calls. - get_valuations silently dropped a malformed date filter and presented unfiltered data as filtered; malformed dates now return invalid_date. - get_balance_sheet returned a generic failure for a reversed custom range because Period's own validation raises past the Date::Error rescue; it now returns the structured invalid_date error. - get_insights documents that its family-wide scope matches the web feed exactly (InsightsController serves Current.family.insights to every member), so the tool exposes nothing the /insights page does not already show the same user. - Tests: limit clamp proven against more insights than the cap, Setting fallbacks stubbed in the provider budget tests, currency precision and reversed-range regression tests added. * refactor(assistant): apply reviewer nitpicks - order declares type alongside its enum, matching sort_by - page-size clamp deduplicated into the base class (MAX_PAGE_SIZE + shared resolved_page_size); dead per-tool copies removed - get_accounts preloads balance rows only when the series is requested - get_income_statement validates the bucket count before running any aggregation work Deliberately unchanged: the balance sheet's monthly_history key. The default response shape stays byte-compatible for existing MCP consumers, and the nested series already states its interval. * fix(assistant): second-round review findings on get_valuations - A reversed date range (start after end) now returns the structured invalid_date error instead of presenting an empty result as filtered data, matching get_balance_sheet's handling. - Page numbers are normalized before pagination: Pagy raises on zero, negative or non-numeric pages. The fix lands as a shared resolved_page helper on the base class and applies to every paginated tool (categories, tags, merchants, transactions, holdings, valuations), since all shared the same page-or-1 pattern; schemas declare minimum: 1. * fix(assistant): round series amounts as BigDecimal before Float conversion Converting to Float first can perturb the value at the requested precision; round the exact decimal, then convert for JSON. * fix(assistant): address maintainer review findings - get_accounts no longer fails the whole listing when one account's start date lies beyond the requested period (start_date derives from the first entry, which can be future-dated); that account simply has no series. The unrescued Period.custom was reachable exactly there. - The balances preload is gone: the series goes through Balance::ChartSeriesBuilder, which runs its own query keyed by account ids, so the eager-loaded rows were loaded and discarded. - Provider::Openai#context_window now delegates to Assistant::TokenBudget, removing the duplicated ENV > Setting > default precedence so prompt assembly and the provider can never disagree about the window. * fix(ai): final no-tools round uses tool_choice none instead of dropping tools Anthropic rejects requests whose messages contain tool_use blocks when no tools are defined, so re-requesting with an empty tool list made the final-round grace die in a provider 400 on Anthropic models. The final round now sends the real tool definitions with tool_choice none, which both providers accept, and the model answers in prose as intended. * fix(assistant): scope every income statement read to the requesting user get_income_statement validated account_ids against the user-scoped statement but computed every total from an unscoped one. IncomeStatement falls back to Current.user, which is nil in the assistant job and the MCP endpoint, so the unscoped reads dropped the included_in_finances_for filter and reported family-wide totals next to ids that had been checked against a narrower set. Route all reads through one memoized user-scoped statement, the idiom get_balance_sheet already uses. Also lets the per-instance memoization in IncomeStatement apply across the eligibility check and the totals. Adds a regression test that fails without the change, plus a companion test asserting eligibility and totals agree on scope. Guard the strictness walk against an empty registry so it cannot silently assert nothing.	2026-08-21 21:58:47 +02:00
Sure Admin (bot)	5c18086089	docs: refresh Sure MCP and external AI setup (#2608) ... * docs: explain self-hosted onboarding modes * docs: refresh MCP and external AI docs * docs: correct MCP auth and tool accuracy * docs: address MCP review comments	2026-08-20 06:37:42 +02:00
Andrew B	c9fbfd9f71	fix(chat): make the assistant response timeout configurable (#2910) ... * fix(chat): make the assistant response timeout configurable (#2893) Self-hosted users running a local model report the chat failing with "assistant not available" after 90 seconds even though the model generates a reply and tokens are billed. Three timeouts are involved and only one was configurable: - OPENAI_REQUEST_TIMEOUT (60s) — already settable, not the blocker. - The browser watchdog in chat_controller.js (90s) — hardcoded, and this is what actually fires. - Chat::UNDELIVERED_RESPONSE_TIMEOUT (60s) — a constant, so raising the client value alone would not have helped. The watchdog cannot be avoided by streaming here: custom OpenAI-compatible providers route through generic_chat_response, which forces synchronous calls, so nothing renders until the whole generation finishes. Time-to-last-token has to beat the deadline. Adds AI_RESPONSE_TIMEOUT (ENV > Setting > 90s default, floored at 30s), exposed on the Self-Hosting settings page and passed to the Stimulus controller at all three mount points — show, index and the sidebar in the application layout, each of which declares data-controller="chat" independently. The server floor is derived from the same value but kept 10s below it. report_timeout answers 200 whether or not it acted and the client only retries on a non-ok response, so a floor at or above the client value would let clock skew strand a pending bubble permanently. Also guards AssistantMessage#append_text!. The watchdog runs in the web process while the job holds its own copy of the message, so a job finishing after the bubble was destroyed or demoted would silently resurrect it alongside the error the user was already shown. * fix(chat): let the watchdog retry when report_timeout declines `report_timeout` answered 200 whether or not `handle_undelivered_response!` acted. The Stimulus watchdog only stops retrying a URL once it sees a 2xx, so a declined report was treated as final. That stranded the bubble whenever the client's clock ran more than SERVER_TIMEOUT_GRACE ahead of the server's: the watchdog posts at its own timeout, the server sees a message younger than its floor and no-ops, and nothing ever retries. The bubble spins forever with no error and no Retry. Answering 409 instead lets the next 5s tick try again, so any amount of skew costs retries rather than a stuck chat. The grace window stays as an optimisation to keep those retries rare, not as the correctness mechanism. * docs(chat): correct the AI_RESPONSE_TIMEOUT ordering guidance The docs, locale string and examples all said to keep OPENAI_REQUEST_TIMEOUT at or above AI_RESPONSE_TIMEOUT. That is backwards. The two limits span different things. OPENAI_REQUEST_TIMEOUT bounds each HTTP call to the model on its own; AI_RESPONSE_TIMEOUT covers the whole turn and its clock starts when the message is queued, so it also absorbs Sidekiq queue time and, for a tool-using turn, two model calls plus the tool run between them. Keeping the chat timeout the larger of the two means a slow model surfaces the specific HTTP timeout error rather than a generic "no response", and the job stops instead of running on after the chat has given up. The shipped 60/90 defaults already had this ordering; only the guidance was wrong. compose.example.ai.yml gets 300/660 so the Ollama example can actually complete a tool-using turn. * fix(chat): claim the pending bubble atomically before appending append_text! read the row's status and then saved, leaving a window in which the watchdog could demote the row to `failed` between the two. The late content would then land on a bubble the user had already been told failed, flipping it back to `complete`. Replaces the read with a conditional UPDATE that only succeeds while the row is still pending, so the check and the state change cannot be separated. Uses a conditional UPDATE rather than with_lock because append_text! is called once per chunk on the streaming path; a row lock and transaction per chunk would be far more expensive. The claim only runs on the first append, since later ones are no longer pending. * test(chat): isolate AI_RESPONSE_TIMEOUT from the environment Chat.response_timeout reads ENV ahead of Setting, so stubbing only the Setting left the assertions at the mercy of the environment they run in. With AI_RESPONSE_TIMEOUT=45 exported, five of these tests failed — the default, floor and grace assertions were all silently measuring the env value. Adds a with_setting_timeout helper that stubs the Setting and clears the variable together, and switches the controller tests to stub Chat.undelivered_response_timeout directly, since what they care about is the resolved floor rather than how it was configured. Both files now pass with or without AI_RESPONSE_TIMEOUT set. * docs(chat): size AI_RESPONSE_TIMEOUT for chained tool calls The guidance assumed a tool-using turn costs two model calls. #2767 landed after this branch was opened and made tool calls iterative: `Assistant::Responder` now loops until `iteration > max_tool_call_iterations`, so a turn runs to 1 + ASSISTANT_MAX_TOOL_CALL_ITERATIONS calls — six by default — with tool execution in between. At the default 60s per-call timeout that is up to 360s of model time against a 90s watchdog. Streaming does not rescue this either. `emit(:output_text)` only fires for a response that carries text, and tool-call-only rounds carry none, so the bubble stays on "Thinking…" through every round regardless of provider. Documents ASSISTANT_MAX_TOOL_CALL_ITERATIONS as the cheaper lever: dropping it to 2 halves the worst case instead of demanding a half-hour timeout, at the cost of failing long tool chains earlier with a clear limit error. compose.example.ai.yml now shows that combination rather than a timeout sized for six calls it never had. * docs(chat): state the whole-turn timeout as a sum, not a maximum The guidance said to keep AI_RESPONSE_TIMEOUT "above" or "the larger of" OPENAI_REQUEST_TIMEOUT. That understates it: the watchdog covers the entire turn, so the bound is (1 + ASSISTANT_MAX_TOOL_CALL_ITERATIONS) * OPENAI_REQUEST_TIMEOUT + tool execution + queue wait Merely exceeding the per-call limit can still leave the chat reporting failure while the worker keeps going. One phrasing was outright wrong: "keep AI_RESPONSE_TIMEOUT the largest of the three" compared a duration against ASSISTANT_MAX_TOOL_CALL_ITERATIONS, which is a count, not seconds. Resizes the examples against the formula — compose.example.ai.yml 1000 -> 1200 and the Ollama doc example 600 -> 720, both now showing the arithmetic — and states plainly that the 90s default is sized for typical cloud latency rather than the worst-case bound, with the formula being what matters once per-call latency approaches the timeout. * docs(chat): list the AI settings fields and tag the formula fence The Settings UI walkthrough listed three of the eight fields on the AI Provider form. JSON Mode, the three Token Budget fields and the new Chat Response Timeout were all missing, so the timeout was only discoverable from the troubleshooting section. Rewrites the list to follow the form's own grouping and uses the labels the form actually renders. Also tags the whole-turn formula fence as `text` (markdownlint MD040). * fix(compose): forward ASSISTANT_MAX_TOOL_CALL_ITERATIONS in standard compose This file enumerates container environment explicitly — there is no env_file — so a variable absent from the x-rails-env anchor never reaches web or worker. The tool-call cap was only named in a comment here, while the docs added in 3360dbf5 tell operators to lower it to keep a turn inside AI_RESPONSE_TIMEOUT. Following that advice on this compose file silently changed nothing: the app kept the default of 5 while the timeout was sized for 3 calls, which lands back on the "no response" error this branch exists to fix. Left with an empty default so the app's own default governs, matching OPENAI_MODEL and LLM_CONTEXT_WINDOW above. compose.example.ai.yml already forwarded it.	2026-08-15 06:12:22 +02:00
4 people	efb7cc3935	Tooling for the wealth + tax agent harness (#2848) ... * Expose the Statement Vault to external agents over MCP A user wants to manage patrimonial history — a document-backed record of a family's wealth where every figure traces back to the statement it came from — by pointing an external agent harness at Sure. That model belongs in the harness, not in Sure: it needs numbered build deltas, golden tests and closed periods that a mutable Postgres row cannot provide. What Sure was missing was the seam. The Statement Vault already does most of the work — original bytes retained, SHA-256 dedup, period detection, account matching with a confidence score, reconciliation against ledger balances, and a month-by-month coverage map — but it is reachable only from the web UI. An agent could not archive a document, cite one, or check for gaps. Adds five preview MCP tools over what already exists, plus a citation grammar for values the agent writes: - upload_account_statement, list_account_statements, get_account_statement, get_statement_coverage - record_valuation, whose source citation is parsed rather than trusted: ["estimated: "] citation [" (grade: A|B|C)"]. An uncited or free-styled value is rejected at the write boundary instead of landing in the ledger looking authoritative. link and reject are deliberately not exposed. Attaching a statement to an account is the human's decision, and the vault UI is where it is made; the agent reports the suggested match and stops there. Assistant.function_classes now takes a user so preview tools stay out of the default surface. They are hidden from tools/list and not callable by name without the preference enabled, and the vault tools re-check the manager role and per-account permissions, since MCP calls never pass through a controller. Docs: the blueprint this implements, and a guide covering which side owns which layer, the vocabulary map between the two, the monthly runbook, and the gaps (non-user holders, non-statement documents, one value per date). No migrations, no API endpoints, no UI. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JFDp9HhXDeswadu4cxFojn * Address review feedback on the vault MCP tools Two non-blocking items from the review pass: Document why get_statement_coverage reads through accessible_by rather than writable_by. It reports which documents exist and writes nothing, so read access is the right bar — and tightening it would hide coverage gaps from people who can already see the figures those gaps sit behind. The comment exists so a future refactor doesn't "fix" it. Close the acknowledged verification gap with tests rather than a one-off manual check. The review noted that nothing proved a real vault payload serializes cleanly out through tools/call — vault responses are richer than the other tools' output, with nested account hashes, decimal balances, dates and a compacted hash. Two integration tests now drive the real /mcp endpoint end to end against a real AccountStatement: one listing it, one uploading bytes and reading back the SHA-256. Permanent regression coverage instead of a smoke test someone has to remember to repeat. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JFDp9HhXDeswadu4cxFojn * docs(llm-guides): replace patrimonial blueprint with its final revision Swap the embedded early draft for the authoritative final revision of the wealth + tax modelling blueprint (MIT © 2026 diegomarino): - rename the domain vocabulary: patrimonial -> wealth, fiscal -> tax (tax_data/, the tax layer, tax_runner) - add §9.5 (the intel file: shape, generation, and the capture loop) - tighten worked examples down to placeholders - add the MIT header; keep the in-repo NOTE block (adapted to the new vocabulary) and the filename untouched so cross-links don't break * docs(llm-guides): align agent-harness guide with blueprint + fix reconcile semantics Follow the blueprint rename (patrimonial -> wealth, fiscal -> tax, fiscal_data/ -> tax_data/, "Phase 7 (fiscal layer)" -> "(the tax layer)") so the two docs stop disagreeing on vocabulary. Correct the reconciliation mapping, which conflated two different invariants: - blueprint reconcile-or-abort (§7 pass 3) is parse-integrity (parsed parts == the document's own printed total); Sure's reconciliation_checks is ledger agreement (statement balances vs the ledger). Sure has no parse-integrity check and never aborts. - opening_balance / closing_balance are user-entered, not auto-extracted, so over MCP reconciliation is "unavailable" until a human fills them. - tolerance differs: blueprint 1.00/account-period vs Sure's fixed 0.01. State in the ownership table, the invariants section, the vocabulary map and the monthly runbook that parse-integrity and the abort belong to the harness extractor. * Correct the vault tools' reconciliation claims and citation parsing Review findings from @diegomarino, all verified against the code before changing anything. The reconciliation claim was the serious one. get_account_statement told agents the checks were "the trustworthy part" and returned "the balances read off it" — but nothing reads balances off a document. MetadataDetector never touches them and create_from_prepared_upload! never sets them; they are user-editable fields in the Statement Vault UI. So a statement archived over MCP always came back with an empty check list, which an agent could easily read as "the document agrees with the ledger" when it means "nobody has entered the figures". The description now says so, and the payload carries a reconciliation_note spelling it out for anything reading only the JSON. Also noted that these checks are ledger agreement, not parse integrity: nothing here verifies a document's parts sum to its printed total. Provenance::Citation had two patterns disagreeing about spacing. GRADE_SUFFIX allowed "(grade:A)" but FORMAT required exactly one space, so that citation passed the pre-check and then parsed as ungraded with the grade swallowed into the text — silently discarding the reliability the caller supplied, which is the one thing this parser exists to prevent. list_account_statements downcases content_sha256 before querying. The column is constrained to lowercase hex, so uppercase input could never match, and an agent would read the empty result as "not archived" and upload a duplicate. Its period filters are renamed overlapping_from / overlapping_until, since they match on overlap and the old names claimed otherwise to anyone reading the schema without the descriptions. has_more now explains that there is no cursor and the way forward is a bigger limit or narrower filters. record_valuation no longer overwrites the entry's notes. Re-recording a date would destroy a note a person had written there. Nothing is removed now: an identical citation is a no-op, a changed one is appended, and the trail of what was cited when survives. Detecting "did this tool write that line?" is not possible — almost any prose parses as a valid ungraded citation — so the code does not guess. Minor: accept urlsafe base64 on upload, and explain in the code why record_valuation checks the account ACL rather than the vault manager role, so nobody "tightens" it into the wrong permission later. Tests cover each: the grade-spacing cases both ways, uppercase SHA lookup, overlap window boundaries, note preservation and no-stacking, the unavailable reconciliation note appearing and disappearing, and — per the review — that the download URL's signed id actually expires, rather than trusting the description's claim. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JFDp9HhXDeswadu4cxFojn * Repair a bad merge in the MCP controller test The merge of main spliced the incoming `tools/call executes update_transaction` test into the middle of the upload round-trip test, before its closing `end`. That left the file one `end` short, so it did not parse — taking out both `ci / lint` (Lint/Syntax) and `ci / test_unit` (the whole file failed to load). Restores the missing `end`. Both tests are kept as their authors wrote them; nothing else changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JFDp9HhXDeswadu4cxFojn * docs: use wealth history wording (#2885) * Stop the vault tools promising verification they don't perform Three findings from the automated review passes, all confirmed against the code before changing anything. The download URL was dead on arrival for the caller it was built for. Sure serves stored files through Active Storage controllers that config/initializers/active_storage_authorization.rb gates on `viewable_by?(Current.user)` — a signed-in browser session. An MCP client has a bearer token and no session, so following the URL would have redirected to sign-in. Removed it rather than leaving a link that cannot work, and the description now points at search_family_files or the vault UI. Coverage called a month `covered` when a document merely existed. An unreconciled statement is not mismatched, so it took the `covered` branch, and the payload carried nothing to correct the reading — the same "advertised verification that never happened" bug fixed last round in get_account_statement, in a second place. Months now carry their own reconciliation_status, and the description says covered means presence, not agreement. Listing filtered visibility after limiting. Beyond underfilling a page, with no cursor and a 100-row cap an accessible statement behind enough newer invisible ones was unreachable. Visibility now lives in the query, mirroring viewable_by? for a statement manager. Also: rescue unexpected upload failures into a tool error instead of a raw exception string, derive the documented size limit from MAX_FILE_SIZE, list every coverage status in mcp.md, and cover the failed-reconciliation and base64-normalisation branches. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JFDp9HhXDeswadu4cxFojn * Keep storage exception detail out of the MCP response The upload_failed message interpolated the exception text, which crosses out to an external agent. A storage failure can carry bucket names, object keys, paths or request details, so the agent now gets a fixed message and the exception stays in the server log. The test asserts the absence of detail rather than pinning the leaked string into the contract. Also fixes a test that did not test what it claimed: the urlsafe-base64 case used a fixture encoding to plain base64, so it exercised the padding branch and never the "-_" translation. It now uses content whose encoding contains both characters and asserts that up front. Renames "rejects content that decodes to zero bytes" to "rejects blank content", which is what it actually covers — Base64.strict_encode64("") is "", which is blank and returns before the decoder runs, so invalid_content is correct and empty_file is not reachable from this path. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JFDp9HhXDeswadu4cxFojn * docs: align wealth blueprint review feedback * Correct the harness runbook: parse before publishing The guide told an implementer to archive each document to Sure first and work from there. That strands them. Sure never returns a document's bytes over MCP — Active Storage serves stored files only to a signed-in browser session — and there is no text fallback either, because statements archived through upload_account_statement never enter the vector store, so search_family_files cannot see them. A statement in Sure is metadata to an agent and nothing more. That blocks exactly three blueprint steps, all of them operating on bank and broker statements: the extractors, the parts-vs-printed-total check, and the glyph decoder. Everything else it parses — tax returns, capital accounts, annual accounts — the harness already holds locally. So the order inverts: the harness ingests into its own vault, extracts there with the whole file in reach, and publishes to Sure afterwards. This restores principle 8 rather than bending it — the recurring pipeline reads from the canonical store, and treating Sure as canonical forced a re-fetch the architecture never sanctioned. Both sides hash the same bytes, so the SHA-256 verifies Sure holds the identical document without moving it. Writes down the two consequences: a statement uploaded straight into Sure's UI can be known but never parsed (reliability C or PENDING until a copy reaches the harness), and neither vault backs up the other. Also drops a stale tools-table row still advertising the 15-minute download URL removed earlier, corrects get_account_statement's description where it suggested search_family_files as a fallback it cannot be, and disambiguates "the vault" in the MCP tool table, which is what misled me in the first place. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01JFDp9HhXDeswadu4cxFojn --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: diegomarino <diegomarino@users.noreply.github.com> Co-authored-by: Sure Admin (bot) <sure-admin@splashblot.com>	2026-08-04 23:33:01 +02:00
Juan José Mata	c6eb7cdeed	Revert "Refactor application workflows and update test coverage" ... This reverts commit 565e049f89.	2026-07-23 13:39:46 -07:00
Juan José Mata	565e049f89	Refactor application workflows and update test coverage	2026-07-21 22:01:59 -07:00
3 people	f62c805c69	docs: clarify local LLM context window tuning (#2661) ... * docs(ai): document local LLM context window tuning * Update compose.example.yml Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Juan José Mata <jjmata@jjmata.com> --------- Signed-off-by: Juan José Mata <jjmata@jjmata.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>	2026-07-17 07:10:29 +02:00
Josh	60d9a70aff	Refresh Pipelock integration for v2.8 receipts (#2406) ... * chore(pipelock): refresh integration for v2.8 receipts * Clarify Pipelock receipt key mounts	2026-06-19 17:16:38 +02:00
JoshandJuan José Mata	ca895416a4	chore(helm): bump pipelock to 2.5.0 and surface 2.5 config (#1913) ... * chore(helm): bump pipelock to 2.5.0 and surface 2.5 config Bumps pipelock.image.tag from 2.2.0 to 2.5.0 and exposes the most relevant 2.5 features as structured Helm values: - pipelock.requestBodyScanning: scan outbound bodies and sensitive headers for prompt-injection and DLP payloads. Disabled by default; roll out with action=warn before flipping to block. - pipelock.healthWatchdog: structured config for the wedge-detection watchdog with an exposeSubsystems toggle for /health detail. - pipelock.mcpToolPolicy.rules: structured values for rendering mcp_tool_policy.rules including redirect-profile references. Also fixes a latent config-validation regression: pipelock 2.x rejects an enabled mcp_tool_policy with no rules, but the chart previously defaulted to enabled=true with an empty rules list, which hard-fails 'pipelock check'. The default is now enabled=false; operators must explicitly enable and provide at least one rule. Refreshes README, CHANGELOG, docs/hosting/pipelock.md, docs/hosting/ai.md, compose example pin comment, and pipelock.example.yaml to call out 2.5 highlights (Audit Packet v0 verifiers, SPIFFE-strict envelopes, scanner attribution on MCP block receipts, pipelock doctor). Also fixes a stale docs/hosting/mcp.md reference to the removed compose.example.pipelock.yml. * chore(helm): fail helm template when mcp_tool_policy enabled with no rules Adds a guard in asserts.tpl so an operator who sets pipelock.mcpToolPolicy.enabled=true without populating pipelock.mcpToolPolicy.rules gets a clear render-time error instead of a container crash-loop with the pipelock validation message. Per CodeRabbit feedback on #1913. * Versions --------- Co-authored-by: Juan José Mata <jjmata@jjmata.com>	2026-05-24 13:50:44 +02:00
Dreamandsokiee	6d22514c01	feat(vector-store): Implement pgvector adapter for self-hosted RAG (#1211) ... * Add conditional migration for vector_store_chunks table Creates the pgvector-backed chunks table when VECTOR_STORE_PROVIDER=pgvector. Enables the vector extension, adds store_id/file_id indexes, and uses vector(1024) column type for embeddings. * Add VectorStore::Embeddable concern for text extraction and embedding Shared concern providing extract_text (PDF via pdf-reader, plain-text as-is), paragraph-boundary chunking (~2000 chars, ~200 overlap), and embed/embed_batch via OpenAI-compatible /v1/embeddings endpoint using Faraday. Configurable via EMBEDDING_MODEL, EMBEDDING_URI_BASE, with fallback to OPENAI_* env vars. * Implement VectorStore::Pgvector adapter with raw SQL Replaces the stub with a full implementation using ActiveRecord::Base.connection with parameterized binds. Supports create_store, delete_store, upload_file (extract+chunk+embed+insert), remove_file, and cosine-similarity search via the <=> operator. * Add registry test for pgvector adapter selection * Configure pgvector in compose.example.ai.yml Switch db image to pgvector/pgvector:pg16, add VECTOR_STORE_PROVIDER, EMBEDDING_MODEL, and EMBEDDING_DIMENSIONS env vars, and include nomic-embed-text in Ollama's pre-loaded models. * Update pgvector docs from scaffolded to ready Document env vars, embedding model setup, pgvector Docker image requirement, and Ollama pull instructions. * Address PR review feedback - Migration: remove env guard, use pgvector_available? check so it runs on plain Postgres (CI) but creates the table on pgvector-capable servers. Add NOT NULL constraints on content/embedding/metadata, unique index on (store_id, file_id, chunk_index). - Pgvector adapter: wrap chunk inserts in a DB transaction to prevent partial file writes. Override supported_extensions to match formats that extract_text can actually parse. - Embeddable: add hard_split fallback for paragraphs exceeding CHUNK_SIZE to avoid overflowing embedding model token limits. * Bump schema version to include vector_store_chunks migration CI uses db:schema:load which checks the version — without this bump, the migration is detected as pending and tests fail to start. * Update 20260316120000_create_vector_store_chunks.rb --------- Co-authored-by: sokiee <sokysrm@gmail.com>	2026-03-20 17:01:31 +01:00
3 people	a0b1029ba9	Documentation for review AI Assistant features, MCP and API additions (#1168) ... * Create MCP server endpoint documentation * Add Assistant Architecture section to AI documentation * Add Users API documentation for account reset and delete endpoints * Document Pipelock CI security scanning in contributing guide * fix: correct scope and error codes in Users API documentation * Exclude `docs/hosting/ai.md` from Pipelock scan --------- Co-authored-by: askmanu[bot] <192355599+askmanu[bot]@users.noreply.github.com> Co-authored-by: Juan José Mata <jjmata@jjmata.com>	2026-03-16 18:24:28 +01:00
LPW	ca8f04040f	Expand AI docs: external assistant, MCP, architecture, troubleshooting (#1115) ... * Expand AI docs: architecture, MCP, external assistant setup, troubleshooting - Add architecture overview explaining two independent AI pipelines (chat assistant vs auto-categorization) - Document MCP callback endpoint (JSON-RPC 2.0, auth, available tools) - Add OpenClaw gateway configuration example - Add Kubernetes network policy guidance (targetPort vs servicePort) - Add Pipelock notes (mcpToolPolicy, NO_PROXY behavior) - Add troubleshooting for "Failed to generate response" with external assistant - Fix stale function list (4 tools -> 7) - Fix incorrect env-vs-UI precedence statement - Fix em-dashes in existing content * Fix troubleshooting curl to use pod env vars Use sh -c so $EXTERNAL_ASSISTANT_TOKEN and $EXTERNAL_ASSISTANT_URL expand inside the pod, not on the local shell.	2026-03-04 11:26:43 +01:00
LPW	84bfe5b7ab	Add external AI assistant with Pipelock security proxy (#1069) ... * feat(helm): add Pipelock ConfigMap, scanning config, and consolidate compose - Add ConfigMap template rendering DLP, response scanning, MCP input/tool scanning, and forward proxy settings from values - Mount ConfigMap as /etc/pipelock/pipelock.yaml volume in deployment - Add checksum/config annotation for automatic pod restart on config change - Gate HTTPS_PROXY/HTTP_PROXY env injection on forwardProxy.enabled (skip in MCP-only mode) - Use hasKey for all boolean values to prevent Helm default swallowing false - Single source of truth for ports (forwardProxy.port/mcpProxy.port) - Pipelock-specific imagePullSecrets with fallback to app secrets - Merge standalone compose.example.pipelock.yml into compose.example.ai.yml - Add pipelock.example.yaml for Docker Compose users - Add exclude-paths to CI workflow for locale file false positives * Add external assistant support (OpenAI-compatible SSE proxy) Allow self-hosted instances to delegate chat to an external AI agent via an OpenAI-compatible streaming endpoint. Configurable per-family through Settings UI or ASSISTANT_TYPE env override. - Assistant::External::Client: SSE streaming HTTP client (no new gems) - Settings UI with type selector, env lock indicator, config status - Helm chart and Docker Compose env var support - 45 tests covering client, config, routing, controller, integration * Add session key routing, email allowlist, and config plumbing Route to the actual OpenClaw session via x-openclaw-session-key header instead of creating isolated sessions. Gate external assistant access behind an email allowlist (EXTERNAL_ASSISTANT_ALLOWED_EMAILS env var). Plumb session_key and allowedEmails through Helm chart, compose, and env template. * Add HTTPS_PROXY support to External::Client for Pipelock integration Net::HTTP does not auto-read HTTPS_PROXY/HTTP_PROXY env vars (unlike Faraday). Explicitly resolve proxy from environment in build_http so outbound traffic to the external assistant routes through Pipelock's forward proxy when enabled. Respects NO_PROXY for internal hosts. * Add UI fields for external assistant config (Setting-backed with env fallback) Follow the same pattern as OpenAI settings: database-backed Setting fields with env var defaults. Self-hosters can now configure the external assistant URL, token, and agent ID from the browser (Settings > Self-Hosting > AI Assistant) instead of requiring env vars. Fields disable when the corresponding env var is set. * Improve external assistant UI labels and add help text Change placeholder to generic OpenAI-compatible URL pattern. Add help text under each field explaining where the values come from: URL from agent provider, token for authentication, agent ID for multi-agent routing. * Add external assistant docs and fix URL help text Add External AI Assistant section to docs/hosting/ai.md covering setup (UI and env vars), how it works, Pipelock security scanning, access control, and Docker Compose example. Drop "chat completions" jargon from URL help text. * Harden external assistant: retry logic, disconnect UI, error handling, and test coverage - Add retry with backoff for transient network errors (no retry after streaming starts) - Add disconnect button with confirmation modal in self-hosting settings - Narrow rescue scope with fallback logging for unexpected errors - Safe cleanup of partial responses on stream interruption - Gate ai_available? on family assistant_type instead of OR-ing all providers - Truncate conversation history to last 20 messages - Proxy-aware HTTP client with NO_PROXY support - Sanitize protocol to use generic headers (X-Agent-Id, X-Session-Key) - Full test coverage for streaming, retries, proxy routing, config, and disconnect * Exclude external assistant client from Pipelock scan-diff False positive: `@token` instance variable flagged as "Credential in URL". Temporary workaround until Pipelock supports inline suppression. * Address review feedback: NO_PROXY boundary fix, SSE done flag, design tokens - Fix NO_PROXY matching to require domain boundary (exact match or .suffix), case-insensitive. Prevents badexample.com matching example.com. - Add done flag to SSE streaming so read_body stops after [DONE] - Move MAX_CONVERSATION_MESSAGES to class level - Use bg-success/bg-destructive design tokens for status indicators - Add rationale comment for pipelock scan exclusion - Update docs last-updated date * Address second round of review feedback - Allowlist email comparison is now case-insensitive and nil-safe - Cap SSE buffer at 1 MB to prevent memory blowup from malformed streams - Don't expose upstream HTTP response body in user-facing errors (log it instead) - Fix frozen string warning on buffer initialization - Fix "builtin" typo in docs (should be "built-in") * Protect completed responses from cleanup, sanitize error messages - Don't destroy a fully streamed assistant message if post-stream metadata update fails (only cleanup partial responses) - Log raw connection/HTTP errors internally, show generic messages to users to avoid leaking network/proxy details - Update test assertions for new error message wording * Fix SSE content guard and NO_PROXY test correctness Use nil check instead of present? for SSE delta content to preserve whitespace-only chunks (newlines, spaces) that can occur in code output. Fix NO_PROXY test to use HTTP_PROXY matching the http:// client URL so the proxy resolution and NO_PROXY bypass logic are actually exercised. * Forward proxy credentials to Net::HTTP Pass proxy_uri.user and proxy_uri.password to Net::HTTP.new so authenticated proxies (http://user:pass@host:port) work correctly. Without this, credentials parsed from the proxy URL were silently dropped. Nil values are safe as positional args when no creds exist. * Update pipelock integration to v0.3.1 with full scanning config Bump Helm image tag from 0.2.7 to 0.3.1. Add missing security sections to both the Helm ConfigMap and compose example config: mcp_tool_policy, mcp_session_binding, and tool_chain_detection. These protect the /mcp endpoint against tool injection, session hijacking, and multi-step exfiltration chains. Add version and mode fields to config files. Enable include_defaults for DLP and response scanning to merge user patterns with the 35 built-in patterns. Remove redundant --mode CLI flag from the Helm deployment template since mode is now in the config file.	2026-03-03 15:47:51 +01:00
Juan José Mata	4e4ca916a1	Update backend table with status and requirements ... Clarify status of non-OpenAI vector store Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>	2026-02-11 15:59:12 +01:00
3 people	9e57954a99	Add Family vector search function call / support for document vault (#961) ... * Add SearchFamilyImportedFiles assistant function with vector store support Implement per-Family document search using OpenAI vector stores, allowing the AI assistant to search through uploaded financial documents (tax returns, statements, contracts, etc.). The architecture is modular with a provider- agnostic VectorStoreConcept interface so other RAG backends can be added. Key components: - Assistant::Function::SearchFamilyImportedFiles - tool callable from any LLM - Provider::VectorStoreConcept - abstract vector store interface - Provider::Openai vector store methods (create, upload, search, delete) - Family::VectorSearchable concern with document management - FamilyDocument model for tracking uploaded files - Migration adding vector_store_id to families and family_documents table https://claude.ai/code/session_01TSkKc7a9Yu2ugm1RvSf4dh * Extract VectorStore adapter layer for swappable backends Replace the Provider::VectorStoreConcept mixin with a standalone adapter architecture under VectorStore::. This cleanly separates vector store concerns from the LLM provider and makes it trivial to swap backends. Components: - VectorStore::Base — abstract interface (create/delete/upload/remove/search) - VectorStore::Openai — uses ruby-openai gem's native vector_stores.search - VectorStore::Pgvector — skeleton for local pgvector + embedding model - VectorStore::Qdrant — skeleton for Qdrant vector DB - VectorStore::Registry — resolves adapter from VECTOR_STORE_PROVIDER env - VectorStore::Response — success/failure wrapper (like Provider::Response) Consumers updated to go through VectorStore.adapter: - Family::VectorSearchable - Assistant::Function::SearchFamilyImportedFiles - FamilyDocument Removed: Provider::VectorStoreConcept, vector store methods from Provider::Openai https://claude.ai/code/session_01TSkKc7a9Yu2ugm1RvSf4dh * Add Vector Store configuration docs to ai.md Documents how to configure the document search feature, covering all three supported backends (OpenAI, pgvector, Qdrant), environment variables, Docker Compose examples, supported file types, and privacy considerations. https://claude.ai/code/session_01TSkKc7a9Yu2ugm1RvSf4dh * No need to specify `imported` in code * Missed a couple more places * Tiny reordering for the human OCD * Update app/models/assistant/function/search_family_files.rb Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: Juan José Mata <jjmata@jjmata.com> * PR comments * More PR comments --------- Signed-off-by: Juan José Mata <jjmata@jjmata.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>	2026-02-11 15:22:56 +01:00
4 people	6f8858b1a6	feat/Add AI-Powered Bank Statement Import (step 1, PDF import & analysis) (#808) ... * feat: Add PDF import with AI-powered document analysis This enhances the import functionality to support PDF files with AI-powered document analysis. When a PDF is uploaded, it is processed by AI to: - Identify the document type (bank statement, credit card statement, etc.) - Generate a summary of the document contents - Extract key metadata (institution, dates, balances, transaction count) After processing, an email is sent to the user asking for next steps. Key changes: - Add PdfImport model for handling PDF document imports - Add Provider::Openai::PdfProcessor for AI document analysis - Add ProcessPdfJob for async PDF processing - Add PdfImportMailer for user notification emails - Update imports controller to detect and handle PDF uploads - Add PDF import option to the new import page - Add i18n translations for all new strings - Add comprehensive tests for the new functionality * Add bank statement import with AI extraction - Create ImportBankStatement assistant function for MCP - Add BankStatementExtractor with chunked processing for small context windows - Register function in assistant configurable - Make PdfImport#pdf_file_content public for extractor access - Increase OpenAI request timeout to 600s for slow local models - Increase DB connection pool to 20 for concurrent operations Tested with M-Pesa bank statement via remote Ollama (qwen3:8b): - Successfully extracted 18 transactions - Generated CSV and created TransactionImport - Works with 3000 char chunks for small context windows * Add pdf-reader gem dependency The BankStatementExtractor uses PDF::Reader to parse bank statement PDFs, but the gem was not properly declared in the Gemfile. This would cause NameError in production when processing bank statements. Added pdf-reader ~> 2.12 to Gemfile dependencies. * Fix transaction deduplication to preserve legitimate duplicates The previous deduplication logic removed ALL duplicate transactions based on [date, amount, name], which would drop legitimate same-day duplicates like multiple ATM withdrawals or card authorizations. Changed to only deduplicate transactions that appear in consecutive chunks (chunking artifacts) while preserving all legitimate duplicates within the same chunk or non-adjacent chunks. * Refactor bank statement extraction to use public provider method Address code review feedback: - Add public extract_bank_statement method to Provider::Openai - Remove direct access to private client via send(:client) - Update ImportBankStatement to use new public method - Add require 'set' to BankStatementExtractor - Remove PII-sensitive content from error logs - Add defensive check for nil response.error - Handle oversized PDF pages in chunking logic - Remove unused process_native and process_generic methods - Update email copy to reflect feature availability - Add guard for nil document_type in email template - Document pdf-reader gem rationale in Gemfile Tested with both OpenAI (gpt-4o) and Ollama (qwen3:8b): - OpenAI: 49 transactions extracted in 30s - Ollama: 40 transactions extracted in 368s - All encapsulation and error handling working correctly * Update schema.rb with ai_summary and document_type columns * Address PR #808 review comments - Rename :csv_file to :import_file across controllers/views/tests - Add PDF test fixture (sample_bank_statement.pdf) - Add supports_pdf_processing? method for graceful degradation - Revert unrelated database.yml pool change (600->3) - Remove month_start_day schema bleed from other PR - Fix PdfProcessor: use .strip instead of .strip_heredoc - Add server-side PDF magic byte validation - Conditionally show PDF import option when AI provider available - Fix ProcessPdfJob: sanitize errors, handle update failure - Move pdf_file attachment from Import to PdfImport - Document deduplication logic limitations - Fix ImportBankStatement: catch specific exceptions only - Remove unnecessary require 'set' - Remove dead json_schema method from PdfProcessor - Reduce default OpenAI timeout from 600s to 60s - Fix nil guard in text mailer template - Add require 'csv' to ImportBankStatement - Remove Gemfile pdf-reader comment * Fix RuboCop indentation in ProcessPdfJob * Refactor PDF import check to use model predicate method Replace is_a?(PdfImport) type check with requires_csv_workflow? predicate that leverages STI inheritance for cleaner controller logic. * Fix missing 'unknown' locale key and schema version mismatch - Add 'unknown: Unknown Document' to document_types locale - Fix schema version to match latest migration (2026_01_24_180211) * Document OPENAI_REQUEST_TIMEOUT env variable Added to .env.local.example and docs/hosting/ai.md * Rename ALLOWED_MIME_TYPES to ALLOWED_CSV_MIME_TYPES for clarity * Add comment explaining requires_csv_workflow? predicate * Remove redundant required_column_keys from PdfImport Base class already returns [] by default * Add ENV toggle to disable PDF processing for non-vision endpoints OPENAI_SUPPORTS_PDF_PROCESSING=false can be used for OpenAI-compatible endpoints (e.g., Ollama) that don't support vision/PDF processing. * Wire up transaction extraction for PDF bank statements - Add extracted_data JSONB column to imports - Add extract_transactions method to PdfImport - Call extraction in ProcessPdfJob for bank statements - Store transactions in extracted_data for later review * Fix ProcessPdfJob retry logic, sanitize and localize errors - Allow retries after partial success (classification ok, extraction failed) - Log sanitized error message instead of raw message to avoid data leakage - Use i18n for user-facing error messages * Add vision-capable model validation for PDF processing * Fix drag-and-drop test to use correct field name csv_file * Schema bleedover from another branch * Fix drag-drop import form field name to match controller * Add vision capability guard to process_pdf method --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com> Co-authored-by: Juan José Mata <jjmata@jjmata.com>	2026-01-30 20:44:25 +01:00
eureka928	02c71bca0a	Add AI Cache Management documentation ... Document the AI cache reset feature including what it does, when to use it, how to reset via UI, and cost implications.	2026-01-26 10:41:14 +01:00
Blaž Dular	8972cb59f0	docs: add env variable for ai debug to docs (#494)	2025-12-23 19:57:32 +01:00
soky srmandJuan José Mata	da114b5b3d	Update ai.md (#263) ... * Update ai.md Change some deprecated models Signed-off-by: soky srm <sokysrm@gmail.com> * Fix typo in AI model description Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> --------- Signed-off-by: soky srm <sokysrm@gmail.com> Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>	2025-10-30 23:38:14 +01:00
Juan José Mata	f18c11c7ac	Update AI model recommendations section ... Added a caution note about model support and testing approach. Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>	2025-10-29 18:52:26 +01:00
Juan José Mata	3f4330eea8	Update AI assistant documentation with version caution ... Added caution note regarding AI assistant support versions. Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>	2025-10-29 18:44:58 +01:00
4 people	a8f318c3f9	Fix "Messages is invalid" error for Ollama/custom LLM providers and add comprehensive AI documentation (#225) ... * Add comprehensive AI/LLM configuration documentation * Fix Chat.start! to use default model when model is nil or empty * Ensure all controllers use Chat.default_model for consistency * Move AI doc inside `hosting/` * Probably too much error handling --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>	2025-10-24 12:04:19 +02:00