76 Commits

Author	SHA1	Message	Date
Oscar	c6a240a183	feat(redbark): add australian bank sync (redbark) (#2794) ... * add redbark provider integration - per family api key provider, built like the lunchflow integration - syncs accounts, balances and transactions from api.redbark.com - account setup flow, settings panel, locales and routes - tests and fixtures * harden redbark integration based on prior provider pr feedback - use DebugLogEntry.capture for sync/import/unlink failures - retry 429s and 5xxs with backoff, raise on page cap instead of truncating - keep raw response bodies out of logs and errors - not null constraints on account columns, migration base 7.2 - persist ignored flag for skipped accounts so they stop nagging setup - validate api key on every save, re-arm status on key rotation - destroy aborts if unlink fails, atomic account create and link - require_admin on mutating actions, see_other on error redirects - single grouped query for item account counts - i18n default connection name, blank password field value - controller and provider tests * fix issues found in second review sweep - add missing syncable scope, without it every family sync raises - kick off a sync on connection create and on key rotation - setup dialog fetches accounts inline for fresh connections and shows api errors - skip balance write when no balance has been fetched yet, never anchor a false zero - exclude stale and non banking accounts from the batched balances call, per account fallback if the batch is rejected - detect the server row ceiling and empty pages instead of silently truncating history - user sync start date only governs the initial backfill, incremental after that - fetch connections before the per account loop so auth errors propagate once - drop untemplated index/show/new/edit routes and dead preload/link_accounts actions - stable dom id on the settings panel so repeat turbo replaces keep working * skip brokerage connections, found in live testing - the transactions endpoint 400s for brokerage connections, they belong to /v1/trades - only import accounts from banking and documents connections - guard transaction fetches for any legacy linked non banking account * address review feedback - treat the truncation header as a pagination signal: split the date window and refetch instead of failing the account - prune stale pending rows from the snapshot so settled pendings cant come back as duplicates - block linking a sure account that already has another provider feed - count setup failures separately from skips and surface an error instead of "all skipped" - add not nulls on redbark_items name and api key - enqueue the destroy job after the flag commits, not inside the transaction - swap bg-gray-400 for bg-surface-inset, drop amounts from info logs, remove i18n default fallbacks - tests for window splitting, pending pruning and encrypted payload round trip * fix issues from convention review - benign skips (unlinked account, blank id, unparseable rows) no longer count as failures, tracked separately so a clean batch reports success - currency parsing goes through extract_currency so hash shaped payloads resolve instead of falling to the default - merchant ids use truncated sha256 instead of md5 - debug log entries for import failures and account sync scheduling failures * bound the raw transactions snapshot to the fetch window - trim raw_transactions_payload to the current fetch window on merge, same as brex - keep rows without a parseable date, drop settled pendings as before - surface skipped rows in the aggregate debug log entry with imported/skipped counts	2026-07-26 07:40:25 +02:00
kai392	6e2cbb727a	fix(ai): prevent Anthropic chat crash on no-argument tool calls (#2755) ... * fix: prevent Anthropic chat crash on no-argument tool calls When the model streams a tool_use block with no arguments (e.g. get_categories), the accumulated input arrives as an empty string. The Anthropic chat parser passed that empty string straight through as function_args, and Assistant::FunctionToolCaller then called JSON.parse("") — which raises "unexpected end of input", surfaces as a Provider::Anthropic::Error, and kills the whole assistant turn. Fix at the source by normalizing empty/nil tool input to an empty JSON object in the parser, plus a defensive guard in FunctionToolCaller so any provider that emits blank arguments cannot crash a turn. Fixes #2722 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test: assert nil-args result as Hash to match jsonb function_result Addresses Codex P1 / CodeRabbit review: EchoFunction returns the parsed params Hash and ToolCall::Function stores it in a jsonb column, so the nil-arguments test must assert the Hash directly instead of JSON.parse. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-26 04:47:15 +02:00
RealDiligent	d61ee3abea	fix: omit SimpleFin pending param when pending transactions are disabled (#2796) ... The SimpleFIN protocol only defines pending=1 (include pending); pending transactions are excluded by default when the param is absent. Bridges presence-check the param, so the pending=0 we sent when the 'Include pending transactions' setting (or SIMPLEFIN_INCLUDE_PENDING=0) was disabled behaved exactly like pending=1, making the setting a no-op — pending transactions kept being downloaded, causing pending/posted duplicates and churn. Omit the pending query param entirely unless pending is enabled, per the spec. Fixes #2440 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-07-26 01:56:42 +02:00
kianrafiee	d267288a45	Add RentCast & Realie integration for automatic property data and valuations (#2727) ... * Add RentCast and Realie AVM providers for property valuation Adds Automated Valuation Model (AVM) provider support so self-hosted users can create property accounts from a US address lookup instead of entering details manually: - Provider::Rentcast and Provider::Realie clients, each fetching the property record (type, year built, square footage) and value estimate in a single API request, registered under a new :property_valuations registry concept - API key fields (encrypted, ENV-overridable) with monthly usage display in Settings > Self-Hosting under "Property Valuation Providers" - New property flow: when a provider key is configured, the method selector offers "Add via RentCast/Realie" alongside manual entry; the lookup form reuses the manual flow's localized address fields and creates the account active with fetched attributes, valuation as balance, and the address saved - SyncPropertyValuationsJob refreshes linked property valuations once a day (never hourly) via config/schedule.yml; each provider enforces its monthly request cap (RentCast 50, Realie 25) with a calendar-month counter shared between creation lookups and refreshes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Allow ENV override of AVM provider monthly request caps The RentCast (50/month) and Realie (25/month) budgets default to the free tier limits but can now be raised for paid plans via RENTCAST_MAX_REQUESTS_PER_MONTH / REALIE_MAX_REQUESTS_PER_MONTH, mirroring the AlphaVantage and Tiingo request limit overrides. The settings descriptions and usage display reflect the effective cap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review feedback: durable request counters, safer refresh job - Move monthly AVM request counters from Rails.cache to a new provider_request_counts table with atomic upsert increments, so the hard budget caps survive cache eviction, restarts, and Redis flushes - SyncPropertyValuationsJob: only mark a property synced when the balance update succeeds (wrapped in a transaction), process stalest valuations first so a tight budget goes where it matters, and report failures via DebugLogEntry.capture instead of Rails.logger - Realie: reject lookups whose returned city/ZIP contradict the entered address (street+state queries can match the wrong city); document that numeric use codes are unpublished and leave the subtype unset - Add Faraday open/request timeouts to both provider clients - Localize all user-facing provider error messages (config/locales/models/provider/en.yml) - Add a check constraint restricting properties.avm_provider to known providers - Use the min-h-80 scale token instead of min-h-[320px] - Tests: exact-argument expectations on the lookup stub, provider stubs in the no-provider test, RentCast/Realie API key update tests, ProviderRequestCount unit tests, failed-balance regression test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Stub AVM provider registry lookups in settings system test Settings::HostingsController#show now resolves the RentCast and Realie providers for usage display; the system test's partial get_provider stubbing treats the new lookups as unexpected invocations without these. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address second-round review: candidate scan, provider reuse, shared row partial - Realie: when the address lookup returns multiple candidates, pick the first one consistent with the entered city/ZIP instead of judging only the first array element; the location-mismatch error now only fires when no candidate matches - SyncPropertyValuationsJob: resolve one provider instance per key for the whole run, so the per-instance request throttle actually spaces requests across properties instead of resetting on each iteration - Extract the AVM method selector row into a shared partial so the manual and provider options render one shape Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add retry middleware and sync index from review feedback - Both AVM clients now retry transient connection failures (same Faraday retry config as Tiingo) so a network blip doesn't burn one of the tight monthly budget's requests - Partial index on properties(avm_provider, avm_last_synced_on) backing the daily sync job's filter and stalest-first ordering Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Key the AVM sync index on avm_last_synced_on The daily job orders by avm_last_synced_on ASC NULLS FIRST across all AVM-linked properties, so leading the partial index with avm_provider prevented it from serving the sort. Rekeyed on avm_last_synced_on with matching null ordering; the partial predicate still covers the filter. Amended in place since the migration is unmerged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Map Realie numeric use codes; skip incomplete addresses in sync job - Realie documents its numeric use codes at docs.realie.ai/api-reference/feature-key (earlier skip reason was wrong — the table exists). Map the residential/land/agricultural codes to Property subtypes, leaving codes without a subtype equivalent (e.g. 1006 mobile/manufactured) unset like the RentCast mapping. - The daily job now skips properties whose address is missing a street or state before resolving a provider, logging a warn-level DebugLogEntry, so a malformed address can't burn a monthly-budget request every day. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Guard currency mismatches in refresh; reset AVM keys in test teardown - The daily job now skips (with a warn-level DebugLogEntry) properties whose account currency no longer matches the provider's valuation currency, checked via the concept's valuation_currency before spending a request — writing a USD valuation into a re-currencied account would corrupt the balance - Hostings controller test teardown now clears rentcast_api_key and realie_api_key alongside the other cached global settings Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Validate AVM lookup inputs before spending a provider request The form marks name and address fields required, but a forged or JS-less submission bypasses that and would burn one of the tight monthly-budget requests on a lookup that can't produce a property. Property::AvmImport now validates name and the full address locally (localized error) before calling the provider. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add preview-and-confirm step to the AVM property lookup RentCast's AVM endpoint is location-based: a plausible but nonexistent address still geocodes and returns an area-derived estimate with no property record behind it. Instead of silently creating an account from that, the lookup now shows what the provider returned — type, year built, area (each "Unknown" when missing), and the estimated value — with an explicit notice when no property record was found, and only the user's confirmation creates the account. The confirm step reuses the fetched data via the form (sanitized server-side), so the flow still costs exactly one provider request. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Require a complete US address before daily valuation refresh The job's guard only required street and state, but a later-blanked city or ZIP silently disables the Realie wrong-city check (blank entered fields count as "no mismatch"), so a refresh could accept another property's valuation. The guard now mirrors the import-time completeness check and also skips addresses edited to a non-US country. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Sign the AVM preview payload; blur valuation in privacy mode - The confirm step now rebuilds everything from a signed, 1-hour message-verifier token generated at lookup time, proving the lookup (and its counted provider request) actually ran — a direct confirm POST with fabricated data can no longer create provider-linked properties that the daily refresh job would then spend quota on. Forged/expired tokens re-render the lookup form with a clear error. - The preview's market value now carries the privacy-sensitive class so privacy mode blurs it like other account amounts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Treat zero Realie valuations as absent; bind preview token to family Review follow-ups from PR #2727: - Realie returns modelValue: 0 when it can't produce an AVM estimate; zero now falls back to the assessed market value instead of syncing a $0 balance, and an all-zero record errors out. - Monetary values now parse via BigDecimal(value.to_s) rather than Float#to_d, avoiding binary float rounding artifacts in balances. - The signed AVM preview token's purpose is scoped to the family that ran the lookup, so a leaked token can't be replayed cross-family. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Refresh AVM property valuations monthly instead of daily Per @jjmata's review on #2727: valuations change slowly and providers enforce tight monthly request caps, so a daily cron adds no value. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>	2026-07-25 04:47:58 +02:00
Max Barbare	51c93649da	feat(snaptrade): replace device-flow OAuth with authorization-code + PKCE flow (#2747) ... * feat(snaptrade): replace device-flow OAuth with authorization-code + PKCE flow Squashed from 16 commits on snaptrade-oauth-apps for a clean rebase onto current upstream/main ahead of opening a PR. * fix(snaptrade): address PR #2747 review feedback on OAuth PKCE flow - Remove unreachable dead-code guard in import_latest_snaptrade_data - Guard apply_oauth_tokens! against a malformed payload missing access_token - Wrap token endpoint network errors in ApiError and retry like data calls - Remove unused Provider::Snaptrade#revoke_token! instance method - Preserve return_to/accountable_type through the SnapTrade portal callback so the account-linking flow no longer drops users back to accounts_path - Show the real absolute OAuth callback URL in self-hosted setup instructions - Refresh brakeman.ignore fingerprint for the connect redirect after the return_to/accountable_type params were added Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8SCCmKX6RphB5E73WSUQQ * fix(snaptrade): don't retry non-idempotent OAuth/API requests CodeRabbit flagged that Provider::Snaptrade retried OAuth token exchanges/refreshes and all API POST/DELETE calls (get_connection_url, delete_connection) after timeouts/connection failures. If the response is lost after SnapTrade already consumed a single-use auth code, rotated the refresh token, or applied a POST/DELETE, replaying the request either fails with invalid_grant on a token that actually succeeded, or risks duplicate side effects. Retries are now limited to GET requests; OAuth token requests and non-GET API calls translate a network failure straight into an ApiError without replay. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrrGkgSBEqhjjBmmH1fcXL * fix(snaptrade): stop querying non-deterministically encrypted token via empty-string compare CodeRabbit flagged that the syncable scope's where.not(oauth_access_token: [nil, ""]) re-encrypts "" with a random IV on every query, so the "" comparison can never match a stored ciphertext and is a silent no-op. No code path ever persists oauth_access_token as "" (only nil or a real token via apply_oauth_tokens!), so the exclusion is unnecessary -- narrowed the scope to a plain NULL check, which encryption handles transparently since nil is never encrypted. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrrGkgSBEqhjjBmmH1fcXL --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-24 22:45:44 +02:00
Juan José Mata	c6eb7cdeed	Revert "Refactor application workflows and update test coverage" ... This reverts commit 565e049f89.	2026-07-23 13:39:46 -07:00
Juan José Mata	565e049f89	Refactor application workflows and update test coverage	2026-07-21 22:01:59 -07:00
Camilo Agudelo	7657f2d08c	Improve Yahoo Finance reliability and Colombian listing support (#2738) ... * Prevent Yahoo crumb cache poisoning * Make Yahoo Finance health status rate-limit aware * Normalize Yahoo Colombian listings Map Yahoo's BVC code to XBOG so search results derive Colombia from canonical exchange metadata and price requests use the .CL suffix with a COP fallback. Spec: .scratch/normalize-yahoo-colombia-listings/spec.md * Move Yahoo health checks to background	2026-07-22 06:53:48 +02:00
Carlos Lindo	5b9fd1e1a0	fix(enable-banking): handle unavailable and overdraft balances (#2578) ... * fix(enable-banking): handle overdraft balance sync * fix(enable-banking): skip nil provider balances * test(enable-banking): cover nil provider balance * fix(enable-banking): avoid stale and unsafe balance logs * fix(enable-banking): guard unavailable balance writes * test(enable-banking): stub unsaved account api id * style(enable-banking): remove extra test blank line --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>	2026-07-17 21:33:32 +02:00
Jestin J Palamuttam	e324170160	fix(provider): resolve Tiingo security currency from countryCode (#2692) ... * fix(provider): resolve Tiingo security currency from countryCode Tiingo's search API never returns the priceCurrency field the code was reading, so currency detection silently failed. Currency is now derived from countryCode via the countries gem's ISO 4217 data, with a best-match tie-break for tickers that collide across countries so the currency shown in search results always matches what fetch_security_prices later returns. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(provider): guard tiingo currency cache against non-US downgrade Addresses PR review feedback (jjmata): the per-ticker currency cache written in search_securities was unconditionally overwritten on every search, keyed only by ticker. A later search whose result set doesn't happen to include a ticker's US cross-listing could silently downgrade a previously-cached USD (the currency actually backing daily price data) to a foreign currency. Only overwrite when the current search's match is US, or nothing is cached yet. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-17 06:37:44 +02:00
Jestin J Palamuttam	e6a0ca597b	feat(provider): add Frankfurter as an exchange-rate provider (#2640) ... * feat(exchange-rates): add Frankfurter as an exchange-rate provider Frankfurter (frankfurter.dev) is a free, keyless FX rates API backed by ECB daily reference rates, with no published rate limit and no auth flow to maintain (unlike Yahoo Finance's reverse-engineered cookie/ crumb auth or TwelveData's fast-exhausting free tier). Follows the Provider::MoexPublic template: Faraday client with retry middleware, SslConfigurable for self-hosted CA support, a light RateLimitable throttle, and a FRANKFURTER_URL env escape hatch for self-hosters. Registered as exchange-rates-only (no security/stock data) and added to the hosting settings dropdown. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * refactor(provider): switch Frankfurter to the v2 API v1 is explicitly marked "frozen" on Frankfurter's own root endpoint; v2 is "current" and covers 201 currencies across 84 central banks vs v1's ~30 ECB-only. Confirmed via the v2 OpenAPI spec and live requests: - Single-date lookups now use GET /rate/{base}/{quote}?date=..., which carries weekends/holidays forward server-side (a Saturday returns a real rate directly), so the provider no longer needs its own lookback-window logic. - Range lookups now use GET /rates?base=...&quotes=...&from=...&to=..., a flat array of { date, base, quote, rate } records (v2's shape) instead of v1's { "rates": { date => currencies } } hash. - Every calendar day in a range is present (v2 gapfills itself), rather than v1's omit-non-trading-days behavior. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(provider): sanitize currency codes before URL path interpolation from/to were only upcased before being interpolated directly into the URL path in fetch_exchange_rate (GET /rate/{from}/{to}). Low risk since currency codes come from validated internal sources, but adds cheap defense-in-depth: strip anything that isn't A-Z, matching the ISO 4217 format real currency codes always take. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-14 02:59:53 +02:00
Jestin Palamuttam	5803023fa7	feat(provider): add native Questrade brokerage provider integration (#2534) ... * Add native Questrade brokerage provider integration Adds a per-family Questrade provider so users can sync their Questrade investment accounts (TFSA, FHSA, RRSP, margin, etc.) directly via Questrade's free personal API, with no paid aggregator. - OAuth2 refresh-token flow with single-use token rotation, persisted under a row lock. Tokens self-renew on each sync; the connected panel lets users paste a fresh token if a connection goes stale (no need to disconnect and re-link). - Imports accounts, balances, positions and activities; multi-currency holdings with per-currency cash holdings; Norbert's Gambit journals. - New-account and link-existing-account flows, settings card with desktop-only setup steps, and connect/update/disconnect. - Restricted to Investment account types. Registered in the provider connection-status registry with a syncable scope so it participates in nightly family sync. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: linting error * fix: refresh token encrypted The OAuth token exchange passed the single-use refresh token as a GET query parameter, so it could leak into URL-based logs (Sentry breadcrumbs, APM spans, debug output). Switch to POST with a form-encoded body (RFC 6749 3.2) so the credential stays out of URLs. Verified Questrade's token endpoint accepts POST (returns 400 for a bad token, not 405). Adds a test asserting the token travels in the body. * Address PR review: authz, data integrity, retries, logging Batch of fixes from the automated PR review: - Require admin for all mutating/linking Questrade actions, and gate existing-account linking through accessible_accounts + write permission (was only Current.family scoped). - Clear requires_update when a fresh token is accepted; use a real 302 redirect (not 422) on full-page failures. - Require refresh_token on all saves (not just create) unless the item is scheduled for deletion. - Migrations target Rails 7.2; questrade_items state columns are NOT NULL. - Background activity dedup keys on Questrade fields (matches the importer) so multiple activities no longer collapse to one. - Persist the normalized account payload; date-scope synthetic cash holdings so daily history is not overwritten. - Retry 429/5xx via a RetryableResponseError instead of hard-failing. - Route provider error bodies to DebugLogEntry instead of Rails.logger / exception messages, so payloads do not leak into application logs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address PR review: atomic linking, sync health, retry loop, USD cash - Wrap account creation + provider linking (+ sync_start_date) in a transaction in both link paths so a link failure rolls back the orphan account. - Surface per-account process/schedule failures in the item sync health instead of always reporting healthy. - Always stamp last_activities_sync once the background fetch completes, so legitimately empty accounts stop being re-queued every sync. - Treat only the account-currency (CAD) balance as primary cash; other currencies (e.g. USD) now surface as separate cash holdings instead of being hidden as primary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address PR review: serialize token exchange, real processor tests - Single-use token race: the SDK now wraps every token exchange (initial and 401 re-auth) in a model-supplied lock that reloads and spends the freshest persisted token (provided.rb#synchronize_exchange). Two concurrent syncs/jobs can no longer double-spend the same refresh token. Adds a test asserting the exchange runs inside the lock with the fresh token. - Replace the all-skipped QuestradeAccount processor test stubs with real fixture-backed tests covering balance anchoring, holdings import, and Buy-trade import (plus blank-symbol / blank-type guards). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Fix indentation of spliced Questrade schema blocks The manually added questrade_accounts/questrade_items create_table blocks sat at column 0 instead of the file 2-space indent, so rubocop flagged them as inconsistent. Re-indent to match the rest of the schema. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Include currency and type in the Questrade activity merge key Two activities that differ only by currency or type could collapse to a single row in merge_activities. Add both fields to activity_key in the importer and the background fetch job so multi-currency imports dedup correctly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address review: infer account currency, Encryptable, safer flag clear From @jjmata's review: - Currency: QuestradeAccount#upsert_from_questrade! no longer hardcodes CAD for every account. upsert_balances! now infers the home currency from the per-currency balances (the currency holding the cash wins, ties broken by total equity, default CAD) so USD-denominated accounts are labelled USD and match the right combinedBalances anchor. Adds tests for USD and CAD cases. - QuestradeItem now includes the shared Encryptable concern instead of reimplementing encryption_ready? inline. - QuestradeActivitiesFetchJob#clear_pending_flag is now best-effort so it can never mask (and swallow) the original error in perform's rescue. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Fix review issues: safe_return_to_path, DebugLogEntry, financial reset, turbo_prefetch, N+1 counts - Add safe_return_to_path to QuestradeItemsController (blocks //evil.com protocol-relative open redirect; same 3-check guard as PR #2591 Wise provider) - Pass return_to through select_accounts and complete_account_setup so users land back on the account they were linking from - Replace Rails.logger.error/warn with DebugLogEntry.capture in controller and unlinking concern (surface errors in the app debug log UI) - Add questrade_items to Family::FinancialDataReset::PROVIDER_ITEM_ASSOCIATIONS so Reset Financial Data actually removes Questrade data - Add when "questrade" case to load_provider_items in providers_controller so the settings panel lazy-load refresh works - Fix turbo_prefetch: false on non-lunchflow provider links in _method_selector.html.erb and select_provider.html.erb (prevents prefetch-cache blank-modal bug for all generic sync providers) - Preload questrade_accounts: :account_provider and build @questrade_account_counts_map in AccountsController; read from map in partial instead of calling .count on associations (eliminates N+1) - Localize default connection name via I18n.t(questrade_items.default_name) - Add default_name key to questrade_items locale Patterns and bugs surfaced during review of PR #2591 (Wise provider). * Cross-apply Wise learnings to Questrade provider Encryption (matched convention from Wise/jjmata review): - Add deterministic: true to QuestradeItem#refresh_token - Add encrypts :raw_payload + :raw_institution_payload to QuestradeItem - Add Encryptable + encrypts :raw_payload, :raw_holdings_payload, :raw_activities_payload, :raw_balances_payload to QuestradeAccount (brokerage-specific columns; matches MercuryAccount/UpAccount pattern) Bug fix: - Add missing RetryableResponseError class to Provider::Questrade (used in with_retries rescue clause but never defined — would cause NameError on any rate-limited or 5xx response) Logging: - Replace Rails.logger.error with DebugLogEntry.capture in QuestradeItem#import_latest_questrade_data, #process_accounts, and #schedule_account_syncs to surface errors in the support UI Consistency: - Extract update_sync_status(sync, key, **i18n_options) helper in QuestradeItem::Syncer, replacing 5 inline sync.update! guard calls - Use blank? instead of ||= for default name fallback in create action Tests: - Add QuestradeItemsControllerTest (18 tests: CRUD, sync, account linking/setup flows, admin guard enforcement) - Add questrade fixtures: questrade_items.yml, questrade_accounts.yml - Add retry/backoff tests to Provider::QuestradeTest (network error, 429, 5xx — all verify MAX_RETRIES exhaustion raises Error) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Signed-off-by: Jestin Palamuttam <34907800+jestinjoshi@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-08 09:00:49 +02:00
threatsurfer	ab32388326	feat(up): flag internal transfers and round-ups as funds_movement (#2460) ... * feat(up): flag internal transfers and round-ups as funds_movement Up populates relationships.transferAccount on transactions that move money between the user's own accounts (including round-ups swept into a Saver), but flatten_transaction dropped it, so these imported as ordinary income/expense and distorted budgets and cashflow. - Provider::Up#flatten_transaction: lift transfer_account_id from relationships.transferAccount.data.id. - UpEntry::Processor: import transfers as funds_movement and persist transfer_account_id in extra["up"]. - Account::ProviderImportAdapter#import_transaction: optional kind: param; an explicit provider kind takes precedence over account-type auto-detection and is applied after the sync-protection check, so user re-categorisations survive re-sync. Complementary to Family#auto_match_transfers!: two-sided transfers between linked accounts are still paired into a Transfer (the matcher does not filter on kind); one-sided movements and round-ups, which the matcher cannot pair, are the cases this fixes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(up): account-type kind wins over provider transfer hint Codex review caught that Up HOME_LOAN accounts map to a Loan account, so a repayment carrying transferAccount would be reclassified from loan_payment to funds_movement (budget-excluded). Make the provider kind: a fallback: activity-label and account-type classification now take precedence, so loan_payment and cc_payment survive. Adds a regression test (loan repayment stays loan_payment), a depository-applies test, and a note that the up_test stub ignores query: intentionally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Gavin Matthews <matthews.gav@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-30 08:19:25 +02:00
ghost	d329a4f69d	feat(kraken): import deposits, withdrawals, staking & fees via Ledgers API (#2451) ... * feat(kraken): fetch and import Ledgers API for deposits, withdrawals, staking, fees Closes #2450 Kraken TradesHistory only returns spot buy/sell trades. The Ledgers API (/0/private/Ledgers) covers deposits, withdrawals, staking rewards, Earn income, and standalone fees — everything that was missing from syncs. Changes: - Provider::Kraken#get_ledgers — new method forwarding start/type/offset params - KrakenItem::Importer#fetch_ledgers — paginated fetch (up to 200 pages) with graceful fallback if the API key lacks Query Ledger Entries permission - Importer#upsert_kraken_account — stores "ledgers" alongside "trades" in raw_transactions_payload - KrakenAccount::LedgerProcessor — new class; maps each supported ledger type (deposit, withdrawal, staking, earn, fee) to a Transaction entry with the correct investment_activity_label, kind, and sign convention; skips trade/ transfer/margin types to avoid double-counting with TradesHistory - KrakenAccount::Processor#process — calls LedgerProcessor after process_trades - Multi-currency: fiat amounts converted via ExchangeRate (non-USD fiat bridged through USD); crypto amounts use the spot price cached in raw_payload["assets"] with a price_missing flag when no price is available - Dedup guard: external_id "kraken_ledger_<id>" + source "kraken" prevents re-importing on repeated syncs * fix(kraken): correct sign convention, fee inclusion, and earn subtype filtering - Sign convention: deposits/staking/earn → negative (inflow), withdrawals/fees → positive (outflow), matching Sure's global convention (inflow is negative) - Fee inclusion: use (amount - fee).abs as abs_impact so withdrawal fees are counted in the total outflow rather than discarded - Earn subtypes: skip allocation/deallocation ledger entries (internal fund movements); only import rewardallocation/bonusallocation as Interest income - DebugLogEntry: replace Rails.logger.warn/error with DebugLogEntry.capture throughout LedgerProcessor and the Ledgers permission fallback in Importer, so support-relevant incidents surface in /settings/debug - Importer test: stub get_ledgers in setup so existing tests do not error on the new fetch_ledgers call * fix(kraken): route duplicate ledger-id warning through DebugLogEntry * perf(kraken): batch ledger idempotency check; strengthen tests Address review feedback (jjmata): - N+1: LedgerProcessor#process_ledger_entry ran `account.entries.exists?(...)` per ledger entry (up to ~10k per sync). Load the existing Kraken external IDs once into a Set and test membership in memory (newly created IDs are added so the same run stays idempotent) — same pattern as #2452. - Tests: the idempotency test now asserts the first pass actually creates the entry (assert_difference) before asserting the second is a no-op; add a guard asserting the second (all-skipped) pass issues a single bulk external_id pluck, not one query per entry. No behavior change to imported entries. * perf(kraken): scope ledger idempotency pluck to kraken_ledger_ prefix Only load existing ledger external IDs (not trade entries) into the idempotency Set, matching the reviewed approach. No behavior change.	2026-06-30 07:35:14 +02:00
Juan José Mata	b416618558	Add SnapTrade OAuth device flow (#2494) ... * Add SnapTrade OAuth device flow * Add SnapTrade OAuth device flow ### Motivation - Integrate SnapTrade device-code OAuth so administrators can start a device authorization flow and poll for tokens using SnapTrade's well-known OAuth metadata endpoint. - Persist and encrypt device-flow token material on `SnaptradeItem` to support long-lived API calls and future refresh handling. ### Description - Add OAuth discovery and device-code support to the SnapTrade provider with `oauth_authorization_server_metadata`, `start_device_authorization`, and `poll_device_token` helper methods and shared `oauth_connection` / `parse_oauth_response` helpers. - Persist OAuth device-flow tokens on `snaptrade_items` via a migration that adds `oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, and `oauth_token_expires_at`, and update `db/schema.rb` accordingly. - Extend `SnaptradeItem` to encrypt stored `oauth_access_token` and `oauth_refresh_token` when ActiveRecord encryption is configured and add `oauth_token_active?` to check token validity. - Add high-level `start_oauth_device_flow` and `complete_oauth_device_flow!` helpers to the `SnaptradeItem::Provided` concern to start authorization and persist token responses. - Expose admin-only member routes and JSON controller actions `start_oauth_device_flow` and `complete_oauth_device_flow` on `SnaptradeItemsController` to drive the device flow from the UI. - Add focused Minitest coverage: `test/models/provider/snaptrade_oauth_test.rb` for provider requests and error handling and `test/models/snaptrade_item_oauth_test.rb` for token persistence. ### Testing - Ran the targeted test suite: `bin/rails test test/models/provider/snaptrade_oauth_test.rb test/models/snaptrade_item_oauth_test.rb test/controllers/snaptrade_items_controller_test.rb`, and all tests passed. - Ran `bin/rubocop` against modified files and no offenses were reported. * Add SnapTrade OAuth device flow ### Motivation - Integrate SnapTrade device-code OAuth so administrators can start a device authorization flow and poll for tokens using SnapTrade's well-known OAuth metadata endpoint. - Persist and encrypt device-flow token material on `SnaptradeItem` to support long-lived API calls and future refresh handling. ### Description - Add OAuth discovery and device-code support to the SnapTrade provider with `oauth_authorization_server_metadata`, `start_device_authorization`, and `poll_device_token` helper methods and shared `oauth_connection` / `parse_oauth_response` helpers. - Persist OAuth device-flow tokens on `snaptrade_items` via a migration that adds `oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, and `oauth_token_expires_at`, and update `db/schema.rb` accordingly. - Extend `SnaptradeItem` to encrypt stored `oauth_access_token` and `oauth_refresh_token` when ActiveRecord encryption is configured and add `oauth_token_active?` to check token validity. - Add high-level `start_oauth_device_flow` and `complete_oauth_device_flow!` helpers to the `SnaptradeItem::Provided` concern to start authorization and persist token responses. - Expose admin-only member routes and JSON controller actions `start_oauth_device_flow` and `complete_oauth_device_flow` on `SnaptradeItemsController` to drive the device flow from the UI. - Add focused Minitest coverage: `test/models/provider/snaptrade_oauth_test.rb` for provider requests and error handling and `test/models/snaptrade_item_oauth_test.rb` for token persistence. ### Testing - Ran the targeted test suite: `bin/rails test test/models/provider/snaptrade_oauth_test.rb test/models/snaptrade_item_oauth_test.rb test/controllers/snaptrade_items_controller_test.rb`, and all tests passed. - Ran `bin/rubocop` against modified files and no offenses were reported. * Add SnapTrade OAuth device flow ### Motivation - Integrate SnapTrade device-code OAuth so administrators can start a device authorization flow and poll for tokens using SnapTrade's well-known OAuth metadata endpoint. - Persist and encrypt device-flow token material on `SnaptradeItem` to support long-lived API calls and future refresh handling. ### Description - Add OAuth discovery and device-code support to the SnapTrade provider with `oauth_authorization_server_metadata`, `start_device_authorization`, and `poll_device_token` helper methods and shared `oauth_connection` / `parse_oauth_response` helpers. - Persist OAuth device-flow tokens on `snaptrade_items` via a migration that adds `oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, and `oauth_token_expires_at`, and update `db/schema.rb` accordingly. - Extend `SnaptradeItem` to encrypt stored `oauth_access_token` and `oauth_refresh_token` when ActiveRecord encryption is configured and add `oauth_token_active?` to check token validity. - Add high-level `start_oauth_device_flow` and `complete_oauth_device_flow!` helpers to the `SnaptradeItem::Provided` concern to start authorization and persist token responses. - Expose admin-only member routes and JSON controller actions `start_oauth_device_flow` and `complete_oauth_device_flow` on `SnaptradeItemsController` to drive the device flow from the UI. - Add focused Minitest coverage: `test/models/provider/snaptrade_oauth_test.rb` for provider requests and error handling and `test/models/snaptrade_item_oauth_test.rb` for token persistence. ### Testing - Ran the targeted test suite: `bin/rails test test/models/provider/snaptrade_oauth_test.rb test/models/snaptrade_item_oauth_test.rb test/controllers/snaptrade_items_controller_test.rb`, and all tests passed. - Ran `bin/rubocop` against modified files and no offenses were reported. * Add SnapTrade OAuth device flow ### Motivation - Integrate SnapTrade device-code OAuth so administrators can start a device authorization flow and poll for tokens using SnapTrade's well-known OAuth metadata endpoint. - Persist and encrypt device-flow token material on `SnaptradeItem` to support long-lived API calls and future refresh handling. ### Description - Add OAuth discovery and device-code support to the SnapTrade provider with `oauth_authorization_server_metadata`, `start_device_authorization`, and `poll_device_token` helper methods and shared `oauth_connection` / `parse_oauth_response` helpers. - Persist OAuth device-flow tokens on `snaptrade_items` via a migration that adds `oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, and `oauth_token_expires_at`, and update `db/schema.rb` accordingly. - Extend `SnaptradeItem` to encrypt stored `oauth_access_token` and `oauth_refresh_token` when ActiveRecord encryption is configured and add `oauth_token_active?` to check token validity. - Add high-level `start_oauth_device_flow` and `complete_oauth_device_flow!` helpers to the `SnaptradeItem::Provided` concern to start authorization and persist token responses. - Expose admin-only member routes and JSON controller actions `start_oauth_device_flow` and `complete_oauth_device_flow` on `SnaptradeItemsController` to drive the device flow from the UI. - Add focused Minitest coverage: `test/models/provider/snaptrade_oauth_test.rb` for provider requests and error handling and `test/models/snaptrade_item_oauth_test.rb` for token persistence. ### Testing - Ran the targeted test suite: `bin/rails test test/models/provider/snaptrade_oauth_test.rb test/models/snaptrade_item_oauth_test.rb test/controllers/snaptrade_items_controller_test.rb`, and all tests passed. - Ran `bin/rubocop` against modified files and no offenses were reported. * Add SnapTrade OAuth device flow (provider, model, controller, routes, migration, tests) ### Motivation - Add support for SnapTrade OAuth 2.0 device authorization flow so users can authorize SnapTrade via device codes in addition to the existing portal flow. - Persist OAuth token metadata on `SnaptradeItem` for subsequent polling and usage by the provider and UI. ### Description - Implemented OAuth device flow in the provider with `oauth_authorization_server_metadata`, `start_device_authorization`, and `poll_device_token`, plus HTTP helper methods `oauth_connection`, `oauth_client_id`, and `parse_oauth_response` in `Provider::Snaptrade`. - Added controller endpoints `start_oauth_device_flow` and `complete_oauth_device_flow` in `SnaptradeItemsController` with robust error handling and helpers `oauth_error_payload` and `parse_oauth_error_body` to surface OAuth error fields. - Extended `SnaptradeItem` with encrypted columns for `oauth_access_token`, `oauth_refresh_token`, token metadata, `oauth_token_active?`, and model methods `start_oauth_device_flow` and `complete_oauth_device_flow!` to store token metadata. - Added migration `AddOauthDeviceFlowToSnaptradeItems` and updated `db/schema.rb` to include the new columns, registered a new initializer `config/initializers/snaptrade.rb` to read `SNAPTRADE_OAUTH_CLIENT_ID`, and updated `.env*.example` files to document `SNAPTRADE_OAUTH_CLIENT_ID`. - Exposed new routes `start_oauth_device_flow` and `complete_oauth_device_flow` for `snaptrade_items`. ### Testing - Added unit tests for provider OAuth behavior in `test/models/provider/snaptrade_oauth_test.rb`, model token persistence in `test/models/snaptrade_item_oauth_test.rb`, and controller error propagation in `test/controllers/snaptrade_items_controller_test.rb`. - Ran the test suite with `bin/rails test` and the full test run (including the new SnapTrade OAuth tests) passed. * Add SnapTrade OAuth device flow (start/poll), store tokens, and tests ### Motivation - Add support for SnapTrade OAuth Device Authorization flow so administrators can start device authorization and poll for tokens without exposing provider internals. - Persist OAuth token metadata on `SnaptradeItem` so the app can reuse and surface token state for SnapTrade integrations. - Improve error handling and sanitization for OAuth API errors returned by SnapTrade to avoid leaking upstream internals. ### Description - Introduces new environment examples and initializer: adds `SNAPTRADE_OAUTH_CLIENT_ID` to `.env.local.example`/`.env.test.example` and configures `Rails.configuration.x.snaptrade.oauth_client_id` in `config/initializers/snaptrade.rb`. - Extends `Provider::Snaptrade` with OAuth device flow support, adding discovery URL, device grant constant, Faraday `oauth_connection`, and methods `oauth_authorization_server_metadata`, `start_device_authorization`, `poll_device_token`, `oauth_client_id`, and `parse_oauth_response`, plus improved retry and API error wrapping. - Adds DB migration and schema changes to store OAuth fields on `snaptrade_items` (`oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, `oauth_token_expires_at`) and marks those attributes encrypted when ActiveRecord encryption is enabled. - Adds `SnaptradeItem` helpers `start_oauth_device_flow`, `complete_oauth_device_flow!`, and `oauth_token_active?` to start/poll and persist token metadata. - Adds controller actions `start_oauth_device_flow` and `complete_oauth_device_flow` with sanitized error responses and helper methods `oauth_error_payload` and `parse_oauth_error_body`, and wires new member routes in `config/routes.rb`. - Updates `SnaptradeItemsController` before_action lists to include the new actions and adds user-facing error message helper `start_oauth_device_flow_error_message`. ### Testing - Added unit tests `test/models/provider/snaptrade_oauth_test.rb` to validate discovery, device authorization request, token polling, missing client ID handling, and OAuth error propagation, and all assertions passed. - Added `test/models/snaptrade_item_oauth_test.rb` to assert `complete_oauth_device_flow!` stores tokens and expiry and that `oauth_token_active?` reports correctly, and the test passed. - Extended `test/controllers/snaptrade_items_controller_test.rb` with controller-level tests confirming sanitized OAuth error payloads and behavior for start/complete endpoints, and these controller tests passed. * Add SnapTrade OAuth device flow support and token storage ### Motivation - Add support for the OAuth 2.0 device authorization flow for SnapTrade so users can link brokerages via a device-code flow without traditional browser-based client redirects. - Persist OAuth token metadata on the SnaptradeItem so tokens can be reused and expiration tracked. - Surface and sanitize provider error payloads to callers while avoiding leakage of internal configuration details. ### Description - Added new DB columns and a migration (`oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, `oauth_token_expires_at`) and updated `db/schema.rb` to reflect the change. - Encrypted the new token fields on `SnaptradeItem` and added `oauth_token_active?`, `start_oauth_device_flow`, and `complete_oauth_device_flow!` helpers to the model. - Implemented OAuth logic in `Provider::Snaptrade` including discovery (`OAUTH_DISCOVERY_URL`), `start_device_authorization`, `poll_device_token`, request parsing, retry handling, and a small Faraday connection wrapper; added configuration accessors for `oauth_client_id` via `config.x.snaptrade`. - Added controller endpoints `start_oauth_device_flow` and `complete_oauth_device_flow` with safe error handling and helpers to format OAuth error payloads, and registered new member routes for `snaptrade_items`. - Added an initializer to load `SNAPTRADE_OAUTH_CLIENT_ID` into `Rails.configuration.x.snaptrade`, and updated `.env.local.example` and `.env.test.example` to include the new env var. - Added unit tests for provider OAuth behavior (`test/models/provider/snaptrade_oauth_test.rb`), SnaptradeItem token persistence (`test/models/snaptrade_item_oauth_test.rb`), and controller error handling (`test/controllers/snaptrade_items_controller_test.rb`), and updated controller tests accordingly. ### Testing - Ran provider-level tests in `Provider::SnaptradeOauthTest` to validate discovery, device authorization, token polling, and error handling, which passed. - Ran model tests in `SnaptradeItemOauthTest` to verify token metadata is stored and `oauth_token_active?` works, which passed. - Ran controller tests in `SnaptradeItemsControllerTest` that exercise the start/complete endpoints and error sanitization, which passed. * Add SnapTrade OAuth device-flow support and token storage ### Motivation - Add support for SnapTrade OAuth device authorization so users can perform device-code based OAuth without embedding secrets in the browser. - Persist OAuth token metadata on `SnaptradeItem` and expose programmatic start/complete endpoints while avoiding leaking provider internals on errors. - Make OAuth client ID configurable via environment or credentials for Sure deployments. ### Description - Introduce `SNAPTRADE_OAUTH_CLIENT_ID` to `.env` examples and add `config.x.snaptrade.oauth_client_id` initializer for configuration. - Add migration `AddOauthDeviceFlowToSnaptradeItems` and update `schema.rb` to add `oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, and `oauth_token_expires_at` to `snaptrade_items`. - Persist and encrypt new token fields in `SnaptradeItem` and add helper methods `oauth_token_active?`, `start_oauth_device_flow`, and `complete_oauth_device_flow!`. - Extend `Provider::Snaptrade` with OAuth discovery, device authorization (`start_device_authorization`), token polling (`poll_device_token`), Faraday-based `oauth_connection`, parsing and error-wrapping logic, and retry handling. - Add controller actions `start_oauth_device_flow` and `complete_oauth_device_flow` to `SnaptradeItemsController`, plus helper methods to format OAuth error payloads and user-facing error messages. - Wire up new routes (`post :start_oauth_device_flow`, `post :complete_oauth_device_flow`) and add `config/initializers/snaptrade.rb`. - Add comprehensive tests for the OAuth flow and controller error handling and update a system test stub to avoid provider leakage during UI tests. ### Testing - Added `Provider::SnaptradeOauthTest` which stubs discovery, device authorization and token endpoints and asserts request payloads and responses; tests passed. - Added `SnaptradeItemOauthTest` which verifies token metadata persistence and `oauth_token_active?`; test passed. - Extended `SnaptradeItemsControllerTest` with scenarios for successful and failing device-flow completion and start flow error handling; tests passed. - Ran the affected system test changes in `TradesTest` (provider stubbing and modal behavior); the updated tests passed.	2026-06-27 06:43:12 +02:00
Frank Li	b88deb3545	fix: use ES256 instead of EdDSA for Coinbase CDP JWT signing (#1888) ... * fix: use ES256 instead of EdDSA for Coinbase CDP JWT signing Coinbase Developer Platform issues EC P-256 keys, not Ed25519. The previous EdDSA implementation would always return Unauthorized for any key generated via portal.cdp.coinbase.com. Switch to ES256 (ECDSA SHA-256) using OpenSSL, and convert the DER-encoded signature to the raw r||s format required by the JWT spec. API Secret field now accepts the full PEM private key directly (including BEGIN/END headers) as provided by Coinbase. * fix(coinbase): address PR review comments - Normalize escaped \n in PEM key before parsing (fixes pasting directly from the Coinbase CDP JSON download file where newlines are \n literals) - Extract parse_ec_private_key helper with docstring explaining both accepted PEM formats - Fix double-space style nit on encoded_header assignment - Remove ed25519 gem from Gemfile (no longer used after ES256 migration) - Add Provider::CoinbaseTest covering: JWT 3-part structure, ES256 alg header, kid claim, CDP payload claims, 64-byte raw r||s signature, escaped-newline key acceptance, and cryptographic signature verification * test(coinbase): fix base64url padding in JWT test assertions * chore: update Gemfile.lock to remove ed25519 gem * Sample credential in test * Comments field, not real key use * test(coinbase): generate EC key dynamically; drop scanner exclusions Generate a fresh P-256 key per test run with OpenSSL::PKey::EC.generate instead of hardcoding a PEM private key. This removes committed key material and the need to exclude the provider and test files from the pipelock secret scan, so those exclusions are removed too. Addresses review findings on hardcoded test key + scan exclusions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Juan José Mata <jjmata@jjmata.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-26 06:37:33 +02:00
Artem Danilov	c65f4828dd	fix(tinkoff): resolve the tradeable listing and price bonds via BondBy (#2413) ... * fix(tinkoff): resolve the tradeable listing and price bonds via BondBy Validated against the live T-Invest API and fixed three correctness issues in Provider::TinkoffInvest: - FindInstrument returns several listings per ticker (e.g. SBER on TQBR plus dark/non-API boards 37M/SPEQ); only the apiTradeAvailableFlag listing has live prices. resolve_short now ranks tradeable-first (then requested-MIC, then exact ticker/ISIN) instead of taking the first match, so GetLastPrices/GetCandles return data. Search now surfaces only tradeable instruments. - find_instruments no longer hard-filters on apiTradeAvailableFlag, so a qualified-investor instrument Tinkoff lists but can't API-trade still resolves (for logos) and otherwise falls back to another price provider. - Bond nominal comes from BondBy (the generic GetInstrumentBy omits it), returning the current amortized nominal so percent-of-par converts correctly. Verified end-to-end: SBER 313.65, T 282.76, LQDT 2.019, and the SFO Split bond (amortized nominal 417.71) all price; logos resolve to real CDN PNGs. * fix(tinkoff): honor requested MIC first, guard amortizing bonds, strip suffix Address review feedback on the resolution/pricing path: - resolve_short now ranks a requested-MIC match BEFORE tradeability, so a security is never priced off another exchange's listing (e.g. a MISX security no longer picks up a tradeable XSPX board's price); tradeability and exact ticker/ISIN remain secondary tie-breaks. - Amortizing bonds: BondBy returns only the current nominal, so applying it to historical percent-of-par closes would underprice them. For amortizing bonds we now return only the live price and skip the candle history; fixed-par bonds and equities keep full history. bond_info exposes nominal + amortizationFlag. - Strip exchange suffixes (.ME/.MOEX/.MISX/.MCX) before querying T-Invest, which only knows the bare SECID — so a stored "T.MOEX" ticker resolves to "T". * fix(tinkoff): don't cache nil resolution results (skip_nil) so a transient empty response isn't locked in for the 24h TTL	2026-06-20 21:46:01 +02:00
Jake	dc2a565b6a	feat(up): add Up Bank (AU) provider integration (#2391) ... * feat(up): add Up Bank (AU) provider integration Adds Up Bank as a per-family, token-based bank sync provider, modelled on the existing Akahu integration. Up uses a JSON:API REST API with a personal access token (Bearer), cursor pagination via links.next, and returns both HELD (pending) and SETTLED transactions from one endpoint. New: - Provider::Up client (JSON:API unwrap, links.next pagination, retries, typed errors, /util/ping) + Provider::UpAdapter (Factory-registered, Depository + Loan). - UpItem / UpAccount models with Provided, Unlinking, Syncer, SyncCompleteEvent, Importer, Processor, Transactions::Processor, and UpEntry::Processor (amount sign flip, HELD->pending, foreignAmount FX, merchant from description, stale-pending pruning). - Family::UpConnectable, UpItemsController, routes, settings panel + connect flow views, accounts index wiring, initializer, en locale, and model tests. Core wiring: - "up" added to Transaction::PENDING_PROVIDERS, the three pending-match SQL blocks in Account::ProviderImportAdapter, Provider::Metadata::REGISTRY, ProviderMerchant/DataEnrichment source enums, ProviderConnectionStatus, settings provider panels, and financial data reset. Migration create_up_items_and_accounts must be run before use. No external API endpoints added (no OpenAPI changes). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(up): dump up tables to schema and make since filter TZ-safe The feature commit added the up_items/up_accounts migration but never re-dumped db/schema.rb, leaving the schema version and tables stale. Add the two table definitions and foreign keys and bump the schema version so a fresh DB load matches the migration. Also format a bare Date `since` as UTC midnight instead of the server's local zone, so `filter[since]` is deterministic regardless of where the app runs (previously shifted by the local UTC offset). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(up): address code review feedback Behavior/correctness: - Persist skipped accounts via a new up_accounts.ignored flag and a needs_setup scope, so skipped accounts stop resurfacing as "needs setup" on every sync. Linking clears the flag. - destroy now checks unlink_all! per-account results and aborts deletion (alert) if any unlink failed, instead of swallowing failures. - render_provider_panel_error redirect uses :see_other (was an invalid 4xx redirect status). - Up provider adapter falls back to item institution name/url when institution_metadata is absent (early return previously blocked it). Resilience/security: - fetch_all_resources guards against an API repeating the same links.next cursor (Set#add?), preventing infinite pagination. - HTTP client validates absolute URLs (from links.next) against Up's HTTPS host before sending the bearer token, preventing credential leakage to untrusted hosts. Diagnostics: - Route provider sync/import failures through DebugLogEntry.capture (controller, UpItem, syncer, unlinking) with family/account context. Low-level HTTP client and currency-normalization warnings keep Rails.logger to match existing provider conventions. Data integrity: - up_accounts.name and currency are NOT NULL (align with model presence validations); account_id stays nullable (allow_nil uniqueness). Forms: - select_existing_account radio is required; controller guards a blank/ unknown up_account_id with a friendly alert instead of RecordNotFound. Tests: - Add UpAccount needs_setup scope test, pagination loop guard test, untrusted-host rejection test; tighten filter[since] assertion to the exact UTC timestamp. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(up): address second-round review feedback - Capture sync/import failures via DebugLogEntry so swallowed errors in account/transaction fetching and transaction processing surface in /settings/debug instead of only Rails.logger. - Gate UP_DEBUG_RAW raw payload dump to local envs to avoid leaking PII (merchant names, amounts, account IDs) in managed/production logs. - Collapse linked/unlinked/total account counts into one memoized query instead of 3 separate COUNTs per rendered item. - Rename "Set Up Up Accounts" locale title to "Link Up Accounts". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(up): add method docstrings and align failed_result keys Add docstrings to all Up provider source files (controller, models, providers, concerns) to satisfy the 80% docstring coverage threshold. Third-round review: failed_result now mirrors import's result shape (accounts_updated/created/failed, transactions_imported/failed) instead of the stale accounts_imported key, so failure results stay consistent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>	2026-06-20 20:33:09 +02:00
Artem Danilov	9de9a23ce2	feat(prices): add T-Invest (T-Bank) securities + brand-logo provider (#2408) ... Adds Provider::TinkoffInvest, a token-based securities provider built on the public T-Invest REST gateway (invest-public-api.tinkoff.ru/rest). It serves prices for Russian instruments (shares, ETF/БПИФ, bonds) and, crucially, brand logos via the T-Invest CDN — the authoritative logo source for MOEX instruments, which ISS (MoexPublic) does not provide. - Registry: register `tinkoff_invest` under the :securities concept; token via ENV TINKOFF_INVEST_API_KEY or encrypted Setting.tinkoff_invest_api_key. - Logos independent of the price provider: Security#import_brand_logo consults T-Invest for a logo whenever a token is configured (after the price-provider metadata fetch, so it never short-circuits website_url backfill). Gated on token presence, not the securities checklist. - display_logo_url: with no website domain, a stored provider logo (T-Invest) now beats the ticker-only Brandfetch lettermark; when a domain exists, Brandfetch still wins (unchanged). - MoexPublic no longer reports moex.com as the issuer website — it's the exchange, not the issuer, and would make Brandfetch render the exchange logo for every instrument and shadow the real brand logo. - Prices: GetCandles (daily, paged) + GetLastPrices; Quotation units+nano/1e9; bonds priced as percent-of-par x nominal (missing nominal raises, not 0). - Settings: encrypted token field (always shown) + provider checkbox + en locale. - Tests for search/info/logo-url/prices/bond/incomplete-candle and display logic. Co-authored-by: Claude <noreply@anthropic.com>	2026-06-19 17:07:09 +02:00
Artem Danilov	b57bb938da	feat(prices): add Moscow Exchange (MOEX ISS) securities + FX provider (#2394) ... * feat(prices): add Moscow Exchange (MOEX ISS) securities + FX provider Add Provider::MoexPublic, a keyless provider built on the free MOEX ISS API (https://iss.moex.com/iss), modeled on Provider::BinancePublic. Securities: shares, funds/ETF/БПИФ (e.g. LQDT), and bonds (OFZ + corporate). Bonds are priced clean — LAST% × FACEVALUE / 100 in the instrument currency, with per-row FACEVALUE for amortizing issues; NKD/accrued coupon excluded. Exchange rates: also implements ExchangeRateConcept for RUB↔{USD,EUR,CNY} via selt TOM instruments (USD000UTSTOM/EUR_RUB__TOM/CNYRUB_TOM); the selt quote is X/RUB, inverted for RUB→X, nil for non-RUB-crossed pairs. Details: - Board/engine resolution via the ISS primary-board flag with a hardcoded priority fallback (TQBR, TQTF, TQOB, TQCB, …). - Instrument currency from CURRENCYID/FACEUNIT (handles USD/CNY eurobonds & FX funds), normalizing legacy SUR/RUR → RUB; default RUB. - Full history via from/till + start= pagination; current price fallback chain LAST → MARKETPRICE → LCURRENTPRICE → LCLOSEPRICE → PREVPRICE → latest history close. - Bare SECID identity, exchange_operating_mic=MISX, country_code=nil (wildcard like Binance); search accepts .ME/.MOEX/.MISX/.MCX aliases and ISIN. - RateLimitable throttling, SslConfigurable, Faraday retry/timeouts; all public methods wrapped in with_provider_response. Wired into Provider::Registry for both :securities and :exchange_rates, the hosting provider-selection UI, locales, and config/exchanges.yml (MISX). Docker-tested (devcontainer, Ruby 3.4.9): 29 new tests green, full provider suite + i18n green, rubocop clean; smoke-tested against live ISS (SBER price, OFZ clean price, USD/RUB FX). * fix(moex): address review — FX weekend lookback, dead branch, translated hints - fetch_exchange_rate now fetches a 10-day lookback window (not just the exact day) so a weekend/holiday request resolves to the prior trading day's close, matching Yahoo's behavior (Codex P2). - Remove dead identical if/else branches in history_row_price (CodeRabbit). - Translate moex_public_hint into ca/fr/hu/vi/zh-CN instead of English copy (CodeRabbit). - Add a test covering the FX prior-trading-day lookback. * fix(moex): guard ISS date parsing; doc TQTE in board priority Address maintainer review (jjmata): - parse_iss_date wraps Date.parse so a malformed ISS TRADEDATE skips just that row (with a contextual log warning) instead of failing the whole history/FX fetch. Used in history_row_price and fx_history. - Add TQTE to the BOARD_PRIORITY doc comment (it was in the constant but missing from the comment). - Add a test covering the unparseable-date skip.	2026-06-18 10:36:20 +02:00
Artem Danilov	fffcaf2345	fix(prices): resolve dashed crypto tickers (BTC-USD, TRX-USD) via BinancePublic (#2372) ... * fix(prices): resolve dashed crypto tickers (BTC-USD, TRX-USD) via BinancePublic Provider::BinancePublic is the dedicated crypto price provider, but its search_securities only matched bare base assets ("BTC") or unseparated Binance pairs ("BTCUSDT"/"BTCUSD"). The canonical "<BASE>-USD" form that Yahoo emits and users paste (e.g. "TRX-USD", "USDT-USD") never matched, so those holdings fell through to an unpriced offline security whenever the stock provider didn't also return the coin. Collapse the base/quote separator in the search query (and strip it in parse_ticker) so "BTC-USD" / "TRX/USDT" are treated like "BTCUSD" and resolve to live Binance pricing. Stablecoins pasted as "USDT-USD" resolve to the synthetic USD price via their stripped base. * fix(prices): only synthesize stablecoin search for USD quote Address review: gate the synthetic stablecoin result on a USD quote (bare coin or "<coin>USD" form). A non-USD quote like "USDTEUR" / "USDT-EUR" has a real Binance pair and now falls through to normal matching instead of being replaced by the USD synthetic. Drops the now-unused base_asset_query helper. Adds a USDT-EUR regression test.	2026-06-17 11:38:34 +02:00
Kelvinchen03	59edcaa986	fix: EODHD lookup for EU mutual funds with EUFUND exchange code (#2212)	2026-06-11 16:17:17 +02:00
Guillem Arias Fauste	f25fe30a41	feat(ai): honor Setting.llm_provider for batch and PDF flows (#2265) ... Auto-categorization, merchant detection/enhancement, and PDF/bank-statement extraction hard-coded Provider::Registry.get_provider(:openai), so selecting Anthropic (or running an Anthropic-only self-hosted install) left those operations using/missing OpenAI rather than the chosen provider. Add Provider::Registry.preferred_llm_provider, which resolves the LLM provider honoring Setting.llm_provider with a configured-provider fallback (mirroring how chat picks its provider), and route all six TODO(#2113) call sites through it: - Family::AutoCategorizer#llm_provider - Family::AutoMerchantDetector#llm_provider - ProviderMerchant::Enhancer#llm_provider - PdfImport (process_pdf + extract_bank_statement) - Assistant::Function::ImportBankStatement Provider::Anthropic already implements auto_categorize / auto_detect_merchants / enhance_provider_merchants (#1984) and process_pdf / extract_bank_statement (#1985), so no provider changes are needed — only the wiring. Closes #2113.	2026-06-09 23:02:26 +02:00
Dan	3627c4d2d1	Respect CoinStats wallet rate limits (#2251)	2026-06-08 22:40:34 +02:00
Tobias Rahloff	fe47c918bb	feat(enable_banking): support MFA/decoupled banks and harden session handling (#2174) ... Decoupled/MFA banks (e.g. VR Bank in Holstein) were hard-blocked because the authorize flow aborted whenever auth_methods[0] was DECOUPLED. Enable Banking's hosted /auth page actually coordinates decoupled SCA and redirects back with a code, so route these banks through it instead: - Provider#start_authorization accepts and forwards an auth_method param - EnableBankingItem#select_auth_method picks the best method (REDIRECT > DECOUPLED > EMBEDDED), filtering by psu_type and skipping hidden methods - Shared begin_authorization! re-fetches ASPSP metadata on each authorize and reauthorize, so the method is always re-derived (no persistence required) - Remove the DECOUPLED block in the controller Also stop the integration from constantly reporting "session expired": - Only a session-level GET /sessions 401/404 flips the connection to requires_update; per-account 401/404 are retried and no longer kill the whole connection - Reconcile session_expires_at from the API's access.valid_until on every sync - Treat an expired session as a graceful requires_update state instead of raising a bare error No schema changes. Adds covering tests.	2026-06-04 19:53:52 +02:00
Guillem Arias Fauste	8ccc434b3d	feat(ai): Anthropic native PDF processing (3/5) (#1985) ... * feat(ai): add Anthropic provider with chat parity (1/5) Introduces Provider::Anthropic alongside Provider::Openai, implementing the LlmConcept chat_response contract over the official anthropic Ruby SDK. Batch ops, PDF, and RAG land in follow-up PRs. - Provider::Anthropic uses Messages API for sync and streaming responses - ChatConfig builds requests with ephemeral prompt-cache markers on the system prompt and the last tool definition - MessageFormatter reconstructs multi-turn history (text + tool_use + tool_result blocks) from raw Message records, including the paired user-role tool_result turn Anthropic requires after every tool_use - ChatParser maps Anthropic Message into the shared ChatResponse Data - Registry, Setting, User, Chat default model wired for ANTHROPIC_* envs and Setting.anthropic_*; LLM_PROVIDER selects between providers - Responder forwards raw conversation_history (Array<Message>) so providers without hosted conversation state can rebuild context - OpenAI provider accepts and ignores the new kwarg (no behavior change) Tests cover provider init, model gating, MessageFormatter for all turn shapes, ChatConfig request building (max_tokens, system cache, tool conversion), ChatParser for text / tool_use / mixed blocks, Registry discovery, and mocked chat_response success / error / function_request paths. Live VCR cassettes recorded in a follow-up with a real key. Stacked PRs: 2/5 batch ops + cost ledger, 3/5 PDF, 4/5 pgvector RAG, 5/5 settings UI + disclosure. * fix(ai): address PR review on Anthropic provider foundation Surface fixes raised by Codex + CodeRabbit on PR 1/5: - Provider::Anthropic#chat_response now accepts (and ignores) a `messages:` kwarg. Assistant::Responder passes both `messages:` (OpenAI-shape) and `conversation_history:` (raw Message records) for cross-provider parity, so the previous signature raised ArgumentError on the first chat turn through the Anthropic provider. - Provider::Anthropic#supports_model? bypasses the `claude` prefix gate when a custom base_url is configured, mirroring the OpenAI provider. Bedrock-shaped IDs like `anthropic.claude-sonnet-4-5-20250929-v1:0` and `claude-opus-4@20250514` are otherwise rejected by Assistant::Provided#get_model_provider and the chat dies. - Setting.anthropic_access_token is now in EncryptedSettingFields::ENCRYPTED_FIELDS so the Anthropic API key is encrypted at rest like every other provider secret. Previously plaintext while siblings (openai_access_token, twelve_data_api_key, external_assistant_token) were ciphertext. - Chat.default_model falls back to whichever provider is actually configured. Previously, with LLM_PROVIDER=anthropic but no Anthropic credentials, the default model resolved to a Claude ID that no registered provider supported, so chats failed even when OpenAI was fully configured. Adds Provider::{Anthropic,Openai}#configured? class methods for the readable callsite. - Provider::Anthropic.effective_model uses `ENV["ANTHROPIC_MODEL"].presence || Setting.anthropic_model` so the Setting lookup is only performed when the env var is absent — the previous `ENV.fetch(KEY, default)` evaluated the default arg eagerly on every call. - Provider::Anthropic::ChatConfig#anthropic_input_schema strips both `:strict` and `"strict"` keys so JSON-decoded schemas with string keys cannot leak the OpenAI-only flag through to Anthropic. Test coverage added: supports_model? bypass on custom endpoints, chat_response messages: kwarg compatibility, default_model fallback in the three credential combinations, configured? against ENV + Setting, strict-flag stripping for both key types, and a `Setting.expects(:anthropic_model).never` assertion proving the ENV-precedence test now exercises the lazy path. All 4365 tests pass (1 pre-existing libvips env error unrelated). * test(chat): make default_model tests resilient to ENV model overrides CodeRabbit flagged on PR review: the new default_model tests asserted against Provider::*::DEFAULT_MODEL, but Chat.default_model actually returns Provider::*.effective_model.presence (which reads OPENAI_MODEL / ANTHROPIC_MODEL from the environment). With either env var set, the tests would fail intermittently even though routing was correct. - New default_model tests now assert against the provider's effective_model directly, so they verify the routing decision (which provider's value wins) without coupling to the constant. - Pre-existing "creates with default model" assertions had the same brittleness; switch them to compare against Chat.default_model so the chosen model is whatever the env / Setting cascade resolves to. Verified by running `ANTHROPIC_MODEL=claude-haiku-4-5 OPENAI_MODEL=gpt-4o bin/rails test test/models/chat_test.rb` — 16 runs, 0 failures (previously 2 pre-existing failures + 0 from the new tests). * fix(ai): address local review on Anthropic foundation - Provider::Anthropic#supports_pdf_processing? bypasses prefix gate for custom endpoints, mirroring supports_model? - Provider::Anthropic#initialize raises Error when custom_endpoint? AND model.blank?, parity with Provider::Openai - stream_chat_response captures partial usage on mid-stream errors and records it via the new on_partial callback so chat_response can skip the duplicate error row in the outer rescue - safe_accumulated_message swallows the secondary failure when the SDK cannot reconstruct a snapshot - langfuse_client memoizes properly (||= instead of =) so repeated calls don't churn Langfuse instances - MessageFormatter sorts tool_calls by created_at then id so the message array is deterministic across replays; skips tool_calls missing both provider_call_id and provider_id rather than sending `id: nil` and getting rejected by Anthropic - Setting.anthropic_access_token default falls back through ENV["ANTHROPIC_API_KEY"].presence (was missing .presence, so an empty-string env value bled through) - User#openai_configured? / #anthropic_configured? delegate to the Provider::* class methods — single source of truth - Assistant::Responder renames the OpenAI-shape history builder conversation_history → openai_messages_payload so the kwarg name matches the local method name (messages: openai_messages_payload, conversation_history: chat_message_records) - Assistant::Builtin stale-history comment updated to reference both builders Adds a streaming chat_response test using ad-hoc subclasses of the SDK event types so the case/when dispatch matches via is_a? without stubbing class-level === behavior. * test(ai): add Anthropic tool_use round-trip + multi-tool turn coverage Addresses @jjmata's "worth confirming" note on PR #1983: tool-use turns from prior assistant messages must round-trip correctly when retrieved from the database. - New `ChatParser → ToolCall::Function → MessageFormatter` test walks the full path: Anthropic response with a tool_use block → ChatFunctionRequest → ToolCall::Function.from_function_request → persisted on the AssistantMessage → MessageFormatter rebuild on the next turn. Asserts the original `tool_use.id` is preserved end-to-end as both `tool_use.id` and the paired `tool_result.tool_use_id`, and that the original `input` hash and serialized result content survive. - New multi-tool assistant turn test confirms two tool_use blocks on a single assistant message render as two tool_use blocks followed by two paired tool_result blocks in a single user-role follow-up, matching Anthropic's required alternation. Both tests exercise the existing PR1 code without behavior changes. * test(ai): require "ostruct" explicitly in Anthropic provider tests OpenStruct is moving out of Ruby's default load path (warning in 3.4+, removed in 3.5+). Tests work today because ActiveSupport transitively loads it, but that's incidental. Match the existing convention in test/controllers/settings/hostings_controller_test.rb which explicitly requires ostruct for the same reason. * fix(ai): sanitize Langfuse warn logs, normalize tool_use.input, dedup history fetch Addresses three open CodeRabbit findings on PR #1983. - Provider::Anthropic Langfuse rescue branches no longer include `e.full_message` in `Rails.logger.warn`. `full_message` bundles the backtrace + cause chain and on some SDK error types includes the serialized request/response payload (prompt, model output). Logs now report `#{e.class}: #{e.message}` only. Three sites: create_langfuse_trace, log_langfuse_generation, upsert_langfuse_trace. Note: Provider::Openai has the same pattern (copy-pasted source) — harmonization deferred to a follow-up cleanup PR; this commit fixes only the Anthropic provider to keep PR scope tight. - MessageFormatter#parse_arguments now coerces any non-Hash parsed result to `{}`. Anthropic's Messages API requires `tool_use.input` to be a JSON object (map); a stored ToolCall::Function record whose arguments parse to a scalar, bool, or array (corrupt row, legacy data, cross-provider bleed) would otherwise produce a payload the API rejects. Normal flow stores Hash arguments end-to-end so the fix is defensive — adds 2 tests covering scalar/array JSON strings and non-String non-Hash inputs. - Assistant::Responder dedups the chat-history fetch. The previous layout fired two near-identical `chat.messages.where(...).includes( :tool_calls).ordered` queries per LLM turn (one for the OpenAI-shape payload, one for the raw-records kwarg). A new memoized `complete_chat_messages` fetches once; `chat_message_records` filters out the current message via `Array#reject`, `openai_messages_payload` iterates the cached array unchanged. One SQL query per turn instead of two. Memoization scope = single Responder instance (per LLM call), so cache invalidation is not a concern. All 4370 tests pass (1 pre-existing libvips env error unrelated). Rubocop + brakeman clean. * fix(ci): replace sk-ant- prefixed test placeholders Pipelock secret scanner pattern-matches `sk-ant-*` as a real Anthropic API key and fails the PR security-scan check. Test stubs and ClimateControl env values used `sk-ant-test`, `sk-ant-from-setting`, `sk-ant-x`, `sk-ant-y` as obvious placeholders, but the scanner does not care about value entropy. Switched to `fake-anthropic-key-*` / `fake-token-*` strings so the scanner stops flagging them. No production code touched, no behavior change — Provider::Anthropic still accepts any non-blank token. * feat(ai): add Anthropic batch ops + LLM cost ledger (2/5) Implements auto_categorize, auto_detect_merchants, and enhance_provider_merchants on Provider::Anthropic via forced tool calls, plus the cost-ledger plumbing they need. - Provider::Anthropic::AutoCategorizer, AutoMerchantDetector, ProviderMerchantEnhancer each define a single output tool whose input_schema mirrors the desired output, then force the model to call it via tool_choice: { type: "tool", name: ..., disable_parallel_tool_use: true }. Anthropic guarantees the tool_use.input matches the schema, so there is no JSON parsing fragility, no <think> tag stripping, and no json_object/json_schema fallback ladders. - Concerns::UsageRecorder mirrors the OpenAI sibling but persists cache_creation_input_tokens / cache_read_input_tokens to dedicated columns instead of metadata. - Migration adds cache_creation_tokens, cache_read_tokens (nullable integers) to llm_usages. OpenAI rows leave them null. - LlmUsage::PRICING gains Claude 4.x rows (opus-4-7 $15/$75, sonnet-4-6 $3/$15, haiku-4-5 $1/$5 per MTok). infer_provider returns "anthropic" for claude-* via the existing exact/prefix lookup. - Provider::Anthropic#chat_response now persists cache columns directly rather than stashing them in metadata. - 25-transaction batch cap mirrors the OpenAI provider so the cost ledger sees the same shape regardless of which provider ran a batch. Tests cover the forced-tool-call path, null/None normalization, case-insensitive merchant matching, the missing-tool_use error path, and Anthropic-specific pricing + provider inference on LlmUsage. Stacked on #1983 (PR 1/5). 3/5 PDF + vision next. * fix(ai): attribute Bedrock model IDs to anthropic + clean nil enum - LlmUsage.infer_provider now returns "anthropic" for Bedrock / Vertex shaped IDs (anthropic.* and anthropic/*), so cost-ledger filtering by provider stays correct even when no per-MTok rate is stored. Previously these IDs fell through to the "openai" default. - AutoCategorizer drops the redundant nil sentinel from the category_name enum — the union type [string, null] already permits null, and some JSON Schema validators reject nil literals inside enum arrays. * test(ai): require "ostruct" in Anthropic batch op tests Same rationale as the PR1 ostruct fix — explicit require so the tests don't depend on ActiveSupport's transitive load when Ruby 3.5+ removes OpenStruct from the default load path. * feat(ai): Anthropic native PDF processing (3/5) Implements process_pdf and extract_bank_statement on Provider::Anthropic using the native `document` content block — no rasterization, no text pre-extraction. - Provider::Anthropic::PdfProcessor classifies the document, summarizes it, and extracts statement metadata via a forced report_document_analysis tool whose input_schema mirrors the existing Provider::Openai output (document_type from Import::DOCUMENT_TYPES, summary, extracted_data). - Provider::Anthropic::BankStatementExtractor returns the same { transactions, period, account_holder, account_number, bank_name, opening_balance, closing_balance } shape via report_bank_statement so downstream pdf_import code is provider-agnostic. - Both attach the PDF as { type: "document", source: { type: "base64", media_type: "application/pdf", data: <b64> } } — Claude 3.5+ / 4.x accept this natively (up to 32MB / 100 pages). No pdf-reader, no pdftoppm, no chunking for typical statements. - supports_pdf_processing? (introduced in PR 1) already returns true for claude-* models, gating process_pdf with a clear error otherwise. - Cost ledger rows are persisted via the shared UsageRecorder concern, including cache_creation/cache_read tokens. Tests verify the document block shape, tool_choice forcing, normalized document_type for unknown classifications, transaction normalization (date / amount / reference → notes), and the missing-tool_use error path. Blank pdf_content raises before any client call. Stacked on #1984 (PR 2/5). 4/5 pgvector RAG next. * fix(ai): guard PDF size + surface bank-statement truncation - PdfProcessor and BankStatementExtractor raise upfront when pdf_content.bytesize exceeds MAX_PDF_BYTES (32 MB, matching Anthropic's hard limit). Previously a 100 MB PDF would be base64-encoded (~133 MB) and packed into the JSON body before the API rejected it — peak heap ~270 MB per Sidekiq worker. - BankStatementExtractor inspects response.stop_reason; when the model hit max_tokens it logs a warning and flags result[:truncated] so downstream callers know the transaction list may be incomplete. - ISO date pattern added to statement_period_start/end schema in PdfProcessor so the model can't return "March 2026" — Anthropic enforces the regex via the tool's input_schema. Tests cover the size guard (raises before any client.messages call), truncated-result flagging, and the warning log path. * test(ai): require "ostruct" in Anthropic PDF tests Match the explicit ostruct require added in PR1/PR2 — same Ruby 3.5+ load-path reason. * fix(llm-usage): include Anthropic cache tokens in estimated_cost calculate_cost only priced prompt + completion tokens, so estimated_cost under-reported every cached call — the cache_creation/cache_read columns this PR added were tracked but never billed. Verified against the Anthropic dashboard: a cached chat turn billed $0.05 but the ledger recorded $0.038; the gap was exactly the unpriced cache tokens. Price them relative to the input rate (Anthropic: cache write 1.25x, read 0.1x) and thread the cache counts from both recorders (chat + batch). OpenAI rows leave the columns null (treated as 0), so they're unaffected. Ledger now reproduces the dashboard ($0.054 for the test turn). * chore(ai): guard chat usage double-record; flag deferred Anthropic batch wiring - Hardening: guard the success-path record_llm_usage with `unless partial_usage_recorded` so a future change that emits partial usage on a normal stream can't silently double-bill (the symptom investigated in the #1984 review). No behavior change today — on_partial only fires from the mid-stream-error rescue, which re-raises past this line. - Notice: the family auto-categorize / merchant-detect / merchant-enhance flows still hardcode get_provider(:openai). Provider::Anthropic now implements those batch ops but they aren't wired into the family flows yet — documented with TODOs at each site for the follow-up. * chore(ai): point family-flow TODOs at tracking issue #2113 * chore(ai): flag deferred Anthropic PDF wiring (TODO #2113) The PDF import + bank-statement-extract flows hardcode get_provider(:openai). Provider::Anthropic implements process_pdf / extract_bank_statement (this PR) but they aren't wired into these paths yet — documented with TODOs at each site. Tracked in #2113 (broadened to cover batch ops + PDF). * chore(anthropic-pdf): drop redundant strip_heredoc; document no-dedup - The squiggly heredoc (<<~) already strips indentation, so the trailing .strip_heredoc was a no-op in both PDF extractors. - Document why BankStatementExtractor intentionally does NOT deduplicate (unlike the OpenAI extractor): we send the whole PDF as one native document block, so there are no overlapping-chunk artifacts to dedupe, and deduping would wrongly merge legitimate same-day, same-amount transactions. * fix(anthropic): cap PDF bytes below the base64-encoded request limit Anthropic's 32 MB limit is on the Messages request body, and the PDF is sent base64-encoded (~4/3 larger) alongside the JSON envelope, so a 32 MB raw PDF encodes to ~42 MB and is rejected. Cap the raw bytes at 3/4 of the request budget minus a 1 MB envelope reserve (~23 MiB). Addresses Codex review on #1985.	2026-06-03 11:30:51 +02:00
Brad	1b8b21760b	feat(provider): Akahu integration (#1921) ... * First pass of Akahu * fix up sync all * conflicts * fix db migration issue? - fix auto selection of akahu account type * Address Akahu PR feedback * Complete provider metadata * Fix PR 1921 CI tests * PR feedback * PR feedback * post merge --------- Co-authored-by: failing <failing@users.noreply.github.com> Co-authored-by: Juan José Mata <jjmata@jjmata.com> Co-authored-by: sure-admin <sure-admin@splashblot.com>	2026-06-02 21:44:57 +02:00
Guillem Arias Fauste	991cb959c1	feat(ai): Anthropic batch ops + LLM cost ledger (2/5) (#1984) ... * feat(ai): add Anthropic provider with chat parity (1/5) Introduces Provider::Anthropic alongside Provider::Openai, implementing the LlmConcept chat_response contract over the official anthropic Ruby SDK. Batch ops, PDF, and RAG land in follow-up PRs. - Provider::Anthropic uses Messages API for sync and streaming responses - ChatConfig builds requests with ephemeral prompt-cache markers on the system prompt and the last tool definition - MessageFormatter reconstructs multi-turn history (text + tool_use + tool_result blocks) from raw Message records, including the paired user-role tool_result turn Anthropic requires after every tool_use - ChatParser maps Anthropic Message into the shared ChatResponse Data - Registry, Setting, User, Chat default model wired for ANTHROPIC_* envs and Setting.anthropic_*; LLM_PROVIDER selects between providers - Responder forwards raw conversation_history (Array<Message>) so providers without hosted conversation state can rebuild context - OpenAI provider accepts and ignores the new kwarg (no behavior change) Tests cover provider init, model gating, MessageFormatter for all turn shapes, ChatConfig request building (max_tokens, system cache, tool conversion), ChatParser for text / tool_use / mixed blocks, Registry discovery, and mocked chat_response success / error / function_request paths. Live VCR cassettes recorded in a follow-up with a real key. Stacked PRs: 2/5 batch ops + cost ledger, 3/5 PDF, 4/5 pgvector RAG, 5/5 settings UI + disclosure. * fix(ai): address PR review on Anthropic provider foundation Surface fixes raised by Codex + CodeRabbit on PR 1/5: - Provider::Anthropic#chat_response now accepts (and ignores) a `messages:` kwarg. Assistant::Responder passes both `messages:` (OpenAI-shape) and `conversation_history:` (raw Message records) for cross-provider parity, so the previous signature raised ArgumentError on the first chat turn through the Anthropic provider. - Provider::Anthropic#supports_model? bypasses the `claude` prefix gate when a custom base_url is configured, mirroring the OpenAI provider. Bedrock-shaped IDs like `anthropic.claude-sonnet-4-5-20250929-v1:0` and `claude-opus-4@20250514` are otherwise rejected by Assistant::Provided#get_model_provider and the chat dies. - Setting.anthropic_access_token is now in EncryptedSettingFields::ENCRYPTED_FIELDS so the Anthropic API key is encrypted at rest like every other provider secret. Previously plaintext while siblings (openai_access_token, twelve_data_api_key, external_assistant_token) were ciphertext. - Chat.default_model falls back to whichever provider is actually configured. Previously, with LLM_PROVIDER=anthropic but no Anthropic credentials, the default model resolved to a Claude ID that no registered provider supported, so chats failed even when OpenAI was fully configured. Adds Provider::{Anthropic,Openai}#configured? class methods for the readable callsite. - Provider::Anthropic.effective_model uses `ENV["ANTHROPIC_MODEL"].presence || Setting.anthropic_model` so the Setting lookup is only performed when the env var is absent — the previous `ENV.fetch(KEY, default)` evaluated the default arg eagerly on every call. - Provider::Anthropic::ChatConfig#anthropic_input_schema strips both `:strict` and `"strict"` keys so JSON-decoded schemas with string keys cannot leak the OpenAI-only flag through to Anthropic. Test coverage added: supports_model? bypass on custom endpoints, chat_response messages: kwarg compatibility, default_model fallback in the three credential combinations, configured? against ENV + Setting, strict-flag stripping for both key types, and a `Setting.expects(:anthropic_model).never` assertion proving the ENV-precedence test now exercises the lazy path. All 4365 tests pass (1 pre-existing libvips env error unrelated). * test(chat): make default_model tests resilient to ENV model overrides CodeRabbit flagged on PR review: the new default_model tests asserted against Provider::*::DEFAULT_MODEL, but Chat.default_model actually returns Provider::*.effective_model.presence (which reads OPENAI_MODEL / ANTHROPIC_MODEL from the environment). With either env var set, the tests would fail intermittently even though routing was correct. - New default_model tests now assert against the provider's effective_model directly, so they verify the routing decision (which provider's value wins) without coupling to the constant. - Pre-existing "creates with default model" assertions had the same brittleness; switch them to compare against Chat.default_model so the chosen model is whatever the env / Setting cascade resolves to. Verified by running `ANTHROPIC_MODEL=claude-haiku-4-5 OPENAI_MODEL=gpt-4o bin/rails test test/models/chat_test.rb` — 16 runs, 0 failures (previously 2 pre-existing failures + 0 from the new tests). * fix(ai): address local review on Anthropic foundation - Provider::Anthropic#supports_pdf_processing? bypasses prefix gate for custom endpoints, mirroring supports_model? - Provider::Anthropic#initialize raises Error when custom_endpoint? AND model.blank?, parity with Provider::Openai - stream_chat_response captures partial usage on mid-stream errors and records it via the new on_partial callback so chat_response can skip the duplicate error row in the outer rescue - safe_accumulated_message swallows the secondary failure when the SDK cannot reconstruct a snapshot - langfuse_client memoizes properly (||= instead of =) so repeated calls don't churn Langfuse instances - MessageFormatter sorts tool_calls by created_at then id so the message array is deterministic across replays; skips tool_calls missing both provider_call_id and provider_id rather than sending `id: nil` and getting rejected by Anthropic - Setting.anthropic_access_token default falls back through ENV["ANTHROPIC_API_KEY"].presence (was missing .presence, so an empty-string env value bled through) - User#openai_configured? / #anthropic_configured? delegate to the Provider::* class methods — single source of truth - Assistant::Responder renames the OpenAI-shape history builder conversation_history → openai_messages_payload so the kwarg name matches the local method name (messages: openai_messages_payload, conversation_history: chat_message_records) - Assistant::Builtin stale-history comment updated to reference both builders Adds a streaming chat_response test using ad-hoc subclasses of the SDK event types so the case/when dispatch matches via is_a? without stubbing class-level === behavior. * test(ai): add Anthropic tool_use round-trip + multi-tool turn coverage Addresses @jjmata's "worth confirming" note on PR #1983: tool-use turns from prior assistant messages must round-trip correctly when retrieved from the database. - New `ChatParser → ToolCall::Function → MessageFormatter` test walks the full path: Anthropic response with a tool_use block → ChatFunctionRequest → ToolCall::Function.from_function_request → persisted on the AssistantMessage → MessageFormatter rebuild on the next turn. Asserts the original `tool_use.id` is preserved end-to-end as both `tool_use.id` and the paired `tool_result.tool_use_id`, and that the original `input` hash and serialized result content survive. - New multi-tool assistant turn test confirms two tool_use blocks on a single assistant message render as two tool_use blocks followed by two paired tool_result blocks in a single user-role follow-up, matching Anthropic's required alternation. Both tests exercise the existing PR1 code without behavior changes. * test(ai): require "ostruct" explicitly in Anthropic provider tests OpenStruct is moving out of Ruby's default load path (warning in 3.4+, removed in 3.5+). Tests work today because ActiveSupport transitively loads it, but that's incidental. Match the existing convention in test/controllers/settings/hostings_controller_test.rb which explicitly requires ostruct for the same reason. * fix(ai): sanitize Langfuse warn logs, normalize tool_use.input, dedup history fetch Addresses three open CodeRabbit findings on PR #1983. - Provider::Anthropic Langfuse rescue branches no longer include `e.full_message` in `Rails.logger.warn`. `full_message` bundles the backtrace + cause chain and on some SDK error types includes the serialized request/response payload (prompt, model output). Logs now report `#{e.class}: #{e.message}` only. Three sites: create_langfuse_trace, log_langfuse_generation, upsert_langfuse_trace. Note: Provider::Openai has the same pattern (copy-pasted source) — harmonization deferred to a follow-up cleanup PR; this commit fixes only the Anthropic provider to keep PR scope tight. - MessageFormatter#parse_arguments now coerces any non-Hash parsed result to `{}`. Anthropic's Messages API requires `tool_use.input` to be a JSON object (map); a stored ToolCall::Function record whose arguments parse to a scalar, bool, or array (corrupt row, legacy data, cross-provider bleed) would otherwise produce a payload the API rejects. Normal flow stores Hash arguments end-to-end so the fix is defensive — adds 2 tests covering scalar/array JSON strings and non-String non-Hash inputs. - Assistant::Responder dedups the chat-history fetch. The previous layout fired two near-identical `chat.messages.where(...).includes( :tool_calls).ordered` queries per LLM turn (one for the OpenAI-shape payload, one for the raw-records kwarg). A new memoized `complete_chat_messages` fetches once; `chat_message_records` filters out the current message via `Array#reject`, `openai_messages_payload` iterates the cached array unchanged. One SQL query per turn instead of two. Memoization scope = single Responder instance (per LLM call), so cache invalidation is not a concern. All 4370 tests pass (1 pre-existing libvips env error unrelated). Rubocop + brakeman clean. * fix(ci): replace sk-ant- prefixed test placeholders Pipelock secret scanner pattern-matches `sk-ant-*` as a real Anthropic API key and fails the PR security-scan check. Test stubs and ClimateControl env values used `sk-ant-test`, `sk-ant-from-setting`, `sk-ant-x`, `sk-ant-y` as obvious placeholders, but the scanner does not care about value entropy. Switched to `fake-anthropic-key-*` / `fake-token-*` strings so the scanner stops flagging them. No production code touched, no behavior change — Provider::Anthropic still accepts any non-blank token. * feat(ai): add Anthropic batch ops + LLM cost ledger (2/5) Implements auto_categorize, auto_detect_merchants, and enhance_provider_merchants on Provider::Anthropic via forced tool calls, plus the cost-ledger plumbing they need. - Provider::Anthropic::AutoCategorizer, AutoMerchantDetector, ProviderMerchantEnhancer each define a single output tool whose input_schema mirrors the desired output, then force the model to call it via tool_choice: { type: "tool", name: ..., disable_parallel_tool_use: true }. Anthropic guarantees the tool_use.input matches the schema, so there is no JSON parsing fragility, no <think> tag stripping, and no json_object/json_schema fallback ladders. - Concerns::UsageRecorder mirrors the OpenAI sibling but persists cache_creation_input_tokens / cache_read_input_tokens to dedicated columns instead of metadata. - Migration adds cache_creation_tokens, cache_read_tokens (nullable integers) to llm_usages. OpenAI rows leave them null. - LlmUsage::PRICING gains Claude 4.x rows (opus-4-7 $15/$75, sonnet-4-6 $3/$15, haiku-4-5 $1/$5 per MTok). infer_provider returns "anthropic" for claude-* via the existing exact/prefix lookup. - Provider::Anthropic#chat_response now persists cache columns directly rather than stashing them in metadata. - 25-transaction batch cap mirrors the OpenAI provider so the cost ledger sees the same shape regardless of which provider ran a batch. Tests cover the forced-tool-call path, null/None normalization, case-insensitive merchant matching, the missing-tool_use error path, and Anthropic-specific pricing + provider inference on LlmUsage. Stacked on #1983 (PR 1/5). 3/5 PDF + vision next. * fix(ai): attribute Bedrock model IDs to anthropic + clean nil enum - LlmUsage.infer_provider now returns "anthropic" for Bedrock / Vertex shaped IDs (anthropic.* and anthropic/*), so cost-ledger filtering by provider stays correct even when no per-MTok rate is stored. Previously these IDs fell through to the "openai" default. - AutoCategorizer drops the redundant nil sentinel from the category_name enum — the union type [string, null] already permits null, and some JSON Schema validators reject nil literals inside enum arrays. * test(ai): require "ostruct" in Anthropic batch op tests Same rationale as the PR1 ostruct fix — explicit require so the tests don't depend on ActiveSupport's transitive load when Ruby 3.5+ removes OpenStruct from the default load path. * fix(llm-usage): include Anthropic cache tokens in estimated_cost calculate_cost only priced prompt + completion tokens, so estimated_cost under-reported every cached call — the cache_creation/cache_read columns this PR added were tracked but never billed. Verified against the Anthropic dashboard: a cached chat turn billed $0.05 but the ledger recorded $0.038; the gap was exactly the unpriced cache tokens. Price them relative to the input rate (Anthropic: cache write 1.25x, read 0.1x) and thread the cache counts from both recorders (chat + batch). OpenAI rows leave the columns null (treated as 0), so they're unaffected. Ledger now reproduces the dashboard ($0.054 for the test turn). * chore(ai): guard chat usage double-record; flag deferred Anthropic batch wiring - Hardening: guard the success-path record_llm_usage with `unless partial_usage_recorded` so a future change that emits partial usage on a normal stream can't silently double-bill (the symptom investigated in the #1984 review). No behavior change today — on_partial only fires from the mid-stream-error rescue, which re-raises past this line. - Notice: the family auto-categorize / merchant-detect / merchant-enhance flows still hardcode get_provider(:openai). Provider::Anthropic now implements those batch ops but they aren't wired into the family flows yet — documented with TODOs at each site for the follow-up. * chore(ai): point family-flow TODOs at tracking issue #2113 * fix(ai): address review findings on cost ledger + categorizer schema Three AI-review findings on #1984: - category_name enum omitted null (codex + coderabbit): the prompt + type allow Claude to abstain on uncertain transactions, but JSON Schema `enum` restricted the value to category names, so null was invalid — forcing miscategorization. Append nil to the enum (the consumer already normalizes null -> uncategorized). - Cache pricing applied to all providers (coderabbit): the 1.25x/0.1x cache multipliers are Anthropic-specific. Gate them on provider == "anthropic" so a non-Anthropic caller passing cache counts isn't billed with the wrong rates. - Negative cache-token counts (coderabbit): add DB check constraints (cache_*_tokens IS NULL OR >= 0), per the repo's DB-level-validation convention. Tests: enum includes nil; non-Anthropic cache tokens aren't priced.	2026-06-01 22:00:48 +02:00
Guillem Arias Fauste	8251b7e4d6	feat(ai): add Anthropic provider with chat parity (1/5) (#1983) ... * feat(ai): add Anthropic provider with chat parity (1/5) Introduces Provider::Anthropic alongside Provider::Openai, implementing the LlmConcept chat_response contract over the official anthropic Ruby SDK. Batch ops, PDF, and RAG land in follow-up PRs. - Provider::Anthropic uses Messages API for sync and streaming responses - ChatConfig builds requests with ephemeral prompt-cache markers on the system prompt and the last tool definition - MessageFormatter reconstructs multi-turn history (text + tool_use + tool_result blocks) from raw Message records, including the paired user-role tool_result turn Anthropic requires after every tool_use - ChatParser maps Anthropic Message into the shared ChatResponse Data - Registry, Setting, User, Chat default model wired for ANTHROPIC_* envs and Setting.anthropic_*; LLM_PROVIDER selects between providers - Responder forwards raw conversation_history (Array<Message>) so providers without hosted conversation state can rebuild context - OpenAI provider accepts and ignores the new kwarg (no behavior change) Tests cover provider init, model gating, MessageFormatter for all turn shapes, ChatConfig request building (max_tokens, system cache, tool conversion), ChatParser for text / tool_use / mixed blocks, Registry discovery, and mocked chat_response success / error / function_request paths. Live VCR cassettes recorded in a follow-up with a real key. Stacked PRs: 2/5 batch ops + cost ledger, 3/5 PDF, 4/5 pgvector RAG, 5/5 settings UI + disclosure. * fix(ai): address PR review on Anthropic provider foundation Surface fixes raised by Codex + CodeRabbit on PR 1/5: - Provider::Anthropic#chat_response now accepts (and ignores) a `messages:` kwarg. Assistant::Responder passes both `messages:` (OpenAI-shape) and `conversation_history:` (raw Message records) for cross-provider parity, so the previous signature raised ArgumentError on the first chat turn through the Anthropic provider. - Provider::Anthropic#supports_model? bypasses the `claude` prefix gate when a custom base_url is configured, mirroring the OpenAI provider. Bedrock-shaped IDs like `anthropic.claude-sonnet-4-5-20250929-v1:0` and `claude-opus-4@20250514` are otherwise rejected by Assistant::Provided#get_model_provider and the chat dies. - Setting.anthropic_access_token is now in EncryptedSettingFields::ENCRYPTED_FIELDS so the Anthropic API key is encrypted at rest like every other provider secret. Previously plaintext while siblings (openai_access_token, twelve_data_api_key, external_assistant_token) were ciphertext. - Chat.default_model falls back to whichever provider is actually configured. Previously, with LLM_PROVIDER=anthropic but no Anthropic credentials, the default model resolved to a Claude ID that no registered provider supported, so chats failed even when OpenAI was fully configured. Adds Provider::{Anthropic,Openai}#configured? class methods for the readable callsite. - Provider::Anthropic.effective_model uses `ENV["ANTHROPIC_MODEL"].presence || Setting.anthropic_model` so the Setting lookup is only performed when the env var is absent — the previous `ENV.fetch(KEY, default)` evaluated the default arg eagerly on every call. - Provider::Anthropic::ChatConfig#anthropic_input_schema strips both `:strict` and `"strict"` keys so JSON-decoded schemas with string keys cannot leak the OpenAI-only flag through to Anthropic. Test coverage added: supports_model? bypass on custom endpoints, chat_response messages: kwarg compatibility, default_model fallback in the three credential combinations, configured? against ENV + Setting, strict-flag stripping for both key types, and a `Setting.expects(:anthropic_model).never` assertion proving the ENV-precedence test now exercises the lazy path. All 4365 tests pass (1 pre-existing libvips env error unrelated). * test(chat): make default_model tests resilient to ENV model overrides CodeRabbit flagged on PR review: the new default_model tests asserted against Provider::*::DEFAULT_MODEL, but Chat.default_model actually returns Provider::*.effective_model.presence (which reads OPENAI_MODEL / ANTHROPIC_MODEL from the environment). With either env var set, the tests would fail intermittently even though routing was correct. - New default_model tests now assert against the provider's effective_model directly, so they verify the routing decision (which provider's value wins) without coupling to the constant. - Pre-existing "creates with default model" assertions had the same brittleness; switch them to compare against Chat.default_model so the chosen model is whatever the env / Setting cascade resolves to. Verified by running `ANTHROPIC_MODEL=claude-haiku-4-5 OPENAI_MODEL=gpt-4o bin/rails test test/models/chat_test.rb` — 16 runs, 0 failures (previously 2 pre-existing failures + 0 from the new tests). * fix(ai): address local review on Anthropic foundation - Provider::Anthropic#supports_pdf_processing? bypasses prefix gate for custom endpoints, mirroring supports_model? - Provider::Anthropic#initialize raises Error when custom_endpoint? AND model.blank?, parity with Provider::Openai - stream_chat_response captures partial usage on mid-stream errors and records it via the new on_partial callback so chat_response can skip the duplicate error row in the outer rescue - safe_accumulated_message swallows the secondary failure when the SDK cannot reconstruct a snapshot - langfuse_client memoizes properly (||= instead of =) so repeated calls don't churn Langfuse instances - MessageFormatter sorts tool_calls by created_at then id so the message array is deterministic across replays; skips tool_calls missing both provider_call_id and provider_id rather than sending `id: nil` and getting rejected by Anthropic - Setting.anthropic_access_token default falls back through ENV["ANTHROPIC_API_KEY"].presence (was missing .presence, so an empty-string env value bled through) - User#openai_configured? / #anthropic_configured? delegate to the Provider::* class methods — single source of truth - Assistant::Responder renames the OpenAI-shape history builder conversation_history → openai_messages_payload so the kwarg name matches the local method name (messages: openai_messages_payload, conversation_history: chat_message_records) - Assistant::Builtin stale-history comment updated to reference both builders Adds a streaming chat_response test using ad-hoc subclasses of the SDK event types so the case/when dispatch matches via is_a? without stubbing class-level === behavior. * test(ai): add Anthropic tool_use round-trip + multi-tool turn coverage Addresses @jjmata's "worth confirming" note on PR #1983: tool-use turns from prior assistant messages must round-trip correctly when retrieved from the database. - New `ChatParser → ToolCall::Function → MessageFormatter` test walks the full path: Anthropic response with a tool_use block → ChatFunctionRequest → ToolCall::Function.from_function_request → persisted on the AssistantMessage → MessageFormatter rebuild on the next turn. Asserts the original `tool_use.id` is preserved end-to-end as both `tool_use.id` and the paired `tool_result.tool_use_id`, and that the original `input` hash and serialized result content survive. - New multi-tool assistant turn test confirms two tool_use blocks on a single assistant message render as two tool_use blocks followed by two paired tool_result blocks in a single user-role follow-up, matching Anthropic's required alternation. Both tests exercise the existing PR1 code without behavior changes. * test(ai): require "ostruct" explicitly in Anthropic provider tests OpenStruct is moving out of Ruby's default load path (warning in 3.4+, removed in 3.5+). Tests work today because ActiveSupport transitively loads it, but that's incidental. Match the existing convention in test/controllers/settings/hostings_controller_test.rb which explicitly requires ostruct for the same reason. * fix(ai): sanitize Langfuse warn logs, normalize tool_use.input, dedup history fetch Addresses three open CodeRabbit findings on PR #1983. - Provider::Anthropic Langfuse rescue branches no longer include `e.full_message` in `Rails.logger.warn`. `full_message` bundles the backtrace + cause chain and on some SDK error types includes the serialized request/response payload (prompt, model output). Logs now report `#{e.class}: #{e.message}` only. Three sites: create_langfuse_trace, log_langfuse_generation, upsert_langfuse_trace. Note: Provider::Openai has the same pattern (copy-pasted source) — harmonization deferred to a follow-up cleanup PR; this commit fixes only the Anthropic provider to keep PR scope tight. - MessageFormatter#parse_arguments now coerces any non-Hash parsed result to `{}`. Anthropic's Messages API requires `tool_use.input` to be a JSON object (map); a stored ToolCall::Function record whose arguments parse to a scalar, bool, or array (corrupt row, legacy data, cross-provider bleed) would otherwise produce a payload the API rejects. Normal flow stores Hash arguments end-to-end so the fix is defensive — adds 2 tests covering scalar/array JSON strings and non-String non-Hash inputs. - Assistant::Responder dedups the chat-history fetch. The previous layout fired two near-identical `chat.messages.where(...).includes( :tool_calls).ordered` queries per LLM turn (one for the OpenAI-shape payload, one for the raw-records kwarg). A new memoized `complete_chat_messages` fetches once; `chat_message_records` filters out the current message via `Array#reject`, `openai_messages_payload` iterates the cached array unchanged. One SQL query per turn instead of two. Memoization scope = single Responder instance (per LLM call), so cache invalidation is not a concern. All 4370 tests pass (1 pre-existing libvips env error unrelated). Rubocop + brakeman clean. * fix(ci): replace sk-ant- prefixed test placeholders Pipelock secret scanner pattern-matches `sk-ant-*` as a real Anthropic API key and fails the PR security-scan check. Test stubs and ClimateControl env values used `sk-ant-test`, `sk-ant-from-setting`, `sk-ant-x`, `sk-ant-y` as obvious placeholders, but the scanner does not care about value entropy. Switched to `fake-anthropic-key-*` / `fake-token-*` strings so the scanner stops flagging them. No production code touched, no behavior change — Provider::Anthropic still accepts any non-blank token.	2026-05-31 16:11:28 +02:00
Sure Admin (bot)	2df10ca4ef	Retry Enable Banking sync with provider-corrected date range (#1801) ... * Clamp Enable Banking sync window * Pipelock noise --------- Co-authored-by: KiloClaw <kiloclaw@openclaw.ai> Co-authored-by: Juan José Mata <jjmata@jjmata.com>	2026-05-17 12:09:51 +02:00
ghost	95f6451b39	feat(sync): add Brex provider connections (#1752) ... * feat(sync): add Brex provider schema Adds Brex item and account tables with per-family credentials, scoped upstream account uniqueness, encrypted token storage, and sanitized provider payload columns. * feat(sync): add Brex provider core Adds Brex item/account models, provider client and adapter support, family connection helpers, and provider enum registration for read-only Brex cash and card data. * feat(sync): add Brex import pipeline Adds Brex account discovery, linked-account sync, cash/card balance processors, transaction import, sanitized metadata handling, and idempotent provider entry processing. * feat(sync): add Brex connection flows Adds Mercury-style Brex connection management, explicit item-scoped account selection and linking, settings provider UI, account index visibility, localized copy, and per-item cache handling. * test(sync): cover Brex provider workflows Adds targeted coverage for Brex provider requests, adapter config, item/account guards, importer behavior, entry processing, and Mercury-style controller flows. * fix(sync): align Brex API edge cases Tightens Brex account fetching against the official card-account response shape, sends transaction start filters as RFC3339 date-times, and keeps provider error bodies out of user-facing messages while expanding provider client guard coverage. * fix(sync): harden Brex provider integration Restrict Brex API base URLs to official hosts, tighten account-selection UI behavior, and add tests for invalid credentials, cache scoping, and provider setup edge cases. * test(sync): avoid Brex secret-shaped fixtures * refactor(sync): extract Brex account flows * fix(sync): address Brex provider review feedback * fix(sync): address Brex review follow-ups Move remaining Brex review cleanup into focused model behavior, tighten link/setup edge cases, localize summaries, and add regression coverage from CodeRabbit feedback. Also records the security-review pass as no-findings after diff-scoped inspection and Brakeman validation. * refactor(sync): split Brex account flow controllers Route Brex account selection and setup actions through small namespaced controllers while keeping existing URLs and helpers stable. Business flow remains in BrexItem::AccountFlow; the main Brex item controller now only handles connection CRUD, provider-panel rendering, destroy, and sync. * fix(sync): address Brex CodeRabbit review * fix(sync): address Brex follow-up review * fix(sync): address Brex review follow-ups * fix(sync): address Brex sync review findings * fix(sync): polish Brex review copy and errors * fix(sync): register Brex provider health * fix(sync): polish Brex bank sync presentation * fix(sync): address Brex review follow-ups * fix(sync): tighten Brex setup params * test(api): stabilize usage rate-limit window * fix(sync): polish Brex setup flow nits * fix(sync): harden Brex setup params * fix(sync): finalize Brex review cleanup --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>	2026-05-13 18:13:48 +02:00
plind	12d799e0b8	fix(binance): support CRYPTO: prefix and USD stablecoins (#1771) ... * fix(binance): support CRYPTO: prefix and USD stablecoins Holdings processors (CoinStats, Coinbase, Kraken, SimpleFIN, Lunchflow, Binance) store crypto securities with a "CRYPTO:" prefix, but Provider::BinancePublic#parse_ticker only accepted Binance-search-style tickers like "BTCUSD". As a result, every fetched price for tickers like CRYPTO:USDT, CRYPTO:USDC, CRYPTO:SOL, CRYPTO:TRUMP, CRYPTO:KAITO failed with "Unsupported Binance ticker". - Strip the CRYPTO: prefix in parse_ticker. - Short-circuit USD-pegged stablecoins (USDT, USDC, BUSD, DAI, FDUSD, TUSD, USDP, PYUSD) to a synthetic flat 1.0 USD price. Binance has no self-pair (USDTUSDT is invalid), and the few stablecoin/USDT pairs that do exist hover at ~1.0 with sub-cent noise. - Default prefixed bare base assets (CRYPTO:SOL etc.) to the …USDT pair (USD). Only when prefixed, so unprefixed garbage like BTCBNB / BTCGBP still returns nil and the existing rejection tests still pass. - fetch_security_info returns links: nil for stablecoins rather than a broken /trade/ URL. Closes #1441. * fix(binance): strip CRYPTO: prefix in search_securities Security::Resolver calls search_provider with the raw holdings-processor symbol (CRYPTO:SOL, CRYPTO:USDT) before any price fetch. Without prefix handling here, first-time crypto imports never resolve to an online Binance security and the new stablecoin/prefix paths in parse_ticker were unreachable for that flow. - Strip CRYPTO: from the search query. - Short-circuit USD stablecoins to a synthetic search result (no exchangeInfo call, no Binance self-pair to find). - Teach parse_ticker the "{stablecoin}USD" form produced by the synthetic result so price fetches route to stablecoin_prices. --------- Co-authored-by: plind-junior <plind-junior@users.noreply.github.com>	2026-05-12 19:41:58 +02:00
ghost	be598aecf0	feat(providers): add Kraken exchange sync (#1759) ... * feat(providers): add Kraken exchange sync Adds family-scoped Kraken API-key connections, read-only balance and trade import, account setup/linking flows, provider status wiring, and focused test coverage. Closes #1758 * test(providers): avoid Kraken sample secret false positive * fix(providers): address Kraken review findings * fix(providers): address Kraken review cleanup * test(imports): stabilize transaction import ordering	2026-05-12 00:22:37 +02:00
Juan José Mata	81cdccb768	[codex] Complete Sophtron account mapping (#1698) ... * Complete Sophtron account mapping * Clarify Sophtron login challenge flow * Add Sophtron connection UI timeout * Treat Sophtron timeout jobs as failed * Reset failed Sophtron connection state * Handle stale Sophtron connection jobs * Advance Sophtron polling timeout * Shorten Sophtron connection timeout * Fix Sophtron modal polling updates * Stabilize Sophtron MFA polling * Give Sophtron OTP challenges more time * Clarify Sophtron institution login failures * Extend Sophtron polling during login progress * Probe Sophtron accounts after completed MFA step * Align Sophtron dialogs with design system * Start Sophtron initial load after linking accounts * Fix Sophtron initial transaction load * Fail Sophtron sync without institution connection * Fix tests * Wrap Sophtron account linking in transaction * Wrap Sophtron provider responses * Fix Sophtron MFA security tests * Guard Sophtron MFA challenge arrays * Respect Sophtron initial load window * Use unique Sophtron MFA answer field ids * Address Sophtron review follow-ups * Fix Sophtron transaction sync refresh * Avoid blocking Sophtron refresh polling * Move Sophtron account helpers to model * Keep Sophtron grouping provider-level * Start new Sophtron institution links * Isolate Sophtron institution connections --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>	2026-05-08 15:15:23 +02:00
GermanDZ	9cc52b9d35	fix: handle OpenAI Responses API stream errors instead of crashing (#1669) ... The streaming code assumed every stream produced a `response.completed` event and dereferenced its data unconditionally, causing `undefined method 'data' for nil` whenever OpenAI emitted `response.failed`, `response.incomplete`, or a top-level `error` event (e.g. expired `previous_response_id`, context-window overflow, transient upstream failures). Surface a descriptive `Provider::Error` instead. - Extend `ChatStreamParser` to recognise `response.failed`, `response.incomplete`, and `error` events and emit an `error` chunk with a `StreamErrorData` payload (event, message, code, details). - In `Provider::Openai#native_chat_response`, detect the missing `response` chunk, build a user-facing error message from the collected error chunk, and raise `Provider::Error`. - Add unit tests for the parser (8 cases) and integration tests for the error path in the chat response flow. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-05 01:22:05 +02:00
Abhinav Dhiman	139c89d0f4	feat(investments): add India investment subtypes and exchange support (#1659) ... * feat(investments): add India investment subtypes and exchange support * fix(yahoo-finance): scope Indian exchange de-duplication per company instead of globally Resolves feedback from Codex and CodeRabbit on #1413. prefer_indian_exchange previously collapsed all Indian securities into a single entry, silently dropping unrelated tickers. Now groups Indian listings by name and only de-duplicates within each group, so distinct companies (e.g. Reliance and Infosys) are preserved while NSE/BSE dual-listings still prefer NSE. - Derive India subtype keys dynamically from Investment::SUBTYPES in tests - Fix missing keyword arguments in Security.new test calls * refactor(yahoo-finance): generalize exchange config and dual-listing de-duplication Replaces hardcoded Indian exchange logic with a declarative EXCHANGE_CONFIG hash that maps ISO MIC codes to Yahoo-specific settings (symbol suffix, default currency, dual-listing group, and preference rank). This makes adding new markets a one-line hash entry instead of scattered conditionals. * fix(yahoo-finance): normalize security names for dual-listing de-duplication * fix(yahoo-finance): skip dual-listing de-duplication when filtering by exchange * fix: address PR review feedback for India market support - fix cache key mismatch in fetch_security_price by normalizing symbol before building cache key - remove dead YAHOO_EXCHANGE_CURRENCY constant - tighten normalize_symbol guard to use end_with?(suffix) instead of include?('.') - remove misleading '# India' comment from Property::SUBTYPES - remove 'rented' property subtype in favor of 'investment_property' - rename 'demat' to 'indian_stocks' for clarity - add INR to CURRENCY_REGION_MAP so India appears first for INR users - add dotted-symbol regression test for normalize_symbol * fix(investments): rename 'demat' subtype to 'indian_stocks' and remove trailing comma	2026-05-05 01:04:29 +02:00
ghost	6c84fc760e	fix(mercury): support named multiple API connections (#1627) ... * fix(mercury): support named multiple connections * fix(mercury): address multi-connection review feedback * fix(mercury): localize connection labels * fix(mercury): strip API tokens before provider calls * test(mercury): localize provider config assertions * fix(mercury): address multi-connection review * refactor(mercury): simplify connection selection failure	2026-05-03 10:56:31 +02:00
Juan José Mata	7b2b1dd367	Rebase PR #784 and fix OpenAI model/chat regressions (#1384) ... * Wire conversation history through OpenAI responses API * Fix RuboCop hash brace spacing in assistant tests * Pipelock ignores * Batch fixes --------- Co-authored-by: sokiee <sokysrm@gmail.com>	2026-04-15 18:45:24 +02:00
soky srm	dcebda05de	Move back to brandfetch (#1427) ... * Move back to brandfetch * Update security.rb * Update security.rb	2026-04-10 17:42:16 +02:00
soky srm	0aca297e9c	Add binance security provider for crypto (#1424) ... * Binance as securities provider * Disable twelve data crypto results * Add logo support and new currency pairs * FIX importer fallback * Add price clamping and optiimize retrieval * Review * Update adding-a-securities-provider.md * day gap miss fix * New fixes * Brandfetch doesn't support crypto. add new CDN * Update _investment_performance.html.erb	2026-04-10 15:43:22 +02:00
soky srm	be42988adf	Add throttling and cross-rate for twelve data (#1396) ... * Add throttling and cross-rate for twelve data * FIX yahoo precision also * FIXES * Update importer.rb * Fixes * Revert job * Fixes	2026-04-07 20:46:05 +02:00
Louis	455c74dcfa	Add Binance support, heavily inspired by the Coinbase one (#1317) ... * feat: add Binance support (Items, Accounts, Importers, Processor, and Sync) * refactor: deduplicate 'stablecoins' constant and push stale_rate filter to SQL --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>	2026-04-07 14:43:17 +02:00
Serge L	15cfcf585d	Fix: Yahoo Finance provider Cookie/Crumb Auth (#1082) ... * Fix: use cookie/crumb auth in healthy? chart endpoint check The health check was calling /v8/finance/chart/AAPL via the plain unauthenticated client. Yahoo Finance requires cookie + crumb authentication on the chart endpoint, so the health check would fail even when credentials are valid. Updated healthy? to use fetch_cookie_and_crumb + authenticated_client, consistent with fetch_security_prices and fetch_chart_data. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix: add cookie/crumb auth to all /v8/finance/chart/ calls fetch_security_prices and fetch_chart_data (used for exchange rates) were calling the chart endpoint without cookie/crumb authentication, inconsistent with healthy? and fetch_security_info. Added auth to both, including the same retry-on-Unauthorized pattern already used in fetch_security_info. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Update user-agent strings in yahoo_finance.rb Updated user-agent strings to reflect current browser versions Signed-off-by: Serge L <serge@souritech.ca> * Fix: Add stale-crumb retry to healthy? and fetch_chart_data Yahoo Finance returns 200 OK with {"chart":{"error":{"code":"Unauthorized"}}} when a cached crumb expires server-side. Both healthy? and fetch_chart_data now mirror the retry pattern already in fetch_security_prices: detect the Unauthorized body, clear the crumb cache, fetch fresh credentials, and retry the request once. Adds a test for the healthy? retry path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Refactor: Extract fetch_authenticated_chart helper to DRY crumb retry logic The cookie/crumb fetch + stale-crumb retry pattern was duplicated across healthy?, fetch_security_prices, and fetch_chart_data. Extract it into a single private fetch_authenticated_chart(symbol, params) helper that centralizes the retry logic; all three call sites now delegate to it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix: Catch JSON::ParserError in fetch_chart_data rescue clause After moving JSON.parse inside fetch_authenticated_chart, a malformed Yahoo response would throw JSON::ParserError through fetch_chart_data's rescue Faraday::Error, breaking the inverse currency pair fallback. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix: Raise AuthenticationError if retry still returns Unauthorized After refreshing the crumb and retrying, if Yahoo still returns an Unauthorized error body the helper now raises AuthenticationError instead of silently returning the error payload. This prevents callers from misinterpreting a persistent auth failure as missing chart data. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix: Raise AuthenticationError after failed retry in fetch_security_info Mirrors the same post-retry Unauthorized check added to fetch_authenticated_chart. Without this, a persistent auth failure on the quoteSummary endpoint would surface as a generic "No security info found" error instead of an AuthenticationError. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Signed-off-by: Serge L <serge@souritech.ca> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-03-01 22:48:48 +01:00
Juan José Mata	90e94f0ad1	Use Langfuse client trace upsert API (#1041) ... Replace direct trace.update calls with client trace upserts so OpenAI provider is compatible with langfuse-ruby 0.1.6 behavior. Add richer warning logs that include full exception details for trace creation, trace upserts, and generation logging failures. Add tests for client-based trace upserts and detailed error logging.	2026-02-23 09:29:21 -05:00
David Gil	ba442d5f26	Implement Indexa Capital provider with real API integration (#933) ... * Add Indexa Capital provider scaffold Generate Indexa Capital provider scaffolding and align credential fields with the API authentication requirements. * Fix PR 926 lint and schema CI failures * Implement Indexa Capital provider with real API integration - Rewrite all broken view templates (were meta-ERB from code generator) - Create missing select_accounts.html.erb template - Implement real API calls: list_accounts via /users/me, get_holdings via /accounts/{number}/fiscal-results, get_account_balance via /accounts/{number}/performance - Add API token auth support (stored token > env token > credentials) - Add api_token column with encryption support - Redesign settings panel: API token prominent, credentials collapsible - Fix account balances display using performance endpoint portfolios - Fix accounts index empty-state guard missing indexa_capital_items - Simplify activities fetch job (no activities API endpoint exists) - Fix i18n interpolation (%%{ -> %{) throughout locale file * Add tests for Indexa Capital provider integration - IndexaCapitalItemTest: validations, credentials, scopes, sync status - IndexaCapitalAccountTest: upsert, holdings, account provider linking - Provider::IndexaCapitalTest: auth modes, API stubs, error handling - IndexaCapitalItemsControllerTest: CRUD, setup, linking, authorization - Fixtures for items (token + credentials) and accounts (mutual + pension) 52 tests, 98 assertions, 0 failures * Address code review feedback from PR #933 - Fix zero balance bug: use `nil?` instead of `present?` so 0 is stored - Fix has_indexa_capital_credentials? to check api_token (was ignored) - Fix build_provider to delegate to Provided concern (was ignoring token) - Fix IndexaCapital section outside encryption_error guard in settings - Add account_number sanitization to prevent path traversal in API URLs - Replace all skipped processor tests with real working tests - Add zero-balance and path-traversal test coverage 61 tests, 107 assertions, 0 failures * Address code review round 2: credentials validation, RuboCop, test quality - Fix RuboCop SpaceInsideArrayLiteralBrackets in credentials check - Chain where.not calls so all three username/document/password must be present - Require all three credentials (||) instead of any one (&&) in validate_configuration! - Move attr_reader to private to avoid exposing credentials publicly - Parse dates with Date.parse in extract_balance for robustness - Remove stale TODO and Crypto from supported_account_types - Order build_provider query deterministically by created_at - Replace no-op holdings assertion with meaningful assert_difference * Address code review round 3: JSON parse safety and test precision - Rescue JSON::ParserError on 2xx responses for clearer error messages - Fix weak balance assertion: set balance to 0 before processing, assert expected value (27093.01 = sum of holdings amounts) * Include Indexa Capital in automatic family sync Add indexa_capital_items to Family::Syncer#child_syncables so balances and holdings refresh on daily auto-sync and login sync, not only on manual sync button clicks. --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <jjmata@jjmata.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>	2026-02-08 18:19:37 +01:00
BitToby	ba6e286b41	feat: add SSL_CA_FILE and SSL_VERIFY environment variables to support… (#894) ... * feat: add SSL_CA_FILE and SSL_VERIFY environment variables to support self-signed certificates in self-hosted environments * fix: NoMethodError by defining SSL helper methods before configure block executes * refactor: Refactor SessionsController to use shared SslConfigurable module and simplify SSL initializer redundant checks * refactor: improve SSL configuration robustness and error detection accuracy * fix:HTTParty SSL options, add file validation guards, prevent Tempfile GC, and redact URLs in error logs * fix: Fix SSL concern indentation and stub Simplefin POST correctly in tests * fix: normalize ssl_verify to always return boolean instead of nil * fix: solve failing SimpleFin test * refactor: trim unused error-handling code from SslConfigurable, replace Tempfile with fixed-path CA bundle, fix namespace pollution in initializers, and add unit tests for core SSL configuration and Langfuse CRL callback. * fix: added require ileutils in the initializer and require ostruct in the test file. * fix: solve autoload conflict that broke provider loading, validate all certs in PEM bundles, and add missing requires.	2026-02-06 18:04:03 +01:00
MkDev11	0afdb1d0fd	Feature/pdf import transaction rows (#846) ... * Add import row generation from PDF extracted data - Add generate_rows_from_extracted_data method to PdfImport - Add import! method to create transactions from PDF rows - Update ProcessPdfJob to generate rows after extraction - Update configured?, cleaned?, publishable? for PDF workflow - Add column_keys, required_column_keys, mapping_steps - Set bank statements to pending status for user review - Add tests for new functionality Closes #844 * Add tests for BankStatementExtractor - Test transaction extraction from PDF content - Test deduplication across chunk boundaries - Test amount normalization for various formats - Test graceful handling of malformed JSON responses - Test error handling for empty/nil PDF content * Fix supports_pdf_processing? to validate effective model The validation was always checking @default_model, but process_pdf allows overriding the model via parameter. This could cause a vision-capable override model to be rejected, or a non-vision-capable override to pass validation only to fail during processing. Changes: - supports_pdf_processing? now accepts optional model parameter - process_pdf passes effective model to validation - Raise Provider::Openai::Error inside with_provider_response for consistent error handling Addresses review feedback from PR#808 * Fix insert_all! bug: explicitly set import_id Rails insert_all! on associations does NOT auto-set the foreign key. Added import_id explicitly and use Import::Row.insert_all! directly. Also reload rows before counting to ensure accurate count. * Fix pending status showing as processing for bank statements with rows When bank statement PDF imports have extracted rows, show a 'Ready for Review' screen with a link to the confirm path instead of the 'Processing' spinner. This addresses the PR feedback that users couldn't reach the review flow even though rows were created. * Gate publishable? on account.present? to prevent import failure PDF imports are created without an account, and import! raises if account is missing. This prevents users from hitting publish and having the job fail. * Wrap generate_rows_from_extracted_data in transaction for atomicity - Clear rows and reset count even when no transactions extracted - Use transaction block to prevent partial updates on failure - Use mapped_rows.size instead of reload for count * Localize transactions count string with i18n helper * Add AccountMapping step for PDF imports when account is nil PDF imports need account selection before publishing. This adds Import::AccountMapping to mapping_steps when account is nil, matching the behavior of TransactionImport and TradeImport. Addresses PR#846 feedback about account selection for PDF imports. * Only include CategoryMapping when rows have non-empty categories PDF extraction doesn't extract categories from bank statements, so the CategoryMapping step would show empty. Now we only include CategoryMapping if rows actually have non-empty category values. This prevents showing an empty mapping step for PDF imports. * Fix PDF import UI flow and account selection - Add direct account selection in PDF import UI instead of AccountMapping - AccountMapping designed for CSV imports with multiple account values - PDF imports need single account for all transactions - Add update action and route for imports controller - Fix controller to handle pdf_import param format from form_with - Show Publish button when import is publishable (account set) - Fix stepper nav: Upload/Configure/Clean non-clickable for PDF imports - Redirect PDF imports from configuration step (auto-configured) - Improve AI prompt to recognize M-PESA/mobile money as bank statements - Fix migration ordering for import_rows table columns * Add guard for invalid account_id in imports#update Prevents silently clearing account when invalid ID is passed. Returns error message instead of confusing 'Account saved' notice. * Localize step names in import nav and add account guard - Use t() helper for all step names (Upload, Configure, Clean, Map, Confirm) - Add guard for invalid account_id in imports#update - Prevents silently clearing account when invalid ID is passed * Make category column migrations idempotent Check if columns exist before adding to prevent duplicate column errors when migrations are re-run with new timestamps. * Add match_path for PDF import step highlighting Fixes step detection when path is nil by using separate match_path for current step highlighting while keeping links disabled. * Rename category migrations and update to Rails 7.2 - Rename class to EnsureCategoryFieldsOnImportRows to avoid conflicts - Rename class to EnsureCategoryIconOnImportRows - Update migration version from 7.1 to 7.2 per guidelines - Rename files to match class names - Add match_path for PDF import step highlighting * Use primary (black) style for Create Account and Save buttons * Remove match_path from auto-completed PDF steps Only step 4 (Confirm) needs match_path for active-step detection. Steps 1-3 are purely informational and always complete. * Add fallback for document type translation Handles nil or unexpected document_type values gracefully. Also removes match_path from auto-completed PDF steps. * Use index-based step number for mobile indicator Fixes 'Step 5 of 4' issue when Map step is dynamically removed. * Fix hostings_controller_test: use blank? instead of nil Setting returns empty string not nil for unset values. * Localize step progress label and use design token * Fix button styling: use design system Tailwind classes btn--primary and btn--secondary CSS classes don't exist. Use actual design system classes from DS::Buttonish. * Fix CRLF line endings in tags_controller_test.rb --------- Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>	2026-02-02 16:27:02 +01:00
Juan José Mata	6b5a5b1877	fix: Show cancellation message when subscription is pending cancellation (#752) ... * fix: Show cancellation message when subscription is pending cancellation When a subscription is cancelled via Stripe, the UI incorrectly showed "Your contribution continues on..." instead of reflecting the cancellation status. This fix adds tracking of `cancel_at_period_end` from Stripe webhooks and displays "Your contribution ends on..." when a subscription has been cancelled but is still active until the billing period ends. https://claude.ai/code/session_01Y8ELTdK1k9o315iSq43TRN * chore: Update schema.rb with cancel_at_period_end column https://claude.ai/code/session_01Y8ELTdK1k9o315iSq43TRN * Schema version --------- Co-authored-by: Claude <noreply@anthropic.com>	2026-01-23 18:55:51 +01:00
soky srm	179552657c	Mercury integration (#723) ... * Initial mercury impl * FIX both mercury and generator class * Finish mercury integration and provider generator * Fix schema * Fix linter and tags * Update routes.rb * Avoid schema drift --------- Signed-off-by: soky srm <sokysrm@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>	2026-01-22 20:37:07 +01:00
soky srm	0e22c60286	FIX Yahoo issues (#636) ... * FIX Yahoo issues * Update yahoo_finance_test.rb * FIX proper cookie access	2026-01-13 16:46:07 +01:00