mirror of
https://github.com/we-promise/sure.git
synced 2026-08-05 16:42:18 +00:00
b338619ef3a8a4c4d4e07ca6265669a54236d2ef
180 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Oscar
|
c6a240a183 |
feat(redbark): add australian bank sync (redbark) (#2794)
* add redbark provider integration - per family api key provider, built like the lunchflow integration - syncs accounts, balances and transactions from api.redbark.com - account setup flow, settings panel, locales and routes - tests and fixtures * harden redbark integration based on prior provider pr feedback - use DebugLogEntry.capture for sync/import/unlink failures - retry 429s and 5xxs with backoff, raise on page cap instead of truncating - keep raw response bodies out of logs and errors - not null constraints on account columns, migration base 7.2 - persist ignored flag for skipped accounts so they stop nagging setup - validate api key on every save, re-arm status on key rotation - destroy aborts if unlink fails, atomic account create and link - require_admin on mutating actions, see_other on error redirects - single grouped query for item account counts - i18n default connection name, blank password field value - controller and provider tests * fix issues found in second review sweep - add missing syncable scope, without it every family sync raises - kick off a sync on connection create and on key rotation - setup dialog fetches accounts inline for fresh connections and shows api errors - skip balance write when no balance has been fetched yet, never anchor a false zero - exclude stale and non banking accounts from the batched balances call, per account fallback if the batch is rejected - detect the server row ceiling and empty pages instead of silently truncating history - user sync start date only governs the initial backfill, incremental after that - fetch connections before the per account loop so auth errors propagate once - drop untemplated index/show/new/edit routes and dead preload/link_accounts actions - stable dom id on the settings panel so repeat turbo replaces keep working * skip brokerage connections, found in live testing - the transactions endpoint 400s for brokerage connections, they belong to /v1/trades - only import accounts from banking and documents connections - guard transaction fetches for any legacy linked non banking account * address review feedback - treat the truncation header as a pagination signal: split the date window and refetch instead of failing the account - prune stale pending rows from the snapshot so settled pendings cant come back as duplicates - block linking a sure account that already has another provider feed - count setup failures separately from skips and surface an error instead of "all skipped" - add not nulls on redbark_items name and api key - enqueue the destroy job after the flag commits, not inside the transaction - swap bg-gray-400 for bg-surface-inset, drop amounts from info logs, remove i18n default fallbacks - tests for window splitting, pending pruning and encrypted payload round trip * fix issues from convention review - benign skips (unlinked account, blank id, unparseable rows) no longer count as failures, tracked separately so a clean batch reports success - currency parsing goes through extract_currency so hash shaped payloads resolve instead of falling to the default - merchant ids use truncated sha256 instead of md5 - debug log entries for import failures and account sync scheduling failures * bound the raw transactions snapshot to the fetch window - trim raw_transactions_payload to the current fetch window on merge, same as brex - keep rows without a parseable date, drop settled pendings as before - surface skipped rows in the aggregate debug log entry with imported/skipped counts |
||
|
kai392
|
6e2cbb727a |
fix(ai): prevent Anthropic chat crash on no-argument tool calls (#2755)
* fix: prevent Anthropic chat crash on no-argument tool calls
When the model streams a tool_use block with no arguments (e.g.
get_categories), the accumulated input arrives as an empty string.
The Anthropic chat parser passed that empty string straight through as
function_args, and Assistant::FunctionToolCaller then called
JSON.parse("") — which raises "unexpected end of input", surfaces as a
Provider::Anthropic::Error, and kills the whole assistant turn.
Fix at the source by normalizing empty/nil tool input to an empty JSON
object in the parser, plus a defensive guard in FunctionToolCaller so
any provider that emits blank arguments cannot crash a turn.
Fixes #2722
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: assert nil-args result as Hash to match jsonb function_result
Addresses Codex P1 / CodeRabbit review: EchoFunction returns the parsed
params Hash and ToolCall::Function stores it in a jsonb column, so the
nil-arguments test must assert the Hash directly instead of JSON.parse.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
RealDiligent
|
d61ee3abea |
fix: omit SimpleFin pending param when pending transactions are disabled (#2796)
The SimpleFIN protocol only defines pending=1 (include pending); pending transactions are excluded by default when the param is absent. Bridges presence-check the param, so the pending=0 we sent when the 'Include pending transactions' setting (or SIMPLEFIN_INCLUDE_PENDING=0) was disabled behaved exactly like pending=1, making the setting a no-op — pending transactions kept being downloaded, causing pending/posted duplicates and churn. Omit the pending query param entirely unless pending is enabled, per the spec. Fixes #2440 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
jdcdp
|
849578a84a |
Add support for trading212 integration in investments sync (#2513)
* Add support for trading212 integration in investments sync * Respect Trading 212 history rate limits * Replace hex colors with design tokens * Changed withdrawal to withdraw * Address PR review feedback for Trading 212 integration - Replace all Rails.logger calls with DebugLogEntry.capture across syncer, importer, processor, provider, controller, and unlinking - Fix destroy action to surface unlink errors via DebugLogEntry - Replace hardcoded 'GBP' currency fallback with Current.family.currency - Replace hardcoded English status strings in Syncer with i18n - Add config/initializers/trading212.rb with DEBUG_RAW ENV toggle - Add comprehensive test coverage: item, account, data_helpers, holdings_processor, activities_processor, importer, syncer, and controller tests - Rewrite syncer and controller tests to match actual interfaces * Fix tests and bugs found during local test execution - Fix standard_ticker: empty string caused nil.upcase via [].first - Fix parse_date: DateTime < Date, so when Date caught DateTime first - Fix ActivitiesProcessor/HoldingsProcessor: missing instruments_map in DataHelpers caused NameError when processing dividends - Fix test: security fixture ticker collision with test data - Fix test: sync_status_summary i18n matching in assertions - Fix test: trading212_provider ConfigurationError test path - Fix test: controller invalid params needs Turbo-Frame header - Fix test: syncer test uses stubs instead of strict expects 116 tests, 257 assertions, 0 failures, 0 errors * Keep raw provider response bodies out of exception messages. * Fix Secrets leak into page HTML, prevents the API key/secret from appearing in the HTML source while keeping the "leave blank to keep existing" UX. * Address review findings: env gate, sync test, uniqueness test, destroy flow - Gate TRADING212_DEBUG_RAW behind Rails.env.local? so staging/production cannot accidentally enable raw payload logging - Assert SyncJob enqueue in sync controller test, not just redirect - Fix cross-item uniqueness test to actually use two different items - Remove rescue in destroy so unlink failures stop the flow (matching Brex/Akahu pattern) instead of proceeding to destroy_later silently * Add Trading212 tables to db/schema.rb for CI test database CI runs db:test:prepare which loads db/schema.rb, not migrations. Without these table definitions, fixture loading fails with PG::UndefinedTable: relation "trading212_accounts" does not exist. * Removed lint complaint * Register Trading212 in ProviderConnectionStatus::PROVIDERS Fixes CI failure: test_provider_registry_covers_syncable_family_provider_item_associations * Fixed bad merge --------- Signed-off-by: jdcdp <47483528+jdcdp@users.noreply.github.com> Co-authored-by: jdcdp <jdcdp@cdm4.net> |
||
|
kianrafiee
|
d267288a45 |
Add RentCast & Realie integration for automatic property data and valuations (#2727)
* Add RentCast and Realie AVM providers for property valuation Adds Automated Valuation Model (AVM) provider support so self-hosted users can create property accounts from a US address lookup instead of entering details manually: - Provider::Rentcast and Provider::Realie clients, each fetching the property record (type, year built, square footage) and value estimate in a single API request, registered under a new :property_valuations registry concept - API key fields (encrypted, ENV-overridable) with monthly usage display in Settings > Self-Hosting under "Property Valuation Providers" - New property flow: when a provider key is configured, the method selector offers "Add via RentCast/Realie" alongside manual entry; the lookup form reuses the manual flow's localized address fields and creates the account active with fetched attributes, valuation as balance, and the address saved - SyncPropertyValuationsJob refreshes linked property valuations once a day (never hourly) via config/schedule.yml; each provider enforces its monthly request cap (RentCast 50, Realie 25) with a calendar-month counter shared between creation lookups and refreshes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Allow ENV override of AVM provider monthly request caps The RentCast (50/month) and Realie (25/month) budgets default to the free tier limits but can now be raised for paid plans via RENTCAST_MAX_REQUESTS_PER_MONTH / REALIE_MAX_REQUESTS_PER_MONTH, mirroring the AlphaVantage and Tiingo request limit overrides. The settings descriptions and usage display reflect the effective cap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review feedback: durable request counters, safer refresh job - Move monthly AVM request counters from Rails.cache to a new provider_request_counts table with atomic upsert increments, so the hard budget caps survive cache eviction, restarts, and Redis flushes - SyncPropertyValuationsJob: only mark a property synced when the balance update succeeds (wrapped in a transaction), process stalest valuations first so a tight budget goes where it matters, and report failures via DebugLogEntry.capture instead of Rails.logger - Realie: reject lookups whose returned city/ZIP contradict the entered address (street+state queries can match the wrong city); document that numeric use codes are unpublished and leave the subtype unset - Add Faraday open/request timeouts to both provider clients - Localize all user-facing provider error messages (config/locales/models/provider/en.yml) - Add a check constraint restricting properties.avm_provider to known providers - Use the min-h-80 scale token instead of min-h-[320px] - Tests: exact-argument expectations on the lookup stub, provider stubs in the no-provider test, RentCast/Realie API key update tests, ProviderRequestCount unit tests, failed-balance regression test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Stub AVM provider registry lookups in settings system test Settings::HostingsController#show now resolves the RentCast and Realie providers for usage display; the system test's partial get_provider stubbing treats the new lookups as unexpected invocations without these. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address second-round review: candidate scan, provider reuse, shared row partial - Realie: when the address lookup returns multiple candidates, pick the first one consistent with the entered city/ZIP instead of judging only the first array element; the location-mismatch error now only fires when no candidate matches - SyncPropertyValuationsJob: resolve one provider instance per key for the whole run, so the per-instance request throttle actually spaces requests across properties instead of resetting on each iteration - Extract the AVM method selector row into a shared partial so the manual and provider options render one shape Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add retry middleware and sync index from review feedback - Both AVM clients now retry transient connection failures (same Faraday retry config as Tiingo) so a network blip doesn't burn one of the tight monthly budget's requests - Partial index on properties(avm_provider, avm_last_synced_on) backing the daily sync job's filter and stalest-first ordering Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Key the AVM sync index on avm_last_synced_on The daily job orders by avm_last_synced_on ASC NULLS FIRST across all AVM-linked properties, so leading the partial index with avm_provider prevented it from serving the sort. Rekeyed on avm_last_synced_on with matching null ordering; the partial predicate still covers the filter. Amended in place since the migration is unmerged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Map Realie numeric use codes; skip incomplete addresses in sync job - Realie documents its numeric use codes at docs.realie.ai/api-reference/feature-key (earlier skip reason was wrong — the table exists). Map the residential/land/agricultural codes to Property subtypes, leaving codes without a subtype equivalent (e.g. 1006 mobile/manufactured) unset like the RentCast mapping. - The daily job now skips properties whose address is missing a street or state before resolving a provider, logging a warn-level DebugLogEntry, so a malformed address can't burn a monthly-budget request every day. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Guard currency mismatches in refresh; reset AVM keys in test teardown - The daily job now skips (with a warn-level DebugLogEntry) properties whose account currency no longer matches the provider's valuation currency, checked via the concept's valuation_currency before spending a request — writing a USD valuation into a re-currencied account would corrupt the balance - Hostings controller test teardown now clears rentcast_api_key and realie_api_key alongside the other cached global settings Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Validate AVM lookup inputs before spending a provider request The form marks name and address fields required, but a forged or JS-less submission bypasses that and would burn one of the tight monthly-budget requests on a lookup that can't produce a property. Property::AvmImport now validates name and the full address locally (localized error) before calling the provider. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add preview-and-confirm step to the AVM property lookup RentCast's AVM endpoint is location-based: a plausible but nonexistent address still geocodes and returns an area-derived estimate with no property record behind it. Instead of silently creating an account from that, the lookup now shows what the provider returned — type, year built, area (each "Unknown" when missing), and the estimated value — with an explicit notice when no property record was found, and only the user's confirmation creates the account. The confirm step reuses the fetched data via the form (sanitized server-side), so the flow still costs exactly one provider request. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Require a complete US address before daily valuation refresh The job's guard only required street and state, but a later-blanked city or ZIP silently disables the Realie wrong-city check (blank entered fields count as "no mismatch"), so a refresh could accept another property's valuation. The guard now mirrors the import-time completeness check and also skips addresses edited to a non-US country. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Sign the AVM preview payload; blur valuation in privacy mode - The confirm step now rebuilds everything from a signed, 1-hour message-verifier token generated at lookup time, proving the lookup (and its counted provider request) actually ran — a direct confirm POST with fabricated data can no longer create provider-linked properties that the daily refresh job would then spend quota on. Forged/expired tokens re-render the lookup form with a clear error. - The preview's market value now carries the privacy-sensitive class so privacy mode blurs it like other account amounts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Treat zero Realie valuations as absent; bind preview token to family Review follow-ups from PR #2727: - Realie returns modelValue: 0 when it can't produce an AVM estimate; zero now falls back to the assessed market value instead of syncing a $0 balance, and an all-zero record errors out. - Monetary values now parse via BigDecimal(value.to_s) rather than Float#to_d, avoiding binary float rounding artifacts in balances. - The signed AVM preview token's purpose is scoped to the family that ran the lookup, so a leaked token can't be replayed cross-family. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Refresh AVM property valuations monthly instead of daily Per @jjmata's review on #2727: valuations change slowly and providers enforce tight monthly request caps, so a daily cron adds no value. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Max Barbare
|
51c93649da |
feat(snaptrade): replace device-flow OAuth with authorization-code + PKCE flow (#2747)
* feat(snaptrade): replace device-flow OAuth with authorization-code + PKCE flow Squashed from 16 commits on snaptrade-oauth-apps for a clean rebase onto current upstream/main ahead of opening a PR. * fix(snaptrade): address PR #2747 review feedback on OAuth PKCE flow - Remove unreachable dead-code guard in import_latest_snaptrade_data - Guard apply_oauth_tokens! against a malformed payload missing access_token - Wrap token endpoint network errors in ApiError and retry like data calls - Remove unused Provider::Snaptrade#revoke_token! instance method - Preserve return_to/accountable_type through the SnapTrade portal callback so the account-linking flow no longer drops users back to accounts_path - Show the real absolute OAuth callback URL in self-hosted setup instructions - Refresh brakeman.ignore fingerprint for the connect redirect after the return_to/accountable_type params were added Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8SCCmKX6RphB5E73WSUQQ * fix(snaptrade): don't retry non-idempotent OAuth/API requests CodeRabbit flagged that Provider::Snaptrade retried OAuth token exchanges/refreshes and all API POST/DELETE calls (get_connection_url, delete_connection) after timeouts/connection failures. If the response is lost after SnapTrade already consumed a single-use auth code, rotated the refresh token, or applied a POST/DELETE, replaying the request either fails with invalid_grant on a token that actually succeeded, or risks duplicate side effects. Retries are now limited to GET requests; OAuth token requests and non-GET API calls translate a network failure straight into an ApiError without replay. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrrGkgSBEqhjjBmmH1fcXL * fix(snaptrade): stop querying non-deterministically encrypted token via empty-string compare CodeRabbit flagged that the syncable scope's where.not(oauth_access_token: [nil, ""]) re-encrypts "" with a random IV on every query, so the "" comparison can never match a stored ciphertext and is a silent no-op. No code path ever persists oauth_access_token as "" (only nil or a real token via apply_oauth_tokens!), so the exclusion is unnecessary -- narrowed the scope to a plain NULL check, which encryption handles transparently since nil is never encrypted. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrrGkgSBEqhjjBmmH1fcXL --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Juan José Mata
|
c6eb7cdeed |
Revert "Refactor application workflows and update test coverage"
This reverts commit
|
||
|
Juan José Mata
|
565e049f89 | Refactor application workflows and update test coverage | ||
|
Camilo Agudelo
|
7657f2d08c |
Improve Yahoo Finance reliability and Colombian listing support (#2738)
* Prevent Yahoo crumb cache poisoning * Make Yahoo Finance health status rate-limit aware * Normalize Yahoo Colombian listings Map Yahoo's BVC code to XBOG so search results derive Colombia from canonical exchange metadata and price requests use the .CL suffix with a COP fallback. Spec: .scratch/normalize-yahoo-colombia-listings/spec.md * Move Yahoo health checks to background |
||
|
Carlos Lindo
|
5b9fd1e1a0 |
fix(enable-banking): handle unavailable and overdraft balances (#2578)
* fix(enable-banking): handle overdraft balance sync * fix(enable-banking): skip nil provider balances * test(enable-banking): cover nil provider balance * fix(enable-banking): avoid stale and unsafe balance logs * fix(enable-banking): guard unavailable balance writes * test(enable-banking): stub unsaved account api id * style(enable-banking): remove extra test blank line --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
Jestin J Palamuttam
|
e324170160 |
fix(provider): resolve Tiingo security currency from countryCode (#2692)
* fix(provider): resolve Tiingo security currency from countryCode Tiingo's search API never returns the priceCurrency field the code was reading, so currency detection silently failed. Currency is now derived from countryCode via the countries gem's ISO 4217 data, with a best-match tie-break for tickers that collide across countries so the currency shown in search results always matches what fetch_security_prices later returns. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(provider): guard tiingo currency cache against non-US downgrade Addresses PR review feedback (jjmata): the per-ticker currency cache written in search_securities was unconditionally overwritten on every search, keyed only by ticker. A later search whose result set doesn't happen to include a ticker's US cross-listing could silently downgrade a previously-cached USD (the currency actually backing daily price data) to a foreign currency. Only overwrite when the current search's match is US, or nothing is cached yet. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Blaž Dular
|
826c4a356e |
feat(bank-sync): Wise integration (#2433)
* feat(wise): add Wise integration with JAR savings account and activity support
- Add WiseItem/WiseAccount models with full sync pipeline (importer, syncer, processor)
- Detect income vs expense using targetAccount == recipientId from borderless accounts API
- Support JAR (SAVINGS) accounts with totalWorth balance and savings subtype
- Fetch JAR activity via profile activities API (INTERBALANCE, BALANCE_CASHBACK, BALANCE_ASSET_FEE)
- Route INTERBALANCE activities to both JAR and STANDARD accounts and link as Transfer records
- Add provider connection status registration, routes, views, and i18n
- Add migration for wise_items and wise_accounts tables
- Add tests for WiseAccount, WiseEntry::Processor, WiseActivity::Processor, WiseItem::Importer, and WiseItem#link_jar_transfers!
* chore(lint): add ignore to security scan (false positive)
* fix(wise): address PR review feedback on activity routing, HTML stripping, rate limiting, and scope extraction
- Replace title string matching in activity_for_account? with resource.id vs balance_id comparison to avoid breakage when users rename JAR accounts on Wise
- Replace gsub(/<[^>]+>/, "") with ActionController::Base.helpers.strip_tags to safely handle user-controlled HTML-like content
- Wrap paginated API calls in with_rate_limit_retry (up to 3 attempts, exponential backoff) to handle 429 responses during fetch_jar_activities and fetch_transfers
- Extract WiseAccount.unlinked scope and remove duplicated left_joins query from controller
* fix(wise): address PR #2433 review feedback
- Wire @wise_items into AccountsController#index so linked accounts appear on /accounts
- Fix find_wise_account_for_linking to preserve .active scope via .merge instead of .then
- Fix cross-currency incoming transfer amount to use targetValue instead of sourceValue
- Add missing select_profiles.session_expired locale key
- Add missing account_name interpolation to link_existing_account success notice
- Use i18n for profile type labels in profile_display_name
- Trigger wise_item.sync_later after account linking in all three linking actions
- Isolate fee transaction failure so it no longer aborts the main transfer import
- Use bare raise in WiseActivity/WiseEntry processors to preserve original backtrace
- Re-raise in WiseItem::Unlinking to prevent silently orphaned Holdings
- Fix avatar badge to use design system tokens (bg-container-inset, text-primary)
* fix(wise): fix INTERBALANCE routing and encrypt pending token in session
* fix(wise): replace hand-rolled buttons/links with DS::Button and DS::Link
Address repeated sure-design DS drift findings: migrate all manual
Tailwind button_to and link_to calls in Wise views to DS::Button
(outline/outline_destructive variants) and DS::Link (secondary variant).
Add missing provider_panel.disconnect locale key for the disconnect button text.
* fix(wise): use render_provider_panel_error in create instead of missing new template
* fix(wise): add missing comma in PROVIDERS array
CI failed with a SyntaxError because the questrade entry wasn't
comma-terminated before the wise entry.
* chore(wise): re-add pipelock:ignore for pending token param
Lost when the token source moved from session to an encrypted params
field in
|
||
|
Jestin J Palamuttam
|
e6a0ca597b |
feat(provider): add Frankfurter as an exchange-rate provider (#2640)
* feat(exchange-rates): add Frankfurter as an exchange-rate provider Frankfurter (frankfurter.dev) is a free, keyless FX rates API backed by ECB daily reference rates, with no published rate limit and no auth flow to maintain (unlike Yahoo Finance's reverse-engineered cookie/ crumb auth or TwelveData's fast-exhausting free tier). Follows the Provider::MoexPublic template: Faraday client with retry middleware, SslConfigurable for self-hosted CA support, a light RateLimitable throttle, and a FRANKFURTER_URL env escape hatch for self-hosters. Registered as exchange-rates-only (no security/stock data) and added to the hosting settings dropdown. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * refactor(provider): switch Frankfurter to the v2 API v1 is explicitly marked "frozen" on Frankfurter's own root endpoint; v2 is "current" and covers 201 currencies across 84 central banks vs v1's ~30 ECB-only. Confirmed via the v2 OpenAPI spec and live requests: - Single-date lookups now use GET /rate/{base}/{quote}?date=..., which carries weekends/holidays forward server-side (a Saturday returns a real rate directly), so the provider no longer needs its own lookback-window logic. - Range lookups now use GET /rates?base=..."es=...&from=...&to=..., a flat array of { date, base, quote, rate } records (v2's shape) instead of v1's { "rates": { date => currencies } } hash. - Every calendar day in a range is present (v2 gapfills itself), rather than v1's omit-non-trading-days behavior. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * fix(provider): sanitize currency codes before URL path interpolation from/to were only upcased before being interpolated directly into the URL path in fetch_exchange_rate (GET /rate/{from}/{to}). Low risk since currency codes come from validated internal sources, but adds cheap defense-in-depth: strip anything that isn't A-Z, matching the ISO 4217 format real currency codes always take. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Juan José Mata
|
8d649bc5b9 |
perf: make provider retry backoff sleeps stubbable in tests (#2552)
Provider::Simplefin, Provider::Akahu and Provider::Up called Kernel.sleep for retry backoff, which bypasses the instance-level stubs(:sleep) the retry tests already set up - so the SimpleFIN retry tests really slept through the 2s/4s/8s exponential backoff (16.8s for the max-retries test alone, 23.5s for the suite). Call sleep on the instance instead (same Kernel method, same behavior) so mocha stubs take effect. The Simplefin with_retries sleep: keyword is renamed to backoff: since it would shadow the method. Provider::SimplefinTest drops from 23.5s to ~1s. Claude-Session: https://claude.ai/code/session_01RpZe2ajeGkPRRBHfaJTfUB Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Jestin Palamuttam
|
5803023fa7 |
feat(provider): add native Questrade brokerage provider integration (#2534)
* Add native Questrade brokerage provider integration Adds a per-family Questrade provider so users can sync their Questrade investment accounts (TFSA, FHSA, RRSP, margin, etc.) directly via Questrade's free personal API, with no paid aggregator. - OAuth2 refresh-token flow with single-use token rotation, persisted under a row lock. Tokens self-renew on each sync; the connected panel lets users paste a fresh token if a connection goes stale (no need to disconnect and re-link). - Imports accounts, balances, positions and activities; multi-currency holdings with per-currency cash holdings; Norbert's Gambit journals. - New-account and link-existing-account flows, settings card with desktop-only setup steps, and connect/update/disconnect. - Restricted to Investment account types. Registered in the provider connection-status registry with a syncable scope so it participates in nightly family sync. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: linting error * fix: refresh token encrypted The OAuth token exchange passed the single-use refresh token as a GET query parameter, so it could leak into URL-based logs (Sentry breadcrumbs, APM spans, debug output). Switch to POST with a form-encoded body (RFC 6749 3.2) so the credential stays out of URLs. Verified Questrade's token endpoint accepts POST (returns 400 for a bad token, not 405). Adds a test asserting the token travels in the body. * Address PR review: authz, data integrity, retries, logging Batch of fixes from the automated PR review: - Require admin for all mutating/linking Questrade actions, and gate existing-account linking through accessible_accounts + write permission (was only Current.family scoped). - Clear requires_update when a fresh token is accepted; use a real 302 redirect (not 422) on full-page failures. - Require refresh_token on all saves (not just create) unless the item is scheduled for deletion. - Migrations target Rails 7.2; questrade_items state columns are NOT NULL. - Background activity dedup keys on Questrade fields (matches the importer) so multiple activities no longer collapse to one. - Persist the normalized account payload; date-scope synthetic cash holdings so daily history is not overwritten. - Retry 429/5xx via a RetryableResponseError instead of hard-failing. - Route provider error bodies to DebugLogEntry instead of Rails.logger / exception messages, so payloads do not leak into application logs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address PR review: atomic linking, sync health, retry loop, USD cash - Wrap account creation + provider linking (+ sync_start_date) in a transaction in both link paths so a link failure rolls back the orphan account. - Surface per-account process/schedule failures in the item sync health instead of always reporting healthy. - Always stamp last_activities_sync once the background fetch completes, so legitimately empty accounts stop being re-queued every sync. - Treat only the account-currency (CAD) balance as primary cash; other currencies (e.g. USD) now surface as separate cash holdings instead of being hidden as primary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address PR review: serialize token exchange, real processor tests - Single-use token race: the SDK now wraps every token exchange (initial and 401 re-auth) in a model-supplied lock that reloads and spends the freshest persisted token (provided.rb#synchronize_exchange). Two concurrent syncs/jobs can no longer double-spend the same refresh token. Adds a test asserting the exchange runs inside the lock with the fresh token. - Replace the all-skipped QuestradeAccount processor test stubs with real fixture-backed tests covering balance anchoring, holdings import, and Buy-trade import (plus blank-symbol / blank-type guards). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Fix indentation of spliced Questrade schema blocks The manually added questrade_accounts/questrade_items create_table blocks sat at column 0 instead of the file 2-space indent, so rubocop flagged them as inconsistent. Re-indent to match the rest of the schema. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Include currency and type in the Questrade activity merge key Two activities that differ only by currency or type could collapse to a single row in merge_activities. Add both fields to activity_key in the importer and the background fetch job so multi-currency imports dedup correctly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address review: infer account currency, Encryptable, safer flag clear From @jjmata's review: - Currency: QuestradeAccount#upsert_from_questrade! no longer hardcodes CAD for every account. upsert_balances! now infers the home currency from the per-currency balances (the currency holding the cash wins, ties broken by total equity, default CAD) so USD-denominated accounts are labelled USD and match the right combinedBalances anchor. Adds tests for USD and CAD cases. - QuestradeItem now includes the shared Encryptable concern instead of reimplementing encryption_ready? inline. - QuestradeActivitiesFetchJob#clear_pending_flag is now best-effort so it can never mask (and swallow) the original error in perform's rescue. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Fix review issues: safe_return_to_path, DebugLogEntry, financial reset, turbo_prefetch, N+1 counts - Add safe_return_to_path to QuestradeItemsController (blocks //evil.com protocol-relative open redirect; same 3-check guard as PR #2591 Wise provider) - Pass return_to through select_accounts and complete_account_setup so users land back on the account they were linking from - Replace Rails.logger.error/warn with DebugLogEntry.capture in controller and unlinking concern (surface errors in the app debug log UI) - Add questrade_items to Family::FinancialDataReset::PROVIDER_ITEM_ASSOCIATIONS so Reset Financial Data actually removes Questrade data - Add when "questrade" case to load_provider_items in providers_controller so the settings panel lazy-load refresh works - Fix turbo_prefetch: false on non-lunchflow provider links in _method_selector.html.erb and select_provider.html.erb (prevents prefetch-cache blank-modal bug for all generic sync providers) - Preload questrade_accounts: :account_provider and build @questrade_account_counts_map in AccountsController; read from map in partial instead of calling .count on associations (eliminates N+1) - Localize default connection name via I18n.t(questrade_items.default_name) - Add default_name key to questrade_items locale Patterns and bugs surfaced during review of PR #2591 (Wise provider). * Cross-apply Wise learnings to Questrade provider Encryption (matched convention from Wise/jjmata review): - Add deterministic: true to QuestradeItem#refresh_token - Add encrypts :raw_payload + :raw_institution_payload to QuestradeItem - Add Encryptable + encrypts :raw_payload, :raw_holdings_payload, :raw_activities_payload, :raw_balances_payload to QuestradeAccount (brokerage-specific columns; matches MercuryAccount/UpAccount pattern) Bug fix: - Add missing RetryableResponseError class to Provider::Questrade (used in with_retries rescue clause but never defined — would cause NameError on any rate-limited or 5xx response) Logging: - Replace Rails.logger.error with DebugLogEntry.capture in QuestradeItem#import_latest_questrade_data, #process_accounts, and #schedule_account_syncs to surface errors in the support UI Consistency: - Extract update_sync_status(sync, key, **i18n_options) helper in QuestradeItem::Syncer, replacing 5 inline sync.update! guard calls - Use blank? instead of ||= for default name fallback in create action Tests: - Add QuestradeItemsControllerTest (18 tests: CRUD, sync, account linking/setup flows, admin guard enforcement) - Add questrade fixtures: questrade_items.yml, questrade_accounts.yml - Add retry/backoff tests to Provider::QuestradeTest (network error, 429, 5xx — all verify MAX_RETRIES exhaustion raises Error) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Signed-off-by: Jestin Palamuttam <34907800+jestinjoshi@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
threatsurfer
|
ab32388326 |
feat(up): flag internal transfers and round-ups as funds_movement (#2460)
* feat(up): flag internal transfers and round-ups as funds_movement Up populates relationships.transferAccount on transactions that move money between the user's own accounts (including round-ups swept into a Saver), but flatten_transaction dropped it, so these imported as ordinary income/expense and distorted budgets and cashflow. - Provider::Up#flatten_transaction: lift transfer_account_id from relationships.transferAccount.data.id. - UpEntry::Processor: import transfers as funds_movement and persist transfer_account_id in extra["up"]. - Account::ProviderImportAdapter#import_transaction: optional kind: param; an explicit provider kind takes precedence over account-type auto-detection and is applied after the sync-protection check, so user re-categorisations survive re-sync. Complementary to Family#auto_match_transfers!: two-sided transfers between linked accounts are still paired into a Transfer (the matcher does not filter on kind); one-sided movements and round-ups, which the matcher cannot pair, are the cases this fixes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(up): account-type kind wins over provider transfer hint Codex review caught that Up HOME_LOAN accounts map to a Loan account, so a repayment carrying transferAccount would be reclassified from loan_payment to funds_movement (budget-excluded). Make the provider kind: a fallback: activity-label and account-type classification now take precedence, so loan_payment and cc_payment survive. Adds a regression test (loan repayment stays loan_payment), a depository-applies test, and a note that the up_test stub ignores query: intentionally. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Gavin Matthews <matthews.gav@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
ghost
|
d329a4f69d |
feat(kraken): import deposits, withdrawals, staking & fees via Ledgers API (#2451)
* feat(kraken): fetch and import Ledgers API for deposits, withdrawals, staking, fees Closes #2450 Kraken TradesHistory only returns spot buy/sell trades. The Ledgers API (/0/private/Ledgers) covers deposits, withdrawals, staking rewards, Earn income, and standalone fees — everything that was missing from syncs. Changes: - Provider::Kraken#get_ledgers — new method forwarding start/type/offset params - KrakenItem::Importer#fetch_ledgers — paginated fetch (up to 200 pages) with graceful fallback if the API key lacks Query Ledger Entries permission - Importer#upsert_kraken_account — stores "ledgers" alongside "trades" in raw_transactions_payload - KrakenAccount::LedgerProcessor — new class; maps each supported ledger type (deposit, withdrawal, staking, earn, fee) to a Transaction entry with the correct investment_activity_label, kind, and sign convention; skips trade/ transfer/margin types to avoid double-counting with TradesHistory - KrakenAccount::Processor#process — calls LedgerProcessor after process_trades - Multi-currency: fiat amounts converted via ExchangeRate (non-USD fiat bridged through USD); crypto amounts use the spot price cached in raw_payload["assets"] with a price_missing flag when no price is available - Dedup guard: external_id "kraken_ledger_<id>" + source "kraken" prevents re-importing on repeated syncs * fix(kraken): correct sign convention, fee inclusion, and earn subtype filtering - Sign convention: deposits/staking/earn → negative (inflow), withdrawals/fees → positive (outflow), matching Sure's global convention (inflow is negative) - Fee inclusion: use (amount - fee).abs as abs_impact so withdrawal fees are counted in the total outflow rather than discarded - Earn subtypes: skip allocation/deallocation ledger entries (internal fund movements); only import rewardallocation/bonusallocation as Interest income - DebugLogEntry: replace Rails.logger.warn/error with DebugLogEntry.capture throughout LedgerProcessor and the Ledgers permission fallback in Importer, so support-relevant incidents surface in /settings/debug - Importer test: stub get_ledgers in setup so existing tests do not error on the new fetch_ledgers call * fix(kraken): route duplicate ledger-id warning through DebugLogEntry * perf(kraken): batch ledger idempotency check; strengthen tests Address review feedback (jjmata): - N+1: LedgerProcessor#process_ledger_entry ran `account.entries.exists?(...)` per ledger entry (up to ~10k per sync). Load the existing Kraken external IDs once into a Set and test membership in memory (newly created IDs are added so the same run stays idempotent) — same pattern as #2452. - Tests: the idempotency test now asserts the first pass actually creates the entry (assert_difference) before asserting the second is a no-op; add a guard asserting the second (all-skipped) pass issues a single bulk external_id pluck, not one query per entry. No behavior change to imported entries. * perf(kraken): scope ledger idempotency pluck to kraken_ledger_ prefix Only load existing ledger external IDs (not trade entries) into the idempotency Set, matching the reviewed approach. No behavior change. |
||
|
Juan José Mata
|
b3f70c8951 |
feat:Add SnapTrade OAuth device flow (#2523)
* Add SnapTrade OAuth connection flow * Restore SnapTrade brokerage portal links * Guard SnapTrade OAuth setup completion * Move SnapTrade OAuth start to POST * Use one SnapTrade item in provider panel * Fix SnapTrade OAuth controller tests * Fix SnapTrade OAuth drawer completion redirect * Update SnapTrade limits message. * Restrict SnapTrade OAuth scopes |
||
|
Juan José Mata
|
b416618558 |
Add SnapTrade OAuth device flow (#2494)
* Add SnapTrade OAuth device flow * Add SnapTrade OAuth device flow ### Motivation - Integrate SnapTrade device-code OAuth so administrators can start a device authorization flow and poll for tokens using SnapTrade's well-known OAuth metadata endpoint. - Persist and encrypt device-flow token material on `SnaptradeItem` to support long-lived API calls and future refresh handling. ### Description - Add OAuth discovery and device-code support to the SnapTrade provider with `oauth_authorization_server_metadata`, `start_device_authorization`, and `poll_device_token` helper methods and shared `oauth_connection` / `parse_oauth_response` helpers. - Persist OAuth device-flow tokens on `snaptrade_items` via a migration that adds `oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, and `oauth_token_expires_at`, and update `db/schema.rb` accordingly. - Extend `SnaptradeItem` to encrypt stored `oauth_access_token` and `oauth_refresh_token` when ActiveRecord encryption is configured and add `oauth_token_active?` to check token validity. - Add high-level `start_oauth_device_flow` and `complete_oauth_device_flow!` helpers to the `SnaptradeItem::Provided` concern to start authorization and persist token responses. - Expose admin-only member routes and JSON controller actions `start_oauth_device_flow` and `complete_oauth_device_flow` on `SnaptradeItemsController` to drive the device flow from the UI. - Add focused Minitest coverage: `test/models/provider/snaptrade_oauth_test.rb` for provider requests and error handling and `test/models/snaptrade_item_oauth_test.rb` for token persistence. ### Testing - Ran the targeted test suite: `bin/rails test test/models/provider/snaptrade_oauth_test.rb test/models/snaptrade_item_oauth_test.rb test/controllers/snaptrade_items_controller_test.rb`, and all tests passed. - Ran `bin/rubocop` against modified files and no offenses were reported. * Add SnapTrade OAuth device flow ### Motivation - Integrate SnapTrade device-code OAuth so administrators can start a device authorization flow and poll for tokens using SnapTrade's well-known OAuth metadata endpoint. - Persist and encrypt device-flow token material on `SnaptradeItem` to support long-lived API calls and future refresh handling. ### Description - Add OAuth discovery and device-code support to the SnapTrade provider with `oauth_authorization_server_metadata`, `start_device_authorization`, and `poll_device_token` helper methods and shared `oauth_connection` / `parse_oauth_response` helpers. - Persist OAuth device-flow tokens on `snaptrade_items` via a migration that adds `oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, and `oauth_token_expires_at`, and update `db/schema.rb` accordingly. - Extend `SnaptradeItem` to encrypt stored `oauth_access_token` and `oauth_refresh_token` when ActiveRecord encryption is configured and add `oauth_token_active?` to check token validity. - Add high-level `start_oauth_device_flow` and `complete_oauth_device_flow!` helpers to the `SnaptradeItem::Provided` concern to start authorization and persist token responses. - Expose admin-only member routes and JSON controller actions `start_oauth_device_flow` and `complete_oauth_device_flow` on `SnaptradeItemsController` to drive the device flow from the UI. - Add focused Minitest coverage: `test/models/provider/snaptrade_oauth_test.rb` for provider requests and error handling and `test/models/snaptrade_item_oauth_test.rb` for token persistence. ### Testing - Ran the targeted test suite: `bin/rails test test/models/provider/snaptrade_oauth_test.rb test/models/snaptrade_item_oauth_test.rb test/controllers/snaptrade_items_controller_test.rb`, and all tests passed. - Ran `bin/rubocop` against modified files and no offenses were reported. * Add SnapTrade OAuth device flow ### Motivation - Integrate SnapTrade device-code OAuth so administrators can start a device authorization flow and poll for tokens using SnapTrade's well-known OAuth metadata endpoint. - Persist and encrypt device-flow token material on `SnaptradeItem` to support long-lived API calls and future refresh handling. ### Description - Add OAuth discovery and device-code support to the SnapTrade provider with `oauth_authorization_server_metadata`, `start_device_authorization`, and `poll_device_token` helper methods and shared `oauth_connection` / `parse_oauth_response` helpers. - Persist OAuth device-flow tokens on `snaptrade_items` via a migration that adds `oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, and `oauth_token_expires_at`, and update `db/schema.rb` accordingly. - Extend `SnaptradeItem` to encrypt stored `oauth_access_token` and `oauth_refresh_token` when ActiveRecord encryption is configured and add `oauth_token_active?` to check token validity. - Add high-level `start_oauth_device_flow` and `complete_oauth_device_flow!` helpers to the `SnaptradeItem::Provided` concern to start authorization and persist token responses. - Expose admin-only member routes and JSON controller actions `start_oauth_device_flow` and `complete_oauth_device_flow` on `SnaptradeItemsController` to drive the device flow from the UI. - Add focused Minitest coverage: `test/models/provider/snaptrade_oauth_test.rb` for provider requests and error handling and `test/models/snaptrade_item_oauth_test.rb` for token persistence. ### Testing - Ran the targeted test suite: `bin/rails test test/models/provider/snaptrade_oauth_test.rb test/models/snaptrade_item_oauth_test.rb test/controllers/snaptrade_items_controller_test.rb`, and all tests passed. - Ran `bin/rubocop` against modified files and no offenses were reported. * Add SnapTrade OAuth device flow ### Motivation - Integrate SnapTrade device-code OAuth so administrators can start a device authorization flow and poll for tokens using SnapTrade's well-known OAuth metadata endpoint. - Persist and encrypt device-flow token material on `SnaptradeItem` to support long-lived API calls and future refresh handling. ### Description - Add OAuth discovery and device-code support to the SnapTrade provider with `oauth_authorization_server_metadata`, `start_device_authorization`, and `poll_device_token` helper methods and shared `oauth_connection` / `parse_oauth_response` helpers. - Persist OAuth device-flow tokens on `snaptrade_items` via a migration that adds `oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, and `oauth_token_expires_at`, and update `db/schema.rb` accordingly. - Extend `SnaptradeItem` to encrypt stored `oauth_access_token` and `oauth_refresh_token` when ActiveRecord encryption is configured and add `oauth_token_active?` to check token validity. - Add high-level `start_oauth_device_flow` and `complete_oauth_device_flow!` helpers to the `SnaptradeItem::Provided` concern to start authorization and persist token responses. - Expose admin-only member routes and JSON controller actions `start_oauth_device_flow` and `complete_oauth_device_flow` on `SnaptradeItemsController` to drive the device flow from the UI. - Add focused Minitest coverage: `test/models/provider/snaptrade_oauth_test.rb` for provider requests and error handling and `test/models/snaptrade_item_oauth_test.rb` for token persistence. ### Testing - Ran the targeted test suite: `bin/rails test test/models/provider/snaptrade_oauth_test.rb test/models/snaptrade_item_oauth_test.rb test/controllers/snaptrade_items_controller_test.rb`, and all tests passed. - Ran `bin/rubocop` against modified files and no offenses were reported. * Add SnapTrade OAuth device flow (provider, model, controller, routes, migration, tests) ### Motivation - Add support for SnapTrade OAuth 2.0 device authorization flow so users can authorize SnapTrade via device codes in addition to the existing portal flow. - Persist OAuth token metadata on `SnaptradeItem` for subsequent polling and usage by the provider and UI. ### Description - Implemented OAuth device flow in the provider with `oauth_authorization_server_metadata`, `start_device_authorization`, and `poll_device_token`, plus HTTP helper methods `oauth_connection`, `oauth_client_id`, and `parse_oauth_response` in `Provider::Snaptrade`. - Added controller endpoints `start_oauth_device_flow` and `complete_oauth_device_flow` in `SnaptradeItemsController` with robust error handling and helpers `oauth_error_payload` and `parse_oauth_error_body` to surface OAuth error fields. - Extended `SnaptradeItem` with encrypted columns for `oauth_access_token`, `oauth_refresh_token`, token metadata, `oauth_token_active?`, and model methods `start_oauth_device_flow` and `complete_oauth_device_flow!` to store token metadata. - Added migration `AddOauthDeviceFlowToSnaptradeItems` and updated `db/schema.rb` to include the new columns, registered a new initializer `config/initializers/snaptrade.rb` to read `SNAPTRADE_OAUTH_CLIENT_ID`, and updated `.env*.example` files to document `SNAPTRADE_OAUTH_CLIENT_ID`. - Exposed new routes `start_oauth_device_flow` and `complete_oauth_device_flow` for `snaptrade_items`. ### Testing - Added unit tests for provider OAuth behavior in `test/models/provider/snaptrade_oauth_test.rb`, model token persistence in `test/models/snaptrade_item_oauth_test.rb`, and controller error propagation in `test/controllers/snaptrade_items_controller_test.rb`. - Ran the test suite with `bin/rails test` and the full test run (including the new SnapTrade OAuth tests) passed. * Add SnapTrade OAuth device flow (start/poll), store tokens, and tests ### Motivation - Add support for SnapTrade OAuth Device Authorization flow so administrators can start device authorization and poll for tokens without exposing provider internals. - Persist OAuth token metadata on `SnaptradeItem` so the app can reuse and surface token state for SnapTrade integrations. - Improve error handling and sanitization for OAuth API errors returned by SnapTrade to avoid leaking upstream internals. ### Description - Introduces new environment examples and initializer: adds `SNAPTRADE_OAUTH_CLIENT_ID` to `.env.local.example`/`.env.test.example` and configures `Rails.configuration.x.snaptrade.oauth_client_id` in `config/initializers/snaptrade.rb`. - Extends `Provider::Snaptrade` with OAuth device flow support, adding discovery URL, device grant constant, Faraday `oauth_connection`, and methods `oauth_authorization_server_metadata`, `start_device_authorization`, `poll_device_token`, `oauth_client_id`, and `parse_oauth_response`, plus improved retry and API error wrapping. - Adds DB migration and schema changes to store OAuth fields on `snaptrade_items` (`oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, `oauth_token_expires_at`) and marks those attributes encrypted when ActiveRecord encryption is enabled. - Adds `SnaptradeItem` helpers `start_oauth_device_flow`, `complete_oauth_device_flow!`, and `oauth_token_active?` to start/poll and persist token metadata. - Adds controller actions `start_oauth_device_flow` and `complete_oauth_device_flow` with sanitized error responses and helper methods `oauth_error_payload` and `parse_oauth_error_body`, and wires new member routes in `config/routes.rb`. - Updates `SnaptradeItemsController` before_action lists to include the new actions and adds user-facing error message helper `start_oauth_device_flow_error_message`. ### Testing - Added unit tests `test/models/provider/snaptrade_oauth_test.rb` to validate discovery, device authorization request, token polling, missing client ID handling, and OAuth error propagation, and all assertions passed. - Added `test/models/snaptrade_item_oauth_test.rb` to assert `complete_oauth_device_flow!` stores tokens and expiry and that `oauth_token_active?` reports correctly, and the test passed. - Extended `test/controllers/snaptrade_items_controller_test.rb` with controller-level tests confirming sanitized OAuth error payloads and behavior for start/complete endpoints, and these controller tests passed. * Add SnapTrade OAuth device flow support and token storage ### Motivation - Add support for the OAuth 2.0 device authorization flow for SnapTrade so users can link brokerages via a device-code flow without traditional browser-based client redirects. - Persist OAuth token metadata on the SnaptradeItem so tokens can be reused and expiration tracked. - Surface and sanitize provider error payloads to callers while avoiding leakage of internal configuration details. ### Description - Added new DB columns and a migration (`oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, `oauth_token_expires_at`) and updated `db/schema.rb` to reflect the change. - Encrypted the new token fields on `SnaptradeItem` and added `oauth_token_active?`, `start_oauth_device_flow`, and `complete_oauth_device_flow!` helpers to the model. - Implemented OAuth logic in `Provider::Snaptrade` including discovery (`OAUTH_DISCOVERY_URL`), `start_device_authorization`, `poll_device_token`, request parsing, retry handling, and a small Faraday connection wrapper; added configuration accessors for `oauth_client_id` via `config.x.snaptrade`. - Added controller endpoints `start_oauth_device_flow` and `complete_oauth_device_flow` with safe error handling and helpers to format OAuth error payloads, and registered new member routes for `snaptrade_items`. - Added an initializer to load `SNAPTRADE_OAUTH_CLIENT_ID` into `Rails.configuration.x.snaptrade`, and updated `.env.local.example` and `.env.test.example` to include the new env var. - Added unit tests for provider OAuth behavior (`test/models/provider/snaptrade_oauth_test.rb`), SnaptradeItem token persistence (`test/models/snaptrade_item_oauth_test.rb`), and controller error handling (`test/controllers/snaptrade_items_controller_test.rb`), and updated controller tests accordingly. ### Testing - Ran provider-level tests in `Provider::SnaptradeOauthTest` to validate discovery, device authorization, token polling, and error handling, which passed. - Ran model tests in `SnaptradeItemOauthTest` to verify token metadata is stored and `oauth_token_active?` works, which passed. - Ran controller tests in `SnaptradeItemsControllerTest` that exercise the start/complete endpoints and error sanitization, which passed. * Add SnapTrade OAuth device-flow support and token storage ### Motivation - Add support for SnapTrade OAuth device authorization so users can perform device-code based OAuth without embedding secrets in the browser. - Persist OAuth token metadata on `SnaptradeItem` and expose programmatic start/complete endpoints while avoiding leaking provider internals on errors. - Make OAuth client ID configurable via environment or credentials for Sure deployments. ### Description - Introduce `SNAPTRADE_OAUTH_CLIENT_ID` to `.env` examples and add `config.x.snaptrade.oauth_client_id` initializer for configuration. - Add migration `AddOauthDeviceFlowToSnaptradeItems` and update `schema.rb` to add `oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, and `oauth_token_expires_at` to `snaptrade_items`. - Persist and encrypt new token fields in `SnaptradeItem` and add helper methods `oauth_token_active?`, `start_oauth_device_flow`, and `complete_oauth_device_flow!`. - Extend `Provider::Snaptrade` with OAuth discovery, device authorization (`start_device_authorization`), token polling (`poll_device_token`), Faraday-based `oauth_connection`, parsing and error-wrapping logic, and retry handling. - Add controller actions `start_oauth_device_flow` and `complete_oauth_device_flow` to `SnaptradeItemsController`, plus helper methods to format OAuth error payloads and user-facing error messages. - Wire up new routes (`post :start_oauth_device_flow`, `post :complete_oauth_device_flow`) and add `config/initializers/snaptrade.rb`. - Add comprehensive tests for the OAuth flow and controller error handling and update a system test stub to avoid provider leakage during UI tests. ### Testing - Added `Provider::SnaptradeOauthTest` which stubs discovery, device authorization and token endpoints and asserts request payloads and responses; tests passed. - Added `SnaptradeItemOauthTest` which verifies token metadata persistence and `oauth_token_active?`; test passed. - Extended `SnaptradeItemsControllerTest` with scenarios for successful and failing device-flow completion and start flow error handling; tests passed. - Ran the affected system test changes in `TradesTest` (provider stubbing and modal behavior); the updated tests passed. |
||
|
Frank Li
|
b88deb3545 |
fix: use ES256 instead of EdDSA for Coinbase CDP JWT signing (#1888)
* fix: use ES256 instead of EdDSA for Coinbase CDP JWT signing Coinbase Developer Platform issues EC P-256 keys, not Ed25519. The previous EdDSA implementation would always return Unauthorized for any key generated via portal.cdp.coinbase.com. Switch to ES256 (ECDSA SHA-256) using OpenSSL, and convert the DER-encoded signature to the raw r||s format required by the JWT spec. API Secret field now accepts the full PEM private key directly (including BEGIN/END headers) as provided by Coinbase. * fix(coinbase): address PR review comments - Normalize escaped \n in PEM key before parsing (fixes pasting directly from the Coinbase CDP JSON download file where newlines are \n literals) - Extract parse_ec_private_key helper with docstring explaining both accepted PEM formats - Fix double-space style nit on encoded_header assignment - Remove ed25519 gem from Gemfile (no longer used after ES256 migration) - Add Provider::CoinbaseTest covering: JWT 3-part structure, ES256 alg header, kid claim, CDP payload claims, 64-byte raw r||s signature, escaped-newline key acceptance, and cryptographic signature verification * test(coinbase): fix base64url padding in JWT test assertions * chore: update Gemfile.lock to remove ed25519 gem * Sample credential in test * Comments field, not real key use * test(coinbase): generate EC key dynamically; drop scanner exclusions Generate a fresh P-256 key per test run with OpenSSL::PKey::EC.generate instead of hardcoding a PEM private key. This removes committed key material and the need to exclude the provider and test files from the pipelock secret scan, so those exclusions are removed too. Addresses review findings on hardcoded test key + scan exclusions. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Juan José Mata <jjmata@jjmata.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Artem Danilov
|
c65f4828dd |
fix(tinkoff): resolve the tradeable listing and price bonds via BondBy (#2413)
* fix(tinkoff): resolve the tradeable listing and price bonds via BondBy Validated against the live T-Invest API and fixed three correctness issues in Provider::TinkoffInvest: - FindInstrument returns several listings per ticker (e.g. SBER on TQBR plus dark/non-API boards 37M/SPEQ); only the apiTradeAvailableFlag listing has live prices. resolve_short now ranks tradeable-first (then requested-MIC, then exact ticker/ISIN) instead of taking the first match, so GetLastPrices/GetCandles return data. Search now surfaces only tradeable instruments. - find_instruments no longer hard-filters on apiTradeAvailableFlag, so a qualified-investor instrument Tinkoff lists but can't API-trade still resolves (for logos) and otherwise falls back to another price provider. - Bond nominal comes from BondBy (the generic GetInstrumentBy omits it), returning the current amortized nominal so percent-of-par converts correctly. Verified end-to-end: SBER 313.65, T 282.76, LQDT 2.019, and the SFO Split bond (amortized nominal 417.71) all price; logos resolve to real CDN PNGs. * fix(tinkoff): honor requested MIC first, guard amortizing bonds, strip suffix Address review feedback on the resolution/pricing path: - resolve_short now ranks a requested-MIC match BEFORE tradeability, so a security is never priced off another exchange's listing (e.g. a MISX security no longer picks up a tradeable XSPX board's price); tradeability and exact ticker/ISIN remain secondary tie-breaks. - Amortizing bonds: BondBy returns only the current nominal, so applying it to historical percent-of-par closes would underprice them. For amortizing bonds we now return only the live price and skip the candle history; fixed-par bonds and equities keep full history. bond_info exposes nominal + amortizationFlag. - Strip exchange suffixes (.ME/.MOEX/.MISX/.MCX) before querying T-Invest, which only knows the bare SECID — so a stored "T.MOEX" ticker resolves to "T". * fix(tinkoff): don't cache nil resolution results (skip_nil) so a transient empty response isn't locked in for the 24h TTL |
||
|
Jake
|
dc2a565b6a |
feat(up): add Up Bank (AU) provider integration (#2391)
* feat(up): add Up Bank (AU) provider integration Adds Up Bank as a per-family, token-based bank sync provider, modelled on the existing Akahu integration. Up uses a JSON:API REST API with a personal access token (Bearer), cursor pagination via links.next, and returns both HELD (pending) and SETTLED transactions from one endpoint. New: - Provider::Up client (JSON:API unwrap, links.next pagination, retries, typed errors, /util/ping) + Provider::UpAdapter (Factory-registered, Depository + Loan). - UpItem / UpAccount models with Provided, Unlinking, Syncer, SyncCompleteEvent, Importer, Processor, Transactions::Processor, and UpEntry::Processor (amount sign flip, HELD->pending, foreignAmount FX, merchant from description, stale-pending pruning). - Family::UpConnectable, UpItemsController, routes, settings panel + connect flow views, accounts index wiring, initializer, en locale, and model tests. Core wiring: - "up" added to Transaction::PENDING_PROVIDERS, the three pending-match SQL blocks in Account::ProviderImportAdapter, Provider::Metadata::REGISTRY, ProviderMerchant/DataEnrichment source enums, ProviderConnectionStatus, settings provider panels, and financial data reset. Migration create_up_items_and_accounts must be run before use. No external API endpoints added (no OpenAPI changes). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(up): dump up tables to schema and make since filter TZ-safe The feature commit added the up_items/up_accounts migration but never re-dumped db/schema.rb, leaving the schema version and tables stale. Add the two table definitions and foreign keys and bump the schema version so a fresh DB load matches the migration. Also format a bare Date `since` as UTC midnight instead of the server's local zone, so `filter[since]` is deterministic regardless of where the app runs (previously shifted by the local UTC offset). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(up): address code review feedback Behavior/correctness: - Persist skipped accounts via a new up_accounts.ignored flag and a needs_setup scope, so skipped accounts stop resurfacing as "needs setup" on every sync. Linking clears the flag. - destroy now checks unlink_all! per-account results and aborts deletion (alert) if any unlink failed, instead of swallowing failures. - render_provider_panel_error redirect uses :see_other (was an invalid 4xx redirect status). - Up provider adapter falls back to item institution name/url when institution_metadata is absent (early return previously blocked it). Resilience/security: - fetch_all_resources guards against an API repeating the same links.next cursor (Set#add?), preventing infinite pagination. - HTTP client validates absolute URLs (from links.next) against Up's HTTPS host before sending the bearer token, preventing credential leakage to untrusted hosts. Diagnostics: - Route provider sync/import failures through DebugLogEntry.capture (controller, UpItem, syncer, unlinking) with family/account context. Low-level HTTP client and currency-normalization warnings keep Rails.logger to match existing provider conventions. Data integrity: - up_accounts.name and currency are NOT NULL (align with model presence validations); account_id stays nullable (allow_nil uniqueness). Forms: - select_existing_account radio is required; controller guards a blank/ unknown up_account_id with a friendly alert instead of RecordNotFound. Tests: - Add UpAccount needs_setup scope test, pagination loop guard test, untrusted-host rejection test; tighten filter[since] assertion to the exact UTC timestamp. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(up): address second-round review feedback - Capture sync/import failures via DebugLogEntry so swallowed errors in account/transaction fetching and transaction processing surface in /settings/debug instead of only Rails.logger. - Gate UP_DEBUG_RAW raw payload dump to local envs to avoid leaking PII (merchant names, amounts, account IDs) in managed/production logs. - Collapse linked/unlinked/total account counts into one memoized query instead of 3 separate COUNTs per rendered item. - Rename "Set Up Up Accounts" locale title to "Link Up Accounts". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(up): add method docstrings and align failed_result keys Add docstrings to all Up provider source files (controller, models, providers, concerns) to satisfy the 80% docstring coverage threshold. Third-round review: failed_result now mirrors import's result shape (accounts_updated/created/failed, transactions_imported/failed) instead of the stale accounts_imported key, so failure results stay consistent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Artem Danilov
|
9de9a23ce2 |
feat(prices): add T-Invest (T-Bank) securities + brand-logo provider (#2408)
Adds Provider::TinkoffInvest, a token-based securities provider built on the public T-Invest REST gateway (invest-public-api.tinkoff.ru/rest). It serves prices for Russian instruments (shares, ETF/БПИФ, bonds) and, crucially, brand logos via the T-Invest CDN — the authoritative logo source for MOEX instruments, which ISS (MoexPublic) does not provide. - Registry: register `tinkoff_invest` under the :securities concept; token via ENV TINKOFF_INVEST_API_KEY or encrypted Setting.tinkoff_invest_api_key. - Logos independent of the price provider: Security#import_brand_logo consults T-Invest for a logo whenever a token is configured (after the price-provider metadata fetch, so it never short-circuits website_url backfill). Gated on token presence, not the securities checklist. - display_logo_url: with no website domain, a stored provider logo (T-Invest) now beats the ticker-only Brandfetch lettermark; when a domain exists, Brandfetch still wins (unchanged). - MoexPublic no longer reports moex.com as the issuer website — it's the exchange, not the issuer, and would make Brandfetch render the exchange logo for every instrument and shadow the real brand logo. - Prices: GetCandles (daily, paged) + GetLastPrices; Quotation units+nano/1e9; bonds priced as percent-of-par x nominal (missing nominal raises, not 0). - Settings: encrypted token field (always shown) + provider checkbox + en locale. - Tests for search/info/logo-url/prices/bond/incomplete-candle and display logic. Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Artem Danilov
|
b57bb938da |
feat(prices): add Moscow Exchange (MOEX ISS) securities + FX provider (#2394)
* feat(prices): add Moscow Exchange (MOEX ISS) securities + FX provider Add Provider::MoexPublic, a keyless provider built on the free MOEX ISS API (https://iss.moex.com/iss), modeled on Provider::BinancePublic. Securities: shares, funds/ETF/БПИФ (e.g. LQDT), and bonds (OFZ + corporate). Bonds are priced clean — LAST% × FACEVALUE / 100 in the instrument currency, with per-row FACEVALUE for amortizing issues; NKD/accrued coupon excluded. Exchange rates: also implements ExchangeRateConcept for RUB↔{USD,EUR,CNY} via selt TOM instruments (USD000UTSTOM/EUR_RUB__TOM/CNYRUB_TOM); the selt quote is X/RUB, inverted for RUB→X, nil for non-RUB-crossed pairs. Details: - Board/engine resolution via the ISS primary-board flag with a hardcoded priority fallback (TQBR, TQTF, TQOB, TQCB, …). - Instrument currency from CURRENCYID/FACEUNIT (handles USD/CNY eurobonds & FX funds), normalizing legacy SUR/RUR → RUB; default RUB. - Full history via from/till + start= pagination; current price fallback chain LAST → MARKETPRICE → LCURRENTPRICE → LCLOSEPRICE → PREVPRICE → latest history close. - Bare SECID identity, exchange_operating_mic=MISX, country_code=nil (wildcard like Binance); search accepts .ME/.MOEX/.MISX/.MCX aliases and ISIN. - RateLimitable throttling, SslConfigurable, Faraday retry/timeouts; all public methods wrapped in with_provider_response. Wired into Provider::Registry for both :securities and :exchange_rates, the hosting provider-selection UI, locales, and config/exchanges.yml (MISX). Docker-tested (devcontainer, Ruby 3.4.9): 29 new tests green, full provider suite + i18n green, rubocop clean; smoke-tested against live ISS (SBER price, OFZ clean price, USD/RUB FX). * fix(moex): address review — FX weekend lookback, dead branch, translated hints - fetch_exchange_rate now fetches a 10-day lookback window (not just the exact day) so a weekend/holiday request resolves to the prior trading day's close, matching Yahoo's behavior (Codex P2). - Remove dead identical if/else branches in history_row_price (CodeRabbit). - Translate moex_public_hint into ca/fr/hu/vi/zh-CN instead of English copy (CodeRabbit). - Add a test covering the FX prior-trading-day lookback. * fix(moex): guard ISS date parsing; doc TQTE in board priority Address maintainer review (jjmata): - parse_iss_date wraps Date.parse so a malformed ISS TRADEDATE skips just that row (with a contextual log warning) instead of failing the whole history/FX fetch. Used in history_row_price and fx_history. - Add TQTE to the BOARD_PRIORITY doc comment (it was in the constant but missing from the comment). - Add a test covering the unparseable-date skip. |
||
|
Artem Danilov
|
fffcaf2345 |
fix(prices): resolve dashed crypto tickers (BTC-USD, TRX-USD) via BinancePublic (#2372)
* fix(prices): resolve dashed crypto tickers (BTC-USD, TRX-USD) via BinancePublic
Provider::BinancePublic is the dedicated crypto price provider, but its
search_securities only matched bare base assets ("BTC") or unseparated
Binance pairs ("BTCUSDT"/"BTCUSD"). The canonical "<BASE>-USD" form that
Yahoo emits and users paste (e.g. "TRX-USD", "USDT-USD") never matched, so
those holdings fell through to an unpriced offline security whenever the
stock provider didn't also return the coin.
Collapse the base/quote separator in the search query (and strip it in
parse_ticker) so "BTC-USD" / "TRX/USDT" are treated like "BTCUSD" and
resolve to live Binance pricing. Stablecoins pasted as "USDT-USD" resolve
to the synthetic USD price via their stripped base.
* fix(prices): only synthesize stablecoin search for USD quote
Address review: gate the synthetic stablecoin result on a USD quote (bare
coin or "<coin>USD" form). A non-USD quote like "USDTEUR" / "USDT-EUR"
has a real Binance pair and now falls through to normal matching instead of
being replaced by the USD synthetic. Drops the now-unused base_asset_query
helper. Adds a USDT-EUR regression test.
|
||
|
Kelvinchen03
|
59edcaa986 | fix: EODHD lookup for EU mutual funds with EUFUND exchange code (#2212) | ||
|
Guillem Arias Fauste
|
f25fe30a41 |
feat(ai): honor Setting.llm_provider for batch and PDF flows (#2265)
Auto-categorization, merchant detection/enhancement, and PDF/bank-statement extraction hard-coded Provider::Registry.get_provider(:openai), so selecting Anthropic (or running an Anthropic-only self-hosted install) left those operations using/missing OpenAI rather than the chosen provider. Add Provider::Registry.preferred_llm_provider, which resolves the LLM provider honoring Setting.llm_provider with a configured-provider fallback (mirroring how chat picks its provider), and route all six TODO(#2113) call sites through it: - Family::AutoCategorizer#llm_provider - Family::AutoMerchantDetector#llm_provider - ProviderMerchant::Enhancer#llm_provider - PdfImport (process_pdf + extract_bank_statement) - Assistant::Function::ImportBankStatement Provider::Anthropic already implements auto_categorize / auto_detect_merchants / enhance_provider_merchants (#1984) and process_pdf / extract_bank_statement (#1985), so no provider changes are needed — only the wiring. Closes #2113. |
||
|
Dan
|
3627c4d2d1 | Respect CoinStats wallet rate limits (#2251) | ||
|
Tobias Rahloff
|
fe47c918bb |
feat(enable_banking): support MFA/decoupled banks and harden session handling (#2174)
Decoupled/MFA banks (e.g. VR Bank in Holstein) were hard-blocked because the authorize flow aborted whenever auth_methods[0] was DECOUPLED. Enable Banking's hosted /auth page actually coordinates decoupled SCA and redirects back with a code, so route these banks through it instead: - Provider#start_authorization accepts and forwards an auth_method param - EnableBankingItem#select_auth_method picks the best method (REDIRECT > DECOUPLED > EMBEDDED), filtering by psu_type and skipping hidden methods - Shared begin_authorization! re-fetches ASPSP metadata on each authorize and reauthorize, so the method is always re-derived (no persistence required) - Remove the DECOUPLED block in the controller Also stop the integration from constantly reporting "session expired": - Only a session-level GET /sessions 401/404 flips the connection to requires_update; per-account 401/404 are retried and no longer kill the whole connection - Reconcile session_expires_at from the API's access.valid_until on every sync - Treat an expired session as a graceful requires_update state instead of raising a bare error No schema changes. Adds covering tests. |
||
|
Guillem Arias Fauste
|
8ccc434b3d |
feat(ai): Anthropic native PDF processing (3/5) (#1985)
* feat(ai): add Anthropic provider with chat parity (1/5)
Introduces Provider::Anthropic alongside Provider::Openai, implementing
the LlmConcept chat_response contract over the official anthropic Ruby
SDK. Batch ops, PDF, and RAG land in follow-up PRs.
- Provider::Anthropic uses Messages API for sync and streaming responses
- ChatConfig builds requests with ephemeral prompt-cache markers on the
system prompt and the last tool definition
- MessageFormatter reconstructs multi-turn history (text + tool_use +
tool_result blocks) from raw Message records, including the paired
user-role tool_result turn Anthropic requires after every tool_use
- ChatParser maps Anthropic Message into the shared ChatResponse Data
- Registry, Setting, User, Chat default model wired for ANTHROPIC_*
envs and Setting.anthropic_*; LLM_PROVIDER selects between providers
- Responder forwards raw conversation_history (Array<Message>) so
providers without hosted conversation state can rebuild context
- OpenAI provider accepts and ignores the new kwarg (no behavior change)
Tests cover provider init, model gating, MessageFormatter for all turn
shapes, ChatConfig request building (max_tokens, system cache, tool
conversion), ChatParser for text / tool_use / mixed blocks, Registry
discovery, and mocked chat_response success / error / function_request
paths. Live VCR cassettes recorded in a follow-up with a real key.
Stacked PRs: 2/5 batch ops + cost ledger, 3/5 PDF, 4/5 pgvector RAG,
5/5 settings UI + disclosure.
* fix(ai): address PR review on Anthropic provider foundation
Surface fixes raised by Codex + CodeRabbit on PR 1/5:
- Provider::Anthropic#chat_response now accepts (and ignores) a
`messages:` kwarg. Assistant::Responder passes both `messages:`
(OpenAI-shape) and `conversation_history:` (raw Message records) for
cross-provider parity, so the previous signature raised
ArgumentError on the first chat turn through the Anthropic provider.
- Provider::Anthropic#supports_model? bypasses the `claude` prefix
gate when a custom base_url is configured, mirroring the OpenAI
provider. Bedrock-shaped IDs like
`anthropic.claude-sonnet-4-5-20250929-v1:0` and
`claude-opus-4@20250514` are otherwise rejected by
Assistant::Provided#get_model_provider and the chat dies.
- Setting.anthropic_access_token is now in
EncryptedSettingFields::ENCRYPTED_FIELDS so the Anthropic API key
is encrypted at rest like every other provider secret. Previously
plaintext while siblings (openai_access_token, twelve_data_api_key,
external_assistant_token) were ciphertext.
- Chat.default_model falls back to whichever provider is actually
configured. Previously, with LLM_PROVIDER=anthropic but no
Anthropic credentials, the default model resolved to a Claude ID
that no registered provider supported, so chats failed even when
OpenAI was fully configured. Adds Provider::{Anthropic,Openai}#configured?
class methods for the readable callsite.
- Provider::Anthropic.effective_model uses
`ENV["ANTHROPIC_MODEL"].presence || Setting.anthropic_model` so the
Setting lookup is only performed when the env var is absent — the
previous `ENV.fetch(KEY, default)` evaluated the default arg
eagerly on every call.
- Provider::Anthropic::ChatConfig#anthropic_input_schema strips both
`:strict` and `"strict"` keys so JSON-decoded schemas with string
keys cannot leak the OpenAI-only flag through to Anthropic.
Test coverage added: supports_model? bypass on custom endpoints,
chat_response messages: kwarg compatibility, default_model fallback
in the three credential combinations, configured? against ENV +
Setting, strict-flag stripping for both key types, and a
`Setting.expects(:anthropic_model).never` assertion proving the
ENV-precedence test now exercises the lazy path.
All 4365 tests pass (1 pre-existing libvips env error unrelated).
* test(chat): make default_model tests resilient to ENV model overrides
CodeRabbit flagged on PR review: the new default_model tests asserted
against Provider::*::DEFAULT_MODEL, but Chat.default_model actually
returns Provider::*.effective_model.presence (which reads
OPENAI_MODEL / ANTHROPIC_MODEL from the environment). With either env
var set, the tests would fail intermittently even though routing was
correct.
- New default_model tests now assert against the provider's
effective_model directly, so they verify the routing decision
(which provider's value wins) without coupling to the constant.
- Pre-existing "creates with default model" assertions had the same
brittleness; switch them to compare against Chat.default_model so
the chosen model is whatever the env / Setting cascade resolves to.
Verified by running `ANTHROPIC_MODEL=claude-haiku-4-5 OPENAI_MODEL=gpt-4o
bin/rails test test/models/chat_test.rb` — 16 runs, 0 failures
(previously 2 pre-existing failures + 0 from the new tests).
* fix(ai): address local review on Anthropic foundation
- Provider::Anthropic#supports_pdf_processing? bypasses prefix gate for
custom endpoints, mirroring supports_model?
- Provider::Anthropic#initialize raises Error when custom_endpoint? AND
model.blank?, parity with Provider::Openai
- stream_chat_response captures partial usage on mid-stream errors and
records it via the new on_partial callback so chat_response can skip
the duplicate error row in the outer rescue
- safe_accumulated_message swallows the secondary failure when the SDK
cannot reconstruct a snapshot
- langfuse_client memoizes properly (||= instead of =) so repeated calls
don't churn Langfuse instances
- MessageFormatter sorts tool_calls by created_at then id so the
message array is deterministic across replays; skips tool_calls
missing both provider_call_id and provider_id rather than sending
`id: nil` and getting rejected by Anthropic
- Setting.anthropic_access_token default falls back through
ENV["ANTHROPIC_API_KEY"].presence (was missing .presence, so an
empty-string env value bled through)
- User#openai_configured? / #anthropic_configured? delegate to the
Provider::* class methods — single source of truth
- Assistant::Responder renames the OpenAI-shape history builder
conversation_history → openai_messages_payload so the kwarg name
matches the local method name (messages: openai_messages_payload,
conversation_history: chat_message_records)
- Assistant::Builtin stale-history comment updated to reference both
builders
Adds a streaming chat_response test using ad-hoc subclasses of the
SDK event types so the case/when dispatch matches via is_a? without
stubbing class-level === behavior.
* test(ai): add Anthropic tool_use round-trip + multi-tool turn coverage
Addresses @jjmata's "worth confirming" note on PR #1983: tool-use turns
from prior assistant messages must round-trip correctly when retrieved
from the database.
- New `ChatParser → ToolCall::Function → MessageFormatter` test walks
the full path: Anthropic response with a tool_use block →
ChatFunctionRequest → ToolCall::Function.from_function_request →
persisted on the AssistantMessage → MessageFormatter rebuild on the
next turn. Asserts the original `tool_use.id` is preserved end-to-end
as both `tool_use.id` and the paired `tool_result.tool_use_id`, and
that the original `input` hash and serialized result content survive.
- New multi-tool assistant turn test confirms two tool_use blocks on a
single assistant message render as two tool_use blocks followed by
two paired tool_result blocks in a single user-role follow-up,
matching Anthropic's required alternation.
Both tests exercise the existing PR1 code without behavior changes.
* test(ai): require "ostruct" explicitly in Anthropic provider tests
OpenStruct is moving out of Ruby's default load path (warning in 3.4+,
removed in 3.5+). Tests work today because ActiveSupport transitively
loads it, but that's incidental. Match the existing convention in
test/controllers/settings/hostings_controller_test.rb which explicitly
requires ostruct for the same reason.
* fix(ai): sanitize Langfuse warn logs, normalize tool_use.input, dedup history fetch
Addresses three open CodeRabbit findings on PR #1983.
- Provider::Anthropic Langfuse rescue branches no longer include
`e.full_message` in `Rails.logger.warn`. `full_message` bundles the
backtrace + cause chain and on some SDK error types includes the
serialized request/response payload (prompt, model output). Logs
now report `#{e.class}: #{e.message}` only. Three sites:
create_langfuse_trace, log_langfuse_generation, upsert_langfuse_trace.
Note: Provider::Openai has the same pattern (copy-pasted source) —
harmonization deferred to a follow-up cleanup PR; this commit fixes
only the Anthropic provider to keep PR scope tight.
- MessageFormatter#parse_arguments now coerces any non-Hash parsed
result to `{}`. Anthropic's Messages API requires `tool_use.input`
to be a JSON object (map); a stored ToolCall::Function record whose
arguments parse to a scalar, bool, or array (corrupt row, legacy
data, cross-provider bleed) would otherwise produce a payload the
API rejects. Normal flow stores Hash arguments end-to-end so the
fix is defensive — adds 2 tests covering scalar/array JSON strings
and non-String non-Hash inputs.
- Assistant::Responder dedups the chat-history fetch. The previous
layout fired two near-identical `chat.messages.where(...).includes(
:tool_calls).ordered` queries per LLM turn (one for the OpenAI-shape
payload, one for the raw-records kwarg). A new memoized
`complete_chat_messages` fetches once; `chat_message_records` filters
out the current message via `Array#reject`, `openai_messages_payload`
iterates the cached array unchanged. One SQL query per turn instead
of two. Memoization scope = single Responder instance (per LLM call),
so cache invalidation is not a concern.
All 4370 tests pass (1 pre-existing libvips env error unrelated).
Rubocop + brakeman clean.
* fix(ci): replace sk-ant- prefixed test placeholders
Pipelock secret scanner pattern-matches `sk-ant-*` as a real Anthropic
API key and fails the PR security-scan check. Test stubs and
ClimateControl env values used `sk-ant-test`, `sk-ant-from-setting`,
`sk-ant-x`, `sk-ant-y` as obvious placeholders, but the scanner does
not care about value entropy.
Switched to `fake-anthropic-key-*` / `fake-token-*` strings so the
scanner stops flagging them. No production code touched, no behavior
change — Provider::Anthropic still accepts any non-blank token.
* feat(ai): add Anthropic batch ops + LLM cost ledger (2/5)
Implements auto_categorize, auto_detect_merchants, and
enhance_provider_merchants on Provider::Anthropic via forced tool calls,
plus the cost-ledger plumbing they need.
- Provider::Anthropic::AutoCategorizer, AutoMerchantDetector,
ProviderMerchantEnhancer each define a single output tool whose
input_schema mirrors the desired output, then force the model to call
it via tool_choice: { type: "tool", name: ..., disable_parallel_tool_use: true }.
Anthropic guarantees the tool_use.input matches the schema, so there
is no JSON parsing fragility, no <think> tag stripping, and no
json_object/json_schema fallback ladders.
- Concerns::UsageRecorder mirrors the OpenAI sibling but persists
cache_creation_input_tokens / cache_read_input_tokens to dedicated
columns instead of metadata.
- Migration adds cache_creation_tokens, cache_read_tokens (nullable
integers) to llm_usages. OpenAI rows leave them null.
- LlmUsage::PRICING gains Claude 4.x rows (opus-4-7 $15/$75, sonnet-4-6
$3/$15, haiku-4-5 $1/$5 per MTok). infer_provider returns "anthropic"
for claude-* via the existing exact/prefix lookup.
- Provider::Anthropic#chat_response now persists cache columns directly
rather than stashing them in metadata.
- 25-transaction batch cap mirrors the OpenAI provider so the cost
ledger sees the same shape regardless of which provider ran a batch.
Tests cover the forced-tool-call path, null/None normalization,
case-insensitive merchant matching, the missing-tool_use error path,
and Anthropic-specific pricing + provider inference on LlmUsage.
Stacked on #1983 (PR 1/5). 3/5 PDF + vision next.
* fix(ai): attribute Bedrock model IDs to anthropic + clean nil enum
- LlmUsage.infer_provider now returns "anthropic" for Bedrock /
Vertex shaped IDs (anthropic.* and anthropic/*), so cost-ledger
filtering by provider stays correct even when no per-MTok rate is
stored. Previously these IDs fell through to the "openai" default.
- AutoCategorizer drops the redundant nil sentinel from the
category_name enum — the union type [string, null] already permits
null, and some JSON Schema validators reject nil literals inside
enum arrays.
* test(ai): require "ostruct" in Anthropic batch op tests
Same rationale as the PR1 ostruct fix — explicit require so the tests
don't depend on ActiveSupport's transitive load when Ruby 3.5+ removes
OpenStruct from the default load path.
* feat(ai): Anthropic native PDF processing (3/5)
Implements process_pdf and extract_bank_statement on Provider::Anthropic
using the native `document` content block — no rasterization, no text
pre-extraction.
- Provider::Anthropic::PdfProcessor classifies the document, summarizes
it, and extracts statement metadata via a forced report_document_analysis
tool whose input_schema mirrors the existing Provider::Openai output
(document_type from Import::DOCUMENT_TYPES, summary, extracted_data).
- Provider::Anthropic::BankStatementExtractor returns the same
{ transactions, period, account_holder, account_number, bank_name,
opening_balance, closing_balance } shape via report_bank_statement so
downstream pdf_import code is provider-agnostic.
- Both attach the PDF as
{ type: "document", source: { type: "base64", media_type: "application/pdf", data: <b64> } }
— Claude 3.5+ / 4.x accept this natively (up to 32MB / 100 pages).
No pdf-reader, no pdftoppm, no chunking for typical statements.
- supports_pdf_processing? (introduced in PR 1) already returns true for
claude-* models, gating process_pdf with a clear error otherwise.
- Cost ledger rows are persisted via the shared UsageRecorder concern,
including cache_creation/cache_read tokens.
Tests verify the document block shape, tool_choice forcing, normalized
document_type for unknown classifications, transaction normalization
(date / amount / reference → notes), and the missing-tool_use error
path. Blank pdf_content raises before any client call.
Stacked on #1984 (PR 2/5). 4/5 pgvector RAG next.
* fix(ai): guard PDF size + surface bank-statement truncation
- PdfProcessor and BankStatementExtractor raise upfront when
pdf_content.bytesize exceeds MAX_PDF_BYTES (32 MB, matching
Anthropic's hard limit). Previously a 100 MB PDF would be
base64-encoded (~133 MB) and packed into the JSON body before
the API rejected it — peak heap ~270 MB per Sidekiq worker.
- BankStatementExtractor inspects response.stop_reason; when the
model hit max_tokens it logs a warning and flags result[:truncated]
so downstream callers know the transaction list may be incomplete.
- ISO date pattern added to statement_period_start/end schema in
PdfProcessor so the model can't return "March 2026" — Anthropic
enforces the regex via the tool's input_schema.
Tests cover the size guard (raises before any client.messages call),
truncated-result flagging, and the warning log path.
* test(ai): require "ostruct" in Anthropic PDF tests
Match the explicit ostruct require added in PR1/PR2 — same Ruby 3.5+
load-path reason.
* fix(llm-usage): include Anthropic cache tokens in estimated_cost
calculate_cost only priced prompt + completion tokens, so estimated_cost
under-reported every cached call — the cache_creation/cache_read columns this PR
added were tracked but never billed. Verified against the Anthropic dashboard: a
cached chat turn billed $0.05 but the ledger recorded $0.038; the gap was exactly
the unpriced cache tokens.
Price them relative to the input rate (Anthropic: cache write 1.25x, read 0.1x)
and thread the cache counts from both recorders (chat + batch). OpenAI rows leave
the columns null (treated as 0), so they're unaffected. Ledger now reproduces the
dashboard ($0.054 for the test turn).
* chore(ai): guard chat usage double-record; flag deferred Anthropic batch wiring
- Hardening: guard the success-path record_llm_usage with
`unless partial_usage_recorded` so a future change that emits partial usage on
a normal stream can't silently double-bill (the symptom investigated in the
#1984 review). No behavior change today — on_partial only fires from the
mid-stream-error rescue, which re-raises past this line.
- Notice: the family auto-categorize / merchant-detect / merchant-enhance flows
still hardcode get_provider(:openai). Provider::Anthropic now implements those
batch ops but they aren't wired into the family flows yet — documented with
TODOs at each site for the follow-up.
* chore(ai): point family-flow TODOs at tracking issue #2113
* chore(ai): flag deferred Anthropic PDF wiring (TODO #2113)
The PDF import + bank-statement-extract flows hardcode get_provider(:openai).
Provider::Anthropic implements process_pdf / extract_bank_statement (this PR)
but they aren't wired into these paths yet — documented with TODOs at each
site. Tracked in #2113 (broadened to cover batch ops + PDF).
* chore(anthropic-pdf): drop redundant strip_heredoc; document no-dedup
- The squiggly heredoc (<<~) already strips indentation, so the trailing
.strip_heredoc was a no-op in both PDF extractors.
- Document why BankStatementExtractor intentionally does NOT deduplicate (unlike
the OpenAI extractor): we send the whole PDF as one native document block, so
there are no overlapping-chunk artifacts to dedupe, and deduping would wrongly
merge legitimate same-day, same-amount transactions.
* fix(anthropic): cap PDF bytes below the base64-encoded request limit
Anthropic's 32 MB limit is on the Messages request body, and the PDF is sent
base64-encoded (~4/3 larger) alongside the JSON envelope, so a 32 MB raw PDF
encodes to ~42 MB and is rejected. Cap the raw bytes at 3/4 of the request
budget minus a 1 MB envelope reserve (~23 MiB). Addresses Codex review on #1985.
|
||
|
Brad
|
1b8b21760b |
feat(provider): Akahu integration (#1921)
* First pass of Akahu * fix up sync all * conflicts * fix db migration issue? - fix auto selection of akahu account type * Address Akahu PR feedback * Complete provider metadata * Fix PR 1921 CI tests * PR feedback * PR feedback * post merge --------- Co-authored-by: failing <failing@users.noreply.github.com> Co-authored-by: Juan José Mata <jjmata@jjmata.com> Co-authored-by: sure-admin <sure-admin@splashblot.com> |
||
|
Guillem Arias Fauste
|
991cb959c1 |
feat(ai): Anthropic batch ops + LLM cost ledger (2/5) (#1984)
* feat(ai): add Anthropic provider with chat parity (1/5)
Introduces Provider::Anthropic alongside Provider::Openai, implementing
the LlmConcept chat_response contract over the official anthropic Ruby
SDK. Batch ops, PDF, and RAG land in follow-up PRs.
- Provider::Anthropic uses Messages API for sync and streaming responses
- ChatConfig builds requests with ephemeral prompt-cache markers on the
system prompt and the last tool definition
- MessageFormatter reconstructs multi-turn history (text + tool_use +
tool_result blocks) from raw Message records, including the paired
user-role tool_result turn Anthropic requires after every tool_use
- ChatParser maps Anthropic Message into the shared ChatResponse Data
- Registry, Setting, User, Chat default model wired for ANTHROPIC_*
envs and Setting.anthropic_*; LLM_PROVIDER selects between providers
- Responder forwards raw conversation_history (Array<Message>) so
providers without hosted conversation state can rebuild context
- OpenAI provider accepts and ignores the new kwarg (no behavior change)
Tests cover provider init, model gating, MessageFormatter for all turn
shapes, ChatConfig request building (max_tokens, system cache, tool
conversion), ChatParser for text / tool_use / mixed blocks, Registry
discovery, and mocked chat_response success / error / function_request
paths. Live VCR cassettes recorded in a follow-up with a real key.
Stacked PRs: 2/5 batch ops + cost ledger, 3/5 PDF, 4/5 pgvector RAG,
5/5 settings UI + disclosure.
* fix(ai): address PR review on Anthropic provider foundation
Surface fixes raised by Codex + CodeRabbit on PR 1/5:
- Provider::Anthropic#chat_response now accepts (and ignores) a
`messages:` kwarg. Assistant::Responder passes both `messages:`
(OpenAI-shape) and `conversation_history:` (raw Message records) for
cross-provider parity, so the previous signature raised
ArgumentError on the first chat turn through the Anthropic provider.
- Provider::Anthropic#supports_model? bypasses the `claude` prefix
gate when a custom base_url is configured, mirroring the OpenAI
provider. Bedrock-shaped IDs like
`anthropic.claude-sonnet-4-5-20250929-v1:0` and
`claude-opus-4@20250514` are otherwise rejected by
Assistant::Provided#get_model_provider and the chat dies.
- Setting.anthropic_access_token is now in
EncryptedSettingFields::ENCRYPTED_FIELDS so the Anthropic API key
is encrypted at rest like every other provider secret. Previously
plaintext while siblings (openai_access_token, twelve_data_api_key,
external_assistant_token) were ciphertext.
- Chat.default_model falls back to whichever provider is actually
configured. Previously, with LLM_PROVIDER=anthropic but no
Anthropic credentials, the default model resolved to a Claude ID
that no registered provider supported, so chats failed even when
OpenAI was fully configured. Adds Provider::{Anthropic,Openai}#configured?
class methods for the readable callsite.
- Provider::Anthropic.effective_model uses
`ENV["ANTHROPIC_MODEL"].presence || Setting.anthropic_model` so the
Setting lookup is only performed when the env var is absent — the
previous `ENV.fetch(KEY, default)` evaluated the default arg
eagerly on every call.
- Provider::Anthropic::ChatConfig#anthropic_input_schema strips both
`:strict` and `"strict"` keys so JSON-decoded schemas with string
keys cannot leak the OpenAI-only flag through to Anthropic.
Test coverage added: supports_model? bypass on custom endpoints,
chat_response messages: kwarg compatibility, default_model fallback
in the three credential combinations, configured? against ENV +
Setting, strict-flag stripping for both key types, and a
`Setting.expects(:anthropic_model).never` assertion proving the
ENV-precedence test now exercises the lazy path.
All 4365 tests pass (1 pre-existing libvips env error unrelated).
* test(chat): make default_model tests resilient to ENV model overrides
CodeRabbit flagged on PR review: the new default_model tests asserted
against Provider::*::DEFAULT_MODEL, but Chat.default_model actually
returns Provider::*.effective_model.presence (which reads
OPENAI_MODEL / ANTHROPIC_MODEL from the environment). With either env
var set, the tests would fail intermittently even though routing was
correct.
- New default_model tests now assert against the provider's
effective_model directly, so they verify the routing decision
(which provider's value wins) without coupling to the constant.
- Pre-existing "creates with default model" assertions had the same
brittleness; switch them to compare against Chat.default_model so
the chosen model is whatever the env / Setting cascade resolves to.
Verified by running `ANTHROPIC_MODEL=claude-haiku-4-5 OPENAI_MODEL=gpt-4o
bin/rails test test/models/chat_test.rb` — 16 runs, 0 failures
(previously 2 pre-existing failures + 0 from the new tests).
* fix(ai): address local review on Anthropic foundation
- Provider::Anthropic#supports_pdf_processing? bypasses prefix gate for
custom endpoints, mirroring supports_model?
- Provider::Anthropic#initialize raises Error when custom_endpoint? AND
model.blank?, parity with Provider::Openai
- stream_chat_response captures partial usage on mid-stream errors and
records it via the new on_partial callback so chat_response can skip
the duplicate error row in the outer rescue
- safe_accumulated_message swallows the secondary failure when the SDK
cannot reconstruct a snapshot
- langfuse_client memoizes properly (||= instead of =) so repeated calls
don't churn Langfuse instances
- MessageFormatter sorts tool_calls by created_at then id so the
message array is deterministic across replays; skips tool_calls
missing both provider_call_id and provider_id rather than sending
`id: nil` and getting rejected by Anthropic
- Setting.anthropic_access_token default falls back through
ENV["ANTHROPIC_API_KEY"].presence (was missing .presence, so an
empty-string env value bled through)
- User#openai_configured? / #anthropic_configured? delegate to the
Provider::* class methods — single source of truth
- Assistant::Responder renames the OpenAI-shape history builder
conversation_history → openai_messages_payload so the kwarg name
matches the local method name (messages: openai_messages_payload,
conversation_history: chat_message_records)
- Assistant::Builtin stale-history comment updated to reference both
builders
Adds a streaming chat_response test using ad-hoc subclasses of the
SDK event types so the case/when dispatch matches via is_a? without
stubbing class-level === behavior.
* test(ai): add Anthropic tool_use round-trip + multi-tool turn coverage
Addresses @jjmata's "worth confirming" note on PR #1983: tool-use turns
from prior assistant messages must round-trip correctly when retrieved
from the database.
- New `ChatParser → ToolCall::Function → MessageFormatter` test walks
the full path: Anthropic response with a tool_use block →
ChatFunctionRequest → ToolCall::Function.from_function_request →
persisted on the AssistantMessage → MessageFormatter rebuild on the
next turn. Asserts the original `tool_use.id` is preserved end-to-end
as both `tool_use.id` and the paired `tool_result.tool_use_id`, and
that the original `input` hash and serialized result content survive.
- New multi-tool assistant turn test confirms two tool_use blocks on a
single assistant message render as two tool_use blocks followed by
two paired tool_result blocks in a single user-role follow-up,
matching Anthropic's required alternation.
Both tests exercise the existing PR1 code without behavior changes.
* test(ai): require "ostruct" explicitly in Anthropic provider tests
OpenStruct is moving out of Ruby's default load path (warning in 3.4+,
removed in 3.5+). Tests work today because ActiveSupport transitively
loads it, but that's incidental. Match the existing convention in
test/controllers/settings/hostings_controller_test.rb which explicitly
requires ostruct for the same reason.
* fix(ai): sanitize Langfuse warn logs, normalize tool_use.input, dedup history fetch
Addresses three open CodeRabbit findings on PR #1983.
- Provider::Anthropic Langfuse rescue branches no longer include
`e.full_message` in `Rails.logger.warn`. `full_message` bundles the
backtrace + cause chain and on some SDK error types includes the
serialized request/response payload (prompt, model output). Logs
now report `#{e.class}: #{e.message}` only. Three sites:
create_langfuse_trace, log_langfuse_generation, upsert_langfuse_trace.
Note: Provider::Openai has the same pattern (copy-pasted source) —
harmonization deferred to a follow-up cleanup PR; this commit fixes
only the Anthropic provider to keep PR scope tight.
- MessageFormatter#parse_arguments now coerces any non-Hash parsed
result to `{}`. Anthropic's Messages API requires `tool_use.input`
to be a JSON object (map); a stored ToolCall::Function record whose
arguments parse to a scalar, bool, or array (corrupt row, legacy
data, cross-provider bleed) would otherwise produce a payload the
API rejects. Normal flow stores Hash arguments end-to-end so the
fix is defensive — adds 2 tests covering scalar/array JSON strings
and non-String non-Hash inputs.
- Assistant::Responder dedups the chat-history fetch. The previous
layout fired two near-identical `chat.messages.where(...).includes(
:tool_calls).ordered` queries per LLM turn (one for the OpenAI-shape
payload, one for the raw-records kwarg). A new memoized
`complete_chat_messages` fetches once; `chat_message_records` filters
out the current message via `Array#reject`, `openai_messages_payload`
iterates the cached array unchanged. One SQL query per turn instead
of two. Memoization scope = single Responder instance (per LLM call),
so cache invalidation is not a concern.
All 4370 tests pass (1 pre-existing libvips env error unrelated).
Rubocop + brakeman clean.
* fix(ci): replace sk-ant- prefixed test placeholders
Pipelock secret scanner pattern-matches `sk-ant-*` as a real Anthropic
API key and fails the PR security-scan check. Test stubs and
ClimateControl env values used `sk-ant-test`, `sk-ant-from-setting`,
`sk-ant-x`, `sk-ant-y` as obvious placeholders, but the scanner does
not care about value entropy.
Switched to `fake-anthropic-key-*` / `fake-token-*` strings so the
scanner stops flagging them. No production code touched, no behavior
change — Provider::Anthropic still accepts any non-blank token.
* feat(ai): add Anthropic batch ops + LLM cost ledger (2/5)
Implements auto_categorize, auto_detect_merchants, and
enhance_provider_merchants on Provider::Anthropic via forced tool calls,
plus the cost-ledger plumbing they need.
- Provider::Anthropic::AutoCategorizer, AutoMerchantDetector,
ProviderMerchantEnhancer each define a single output tool whose
input_schema mirrors the desired output, then force the model to call
it via tool_choice: { type: "tool", name: ..., disable_parallel_tool_use: true }.
Anthropic guarantees the tool_use.input matches the schema, so there
is no JSON parsing fragility, no <think> tag stripping, and no
json_object/json_schema fallback ladders.
- Concerns::UsageRecorder mirrors the OpenAI sibling but persists
cache_creation_input_tokens / cache_read_input_tokens to dedicated
columns instead of metadata.
- Migration adds cache_creation_tokens, cache_read_tokens (nullable
integers) to llm_usages. OpenAI rows leave them null.
- LlmUsage::PRICING gains Claude 4.x rows (opus-4-7 $15/$75, sonnet-4-6
$3/$15, haiku-4-5 $1/$5 per MTok). infer_provider returns "anthropic"
for claude-* via the existing exact/prefix lookup.
- Provider::Anthropic#chat_response now persists cache columns directly
rather than stashing them in metadata.
- 25-transaction batch cap mirrors the OpenAI provider so the cost
ledger sees the same shape regardless of which provider ran a batch.
Tests cover the forced-tool-call path, null/None normalization,
case-insensitive merchant matching, the missing-tool_use error path,
and Anthropic-specific pricing + provider inference on LlmUsage.
Stacked on #1983 (PR 1/5). 3/5 PDF + vision next.
* fix(ai): attribute Bedrock model IDs to anthropic + clean nil enum
- LlmUsage.infer_provider now returns "anthropic" for Bedrock /
Vertex shaped IDs (anthropic.* and anthropic/*), so cost-ledger
filtering by provider stays correct even when no per-MTok rate is
stored. Previously these IDs fell through to the "openai" default.
- AutoCategorizer drops the redundant nil sentinel from the
category_name enum — the union type [string, null] already permits
null, and some JSON Schema validators reject nil literals inside
enum arrays.
* test(ai): require "ostruct" in Anthropic batch op tests
Same rationale as the PR1 ostruct fix — explicit require so the tests
don't depend on ActiveSupport's transitive load when Ruby 3.5+ removes
OpenStruct from the default load path.
* fix(llm-usage): include Anthropic cache tokens in estimated_cost
calculate_cost only priced prompt + completion tokens, so estimated_cost
under-reported every cached call — the cache_creation/cache_read columns this PR
added were tracked but never billed. Verified against the Anthropic dashboard: a
cached chat turn billed $0.05 but the ledger recorded $0.038; the gap was exactly
the unpriced cache tokens.
Price them relative to the input rate (Anthropic: cache write 1.25x, read 0.1x)
and thread the cache counts from both recorders (chat + batch). OpenAI rows leave
the columns null (treated as 0), so they're unaffected. Ledger now reproduces the
dashboard ($0.054 for the test turn).
* chore(ai): guard chat usage double-record; flag deferred Anthropic batch wiring
- Hardening: guard the success-path record_llm_usage with
`unless partial_usage_recorded` so a future change that emits partial usage on
a normal stream can't silently double-bill (the symptom investigated in the
#1984 review). No behavior change today — on_partial only fires from the
mid-stream-error rescue, which re-raises past this line.
- Notice: the family auto-categorize / merchant-detect / merchant-enhance flows
still hardcode get_provider(:openai). Provider::Anthropic now implements those
batch ops but they aren't wired into the family flows yet — documented with
TODOs at each site for the follow-up.
* chore(ai): point family-flow TODOs at tracking issue #2113
* fix(ai): address review findings on cost ledger + categorizer schema
Three AI-review findings on #1984:
- category_name enum omitted null (codex + coderabbit): the prompt + type allow
Claude to abstain on uncertain transactions, but JSON Schema `enum` restricted
the value to category names, so null was invalid — forcing miscategorization.
Append nil to the enum (the consumer already normalizes null -> uncategorized).
- Cache pricing applied to all providers (coderabbit): the 1.25x/0.1x cache
multipliers are Anthropic-specific. Gate them on provider == "anthropic" so a
non-Anthropic caller passing cache counts isn't billed with the wrong rates.
- Negative cache-token counts (coderabbit): add DB check constraints
(cache_*_tokens IS NULL OR >= 0), per the repo's DB-level-validation convention.
Tests: enum includes nil; non-Anthropic cache tokens aren't priced.
|
||
|
Guillem Arias Fauste
|
8251b7e4d6 |
feat(ai): add Anthropic provider with chat parity (1/5) (#1983)
* feat(ai): add Anthropic provider with chat parity (1/5)
Introduces Provider::Anthropic alongside Provider::Openai, implementing
the LlmConcept chat_response contract over the official anthropic Ruby
SDK. Batch ops, PDF, and RAG land in follow-up PRs.
- Provider::Anthropic uses Messages API for sync and streaming responses
- ChatConfig builds requests with ephemeral prompt-cache markers on the
system prompt and the last tool definition
- MessageFormatter reconstructs multi-turn history (text + tool_use +
tool_result blocks) from raw Message records, including the paired
user-role tool_result turn Anthropic requires after every tool_use
- ChatParser maps Anthropic Message into the shared ChatResponse Data
- Registry, Setting, User, Chat default model wired for ANTHROPIC_*
envs and Setting.anthropic_*; LLM_PROVIDER selects between providers
- Responder forwards raw conversation_history (Array<Message>) so
providers without hosted conversation state can rebuild context
- OpenAI provider accepts and ignores the new kwarg (no behavior change)
Tests cover provider init, model gating, MessageFormatter for all turn
shapes, ChatConfig request building (max_tokens, system cache, tool
conversion), ChatParser for text / tool_use / mixed blocks, Registry
discovery, and mocked chat_response success / error / function_request
paths. Live VCR cassettes recorded in a follow-up with a real key.
Stacked PRs: 2/5 batch ops + cost ledger, 3/5 PDF, 4/5 pgvector RAG,
5/5 settings UI + disclosure.
* fix(ai): address PR review on Anthropic provider foundation
Surface fixes raised by Codex + CodeRabbit on PR 1/5:
- Provider::Anthropic#chat_response now accepts (and ignores) a
`messages:` kwarg. Assistant::Responder passes both `messages:`
(OpenAI-shape) and `conversation_history:` (raw Message records) for
cross-provider parity, so the previous signature raised
ArgumentError on the first chat turn through the Anthropic provider.
- Provider::Anthropic#supports_model? bypasses the `claude` prefix
gate when a custom base_url is configured, mirroring the OpenAI
provider. Bedrock-shaped IDs like
`anthropic.claude-sonnet-4-5-20250929-v1:0` and
`claude-opus-4@20250514` are otherwise rejected by
Assistant::Provided#get_model_provider and the chat dies.
- Setting.anthropic_access_token is now in
EncryptedSettingFields::ENCRYPTED_FIELDS so the Anthropic API key
is encrypted at rest like every other provider secret. Previously
plaintext while siblings (openai_access_token, twelve_data_api_key,
external_assistant_token) were ciphertext.
- Chat.default_model falls back to whichever provider is actually
configured. Previously, with LLM_PROVIDER=anthropic but no
Anthropic credentials, the default model resolved to a Claude ID
that no registered provider supported, so chats failed even when
OpenAI was fully configured. Adds Provider::{Anthropic,Openai}#configured?
class methods for the readable callsite.
- Provider::Anthropic.effective_model uses
`ENV["ANTHROPIC_MODEL"].presence || Setting.anthropic_model` so the
Setting lookup is only performed when the env var is absent — the
previous `ENV.fetch(KEY, default)` evaluated the default arg
eagerly on every call.
- Provider::Anthropic::ChatConfig#anthropic_input_schema strips both
`:strict` and `"strict"` keys so JSON-decoded schemas with string
keys cannot leak the OpenAI-only flag through to Anthropic.
Test coverage added: supports_model? bypass on custom endpoints,
chat_response messages: kwarg compatibility, default_model fallback
in the three credential combinations, configured? against ENV +
Setting, strict-flag stripping for both key types, and a
`Setting.expects(:anthropic_model).never` assertion proving the
ENV-precedence test now exercises the lazy path.
All 4365 tests pass (1 pre-existing libvips env error unrelated).
* test(chat): make default_model tests resilient to ENV model overrides
CodeRabbit flagged on PR review: the new default_model tests asserted
against Provider::*::DEFAULT_MODEL, but Chat.default_model actually
returns Provider::*.effective_model.presence (which reads
OPENAI_MODEL / ANTHROPIC_MODEL from the environment). With either env
var set, the tests would fail intermittently even though routing was
correct.
- New default_model tests now assert against the provider's
effective_model directly, so they verify the routing decision
(which provider's value wins) without coupling to the constant.
- Pre-existing "creates with default model" assertions had the same
brittleness; switch them to compare against Chat.default_model so
the chosen model is whatever the env / Setting cascade resolves to.
Verified by running `ANTHROPIC_MODEL=claude-haiku-4-5 OPENAI_MODEL=gpt-4o
bin/rails test test/models/chat_test.rb` — 16 runs, 0 failures
(previously 2 pre-existing failures + 0 from the new tests).
* fix(ai): address local review on Anthropic foundation
- Provider::Anthropic#supports_pdf_processing? bypasses prefix gate for
custom endpoints, mirroring supports_model?
- Provider::Anthropic#initialize raises Error when custom_endpoint? AND
model.blank?, parity with Provider::Openai
- stream_chat_response captures partial usage on mid-stream errors and
records it via the new on_partial callback so chat_response can skip
the duplicate error row in the outer rescue
- safe_accumulated_message swallows the secondary failure when the SDK
cannot reconstruct a snapshot
- langfuse_client memoizes properly (||= instead of =) so repeated calls
don't churn Langfuse instances
- MessageFormatter sorts tool_calls by created_at then id so the
message array is deterministic across replays; skips tool_calls
missing both provider_call_id and provider_id rather than sending
`id: nil` and getting rejected by Anthropic
- Setting.anthropic_access_token default falls back through
ENV["ANTHROPIC_API_KEY"].presence (was missing .presence, so an
empty-string env value bled through)
- User#openai_configured? / #anthropic_configured? delegate to the
Provider::* class methods — single source of truth
- Assistant::Responder renames the OpenAI-shape history builder
conversation_history → openai_messages_payload so the kwarg name
matches the local method name (messages: openai_messages_payload,
conversation_history: chat_message_records)
- Assistant::Builtin stale-history comment updated to reference both
builders
Adds a streaming chat_response test using ad-hoc subclasses of the
SDK event types so the case/when dispatch matches via is_a? without
stubbing class-level === behavior.
* test(ai): add Anthropic tool_use round-trip + multi-tool turn coverage
Addresses @jjmata's "worth confirming" note on PR #1983: tool-use turns
from prior assistant messages must round-trip correctly when retrieved
from the database.
- New `ChatParser → ToolCall::Function → MessageFormatter` test walks
the full path: Anthropic response with a tool_use block →
ChatFunctionRequest → ToolCall::Function.from_function_request →
persisted on the AssistantMessage → MessageFormatter rebuild on the
next turn. Asserts the original `tool_use.id` is preserved end-to-end
as both `tool_use.id` and the paired `tool_result.tool_use_id`, and
that the original `input` hash and serialized result content survive.
- New multi-tool assistant turn test confirms two tool_use blocks on a
single assistant message render as two tool_use blocks followed by
two paired tool_result blocks in a single user-role follow-up,
matching Anthropic's required alternation.
Both tests exercise the existing PR1 code without behavior changes.
* test(ai): require "ostruct" explicitly in Anthropic provider tests
OpenStruct is moving out of Ruby's default load path (warning in 3.4+,
removed in 3.5+). Tests work today because ActiveSupport transitively
loads it, but that's incidental. Match the existing convention in
test/controllers/settings/hostings_controller_test.rb which explicitly
requires ostruct for the same reason.
* fix(ai): sanitize Langfuse warn logs, normalize tool_use.input, dedup history fetch
Addresses three open CodeRabbit findings on PR #1983.
- Provider::Anthropic Langfuse rescue branches no longer include
`e.full_message` in `Rails.logger.warn`. `full_message` bundles the
backtrace + cause chain and on some SDK error types includes the
serialized request/response payload (prompt, model output). Logs
now report `#{e.class}: #{e.message}` only. Three sites:
create_langfuse_trace, log_langfuse_generation, upsert_langfuse_trace.
Note: Provider::Openai has the same pattern (copy-pasted source) —
harmonization deferred to a follow-up cleanup PR; this commit fixes
only the Anthropic provider to keep PR scope tight.
- MessageFormatter#parse_arguments now coerces any non-Hash parsed
result to `{}`. Anthropic's Messages API requires `tool_use.input`
to be a JSON object (map); a stored ToolCall::Function record whose
arguments parse to a scalar, bool, or array (corrupt row, legacy
data, cross-provider bleed) would otherwise produce a payload the
API rejects. Normal flow stores Hash arguments end-to-end so the
fix is defensive — adds 2 tests covering scalar/array JSON strings
and non-String non-Hash inputs.
- Assistant::Responder dedups the chat-history fetch. The previous
layout fired two near-identical `chat.messages.where(...).includes(
:tool_calls).ordered` queries per LLM turn (one for the OpenAI-shape
payload, one for the raw-records kwarg). A new memoized
`complete_chat_messages` fetches once; `chat_message_records` filters
out the current message via `Array#reject`, `openai_messages_payload`
iterates the cached array unchanged. One SQL query per turn instead
of two. Memoization scope = single Responder instance (per LLM call),
so cache invalidation is not a concern.
All 4370 tests pass (1 pre-existing libvips env error unrelated).
Rubocop + brakeman clean.
* fix(ci): replace sk-ant- prefixed test placeholders
Pipelock secret scanner pattern-matches `sk-ant-*` as a real Anthropic
API key and fails the PR security-scan check. Test stubs and
ClimateControl env values used `sk-ant-test`, `sk-ant-from-setting`,
`sk-ant-x`, `sk-ant-y` as obvious placeholders, but the scanner does
not care about value entropy.
Switched to `fake-anthropic-key-*` / `fake-token-*` strings so the
scanner stops flagging them. No production code touched, no behavior
change — Provider::Anthropic still accepts any non-blank token.
|
||
|
William Wei Ming
|
f653fec96f | Add HTTP timeout config to Provider::Github Octokit client (#2061) | ||
|
Brian Richard
|
174f7e6be6 |
feat(binance): add full account sync and transaction processing (#1822)
* feat(binance): add full account sync and transaction processing - Fixed a bug that hindered Account setup - Wire up Binance accounts, sync statistics, and unlinked account tracking in the accounts dashboard. - Support setting a sync_start_date during Binance account setup. - Set Binance accounts' opening balance to zero to ensure the ledger builds cleanly from the actual trade history. - Expand the Binance importer and processor to handle Spot, Margin, Earn, P2P, and Futures trades and assets. - Implement TransactionBuilder to parse raw Binance trades, accurately calculating fees, base/quote asset amounts, and market values for proper ledger integration. - Update Binance API timeout (`recvWindow`) to 60,000ms to prevent connection drops. These changes provide comprehensive support for tracking Binance portfolios, ensuring accurate historical ledgers and proper visibility of sync statuses in the frontend dashboard. * refactor(binance): enforce strong params, double-entry safety, and native fiat currency support - Implement strong parameters in BinanceItemsController#complete_account_setup to satisfy Rails security guidelines. - Add robust date parsing with a grace fallback to prevent controller crashes on malformed sync start dates. - Wrap P2P transaction creations inside a database transaction block to guarantee ledger integrity and prevent orphan records. - Optimize P2P deduplication queries by batching checks for both transaction and funding external IDs. - Shift P2P entry persistence from forced USD tracking to native fiat values extracted directly from the Binance API payload. - Update BinanceAccount::ProcessorTest assertions and fixtures to validate native fiat and fee calculation logic. * fix(binance): process sync trades before caching transaction payload - Reorder Binance processor execution to insert trade records into the database prior to updating the `raw_transactions_payload` cache. This guarantees that if a database insertion fails, the cache won't prematurely mark the sync as successful, ensuring the data is retried on the next run. - Move `set_opening_anchor_balance(balance: 0)` out of the generic crypto exchange account builder and apply it specifically during Binance account creation. - Refactor date parsing in BinanceItemsController to explicitly catch `ArgumentError` via a block instead of using a blanket inline `rescue`. - Clean up the `setup_accounts` view template by removing hardcoded default translation strings. * fix(binance): enhance trade sync logic and error propagation - Pass `startTime` (from `sync_start_date`) to spot and futures trade endpoints on initial sync to optimize data fetching. - Include previously synced futures pairs alongside spot pairs when resolving relevant symbols to properly recover sold-out assets. - Re-raise exceptions in processor rescue blocks to prevent silent failures and ensure errors are correctly propagated to background jobs. - Decrease Binance API `recvWindow` from 60000ms to 5000ms to align with recommended default timeout values. |
||
|
Sure Admin (bot)
|
2df10ca4ef |
Retry Enable Banking sync with provider-corrected date range (#1801)
* Clamp Enable Banking sync window * Pipelock noise --------- Co-authored-by: KiloClaw <kiloclaw@openclaw.ai> Co-authored-by: Juan José Mata <jjmata@jjmata.com> |
||
|
ghost
|
95f6451b39 |
feat(sync): add Brex provider connections (#1752)
* feat(sync): add Brex provider schema Adds Brex item and account tables with per-family credentials, scoped upstream account uniqueness, encrypted token storage, and sanitized provider payload columns. * feat(sync): add Brex provider core Adds Brex item/account models, provider client and adapter support, family connection helpers, and provider enum registration for read-only Brex cash and card data. * feat(sync): add Brex import pipeline Adds Brex account discovery, linked-account sync, cash/card balance processors, transaction import, sanitized metadata handling, and idempotent provider entry processing. * feat(sync): add Brex connection flows Adds Mercury-style Brex connection management, explicit item-scoped account selection and linking, settings provider UI, account index visibility, localized copy, and per-item cache handling. * test(sync): cover Brex provider workflows Adds targeted coverage for Brex provider requests, adapter config, item/account guards, importer behavior, entry processing, and Mercury-style controller flows. * fix(sync): align Brex API edge cases Tightens Brex account fetching against the official card-account response shape, sends transaction start filters as RFC3339 date-times, and keeps provider error bodies out of user-facing messages while expanding provider client guard coverage. * fix(sync): harden Brex provider integration Restrict Brex API base URLs to official hosts, tighten account-selection UI behavior, and add tests for invalid credentials, cache scoping, and provider setup edge cases. * test(sync): avoid Brex secret-shaped fixtures * refactor(sync): extract Brex account flows * fix(sync): address Brex provider review feedback * fix(sync): address Brex review follow-ups Move remaining Brex review cleanup into focused model behavior, tighten link/setup edge cases, localize summaries, and add regression coverage from CodeRabbit feedback. Also records the security-review pass as no-findings after diff-scoped inspection and Brakeman validation. * refactor(sync): split Brex account flow controllers Route Brex account selection and setup actions through small namespaced controllers while keeping existing URLs and helpers stable. Business flow remains in BrexItem::AccountFlow; the main Brex item controller now only handles connection CRUD, provider-panel rendering, destroy, and sync. * fix(sync): address Brex CodeRabbit review * fix(sync): address Brex follow-up review * fix(sync): address Brex review follow-ups * fix(sync): address Brex sync review findings * fix(sync): polish Brex review copy and errors * fix(sync): register Brex provider health * fix(sync): polish Brex bank sync presentation * fix(sync): address Brex review follow-ups * fix(sync): tighten Brex setup params * test(api): stabilize usage rate-limit window * fix(sync): polish Brex setup flow nits * fix(sync): harden Brex setup params * fix(sync): finalize Brex review cleanup --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
Gian-Reto Tarnutzer
|
ce5d7dd736 |
Add Interactive Brokers Provider (#1722)
* Display multi-currency holdings correctly * Implement IBKR provider * Fix: Use historical exchange rate for historical prices * Add brokerage exchange rate for trades * Sync historical balances from IBKR * Add logos in activity history * Fix privacy mode blur in account view * Improve IBKR XML Flex report parser errors |
||
|
plind
|
12d799e0b8 |
fix(binance): support CRYPTO: prefix and USD stablecoins (#1771)
* fix(binance): support CRYPTO: prefix and USD stablecoins Holdings processors (CoinStats, Coinbase, Kraken, SimpleFIN, Lunchflow, Binance) store crypto securities with a "CRYPTO:" prefix, but Provider::BinancePublic#parse_ticker only accepted Binance-search-style tickers like "BTCUSD". As a result, every fetched price for tickers like CRYPTO:USDT, CRYPTO:USDC, CRYPTO:SOL, CRYPTO:TRUMP, CRYPTO:KAITO failed with "Unsupported Binance ticker". - Strip the CRYPTO: prefix in parse_ticker. - Short-circuit USD-pegged stablecoins (USDT, USDC, BUSD, DAI, FDUSD, TUSD, USDP, PYUSD) to a synthetic flat 1.0 USD price. Binance has no self-pair (USDTUSDT is invalid), and the few stablecoin/USDT pairs that do exist hover at ~1.0 with sub-cent noise. - Default prefixed bare base assets (CRYPTO:SOL etc.) to the …USDT pair (USD). Only when prefixed, so unprefixed garbage like BTCBNB / BTCGBP still returns nil and the existing rejection tests still pass. - fetch_security_info returns links: nil for stablecoins rather than a broken /trade/ URL. Closes #1441. * fix(binance): strip CRYPTO: prefix in search_securities Security::Resolver calls search_provider with the raw holdings-processor symbol (CRYPTO:SOL, CRYPTO:USDT) before any price fetch. Without prefix handling here, first-time crypto imports never resolve to an online Binance security and the new stablecoin/prefix paths in parse_ticker were unreachable for that flow. - Strip CRYPTO: from the search query. - Short-circuit USD stablecoins to a synthetic search result (no exchangeInfo call, no Binance self-pair to find). - Teach parse_ticker the "{stablecoin}USD" form produced by the synthetic result so price fetches route to stablecoin_prices. --------- Co-authored-by: plind-junior <plind-junior@users.noreply.github.com> |
||
|
ghost
|
be598aecf0 |
feat(providers): add Kraken exchange sync (#1759)
* feat(providers): add Kraken exchange sync Adds family-scoped Kraken API-key connections, read-only balance and trade import, account setup/linking flows, provider status wiring, and focused test coverage. Closes #1758 * test(providers): avoid Kraken sample secret false positive * fix(providers): address Kraken review findings * fix(providers): address Kraken review cleanup * test(imports): stabilize transaction import ordering |
||
|
Juan José Mata
|
f6f9feba8a |
Bank Sync cleanup (#1710)
* feat(settings/providers): surface connection status in section headers
Lifts the per-panel status indicator up to each collapsed accordion
header so admins can see at a glance which providers are connected
without expanding every section. Connected providers sort first.
- Add optional status: and meta: locals to settings/_section partial;
pill hides via group-open:hidden when the section is expanded
- New settings/providers/_status_pill partial (ok/warn/err/off states)
- Add SettingsHelper#provider_summary to centralise the connected-vs-not
logic already scattered across panel partials
- Refactor show.html.erb to pass status to every section and sort
family_panels by connection state
- Add settings.providers.status.* i18n keys
- Add system tests asserting pill renders and sort order
https://claude.ai/code/session_01KW2HCN9rP1fiyQuw7Cju9D
* feat(settings/providers): group providers into Connected and Available
Partition the provider list in the controller into @connected_providers
and @available_providers based on provider_summary status, and render
each group under its own heading with a count. Auto-open the section
when only one provider is connected. Adds an empty-state line when
nothing is connected yet.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
* feat(settings/providers): health strip, action-needed group, and sync error surfacing
- Extend provider_summary to return :err/:warn with meta text by checking
latest sync per item (window function, same pattern as ProviderConnectionStatus)
and Enable Banking session expiry within 7 days
- Partition provider entries into three groups: Connected (:ok), Action needed
(:warn/:err, auto-opened), Available (:off)
- Add Settings::HealthSummary ViewComponent — four-tile grid showing Connected,
Action needed, Errors, and Accounts synced counts
- Render health strip directly under page description; omit Action needed heading
when group is empty
- Add i18n keys for tile labels, group heading, and all meta strings
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(settings/providers): card grid for available providers with connect drawer
- Add Provider::Metadata registry with static display data (region, kind,
tier, maturity, logo) for all 11 providers
- Add Settings::ProviderCard ViewComponent rendering logo square, name,
Beta/Alpha pill, meta line (region · type · tier), tagline, and Connect link
- Add connect_form action + route (GET /settings/providers/:key/connect_form)
that opens the existing panel partial or config form in a DS::Dialog drawer
- Replace the Available accordion loop with a 2-column responsive card grid;
empty state when all providers are connected
- Fix layout override: use turbo_rails/frame layout for frame requests so the
drawer response is not wrapped in the full settings layout (was causing
Turbo to pick the empty outer drawer frame instead of the filled one)
- Add SyncAllProvidersJob and last_sync_all_attempted_at migration (sync-all
throttle support)
- Unify Connected + Action needed into a single "Your connections" section;
items with warn/err status auto-open
- Fix Enable Banking grouping: items with expired sessions were returning
:off (Available) instead of :warn (Your connections); gate now checks
any? instead of any?(&:session_valid?)
- Add reconsent_required locale key for fully-expired EB sessions
- Surface Beta/Alpha maturity pills on connected provider accordion rows
via new badge: param on settings_section helper
- Add i18n taglines for all 11 providers; add connect and empty_available keys
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(settings): retire /settings/bank_sync; merge into providers page
- Delete Settings::BankSyncController and its views (the providers page is
now a strict superset of what bank_sync offered)
- Add permanent 301 redirect: GET /settings/bank_sync → /settings/providers
- Collapse nav to a single "Bank Sync" entry pointing at /settings/providers;
remove the duplicate admin-only "Providers" entry from the Advanced section
- Remove "Providers" from SETTINGS_ORDER; point "Bank Sync" at
settings_providers_path for next/prev navigation
- Rename page title to "Bank Sync"; replace admin-credential lede with
user-facing copy ("Connect external accounts…")
- Update breadcrumb: Home → Bank sync
- Add controller test asserting 301 status and Location header
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Migrations are 7.2 here
* Minimize schema noise
* Schema duplication
* Small copy edits
* Fix tests
* Address provider settings review feedback
* refactor(settings/providers): finish design-review cleanup pass
Picks up the remaining items from Claude Design's review of #1710
that the previous review-feedback commit didn't cover.
DS / casing
- Sentence-case the page title ("Bank Sync" -> "Bank sync") and
align the nav label.
- Drop the card hover-lift (shadow-border-sm) in favour of
bg-container-hover; per the DS, card hover is colour-only.
- Whole-tile click target on each provider card — the inner
"Connect ->" link was a hit-target inversion.
- Set Sync all to whitespace-nowrap so the label stops wrapping at
narrow viewport widths.
UX simplifications
- Drop the four health-summary tiles (per-row warn/err pills already
surface the signal at the scale this app sees). Removes
Settings::HealthSummary, the @health_counts controller block, and
the now-unused health.* locale keys.
- Hide "Your connections" heading + empty-state line when no
providers are connected — the lede already invites a connect.
- Drop the redundant "Free" tier from per-card meta lines (printed
10x for one fact); "Paid" still surfaces on Plaid.
Tests updated to drop the obsolete tiles assertion and switch the
provider-card click selector to look up the (now whole-card) anchor
by provider name.
* feat(settings/providers): replace Add another provider CTA with a search + kind filter
Per the design review, the "Add another provider · Browse providers"
card was a redirect to content one scroll-tick away. A search input
plus kind chips lets users self-segment the catalog and is the right
tool once it grows beyond the four to twelve providers we ship today.
- New providers_filter Stimulus controller — case-insensitive free
text search across name/region/kind, plus a chip group with
All / Banks / Crypto / Investment that toggle visibility via
Tailwind's `hidden` class.
- _search_filters partial: search box (count-pluralized placeholder)
+ chip group, ARIA-labelled and aria-pressed for the chips.
- ProviderCard exposes filter_data (target + name/region/kind data
attrs) so the controller can match without re-rendering.
- Lunchflow's `kind` was "Lunch" — switched to "Bank" so it falls
under the Banks chip alongside its actual offering (it aggregates
banks).
- Drops the add_provider_cta partial and its locale entries; adds
search_filters.* and an empty_filter message.
* Private method fix
* refactor(settings/providers): drawer cleanup, header lock-up, trust statement
Per the design review's §07.
- Drop the trailing "Configured / Not configured" footer status from
every provider panel (binance, coinbase, coinstats, indexa_capital,
lunchflow, mercury, simplefin, snaptrade, sophtron, provider_form).
The parent details section's status pill already carries that
signal; the footer was redundant — and the copy/styling was
inconsistent across panels (free-text vs. dot pill, "configured"
vs. "not connected").
- Connect drawer gets a header lock-up: small logo chip + provider
name + maturity badge, mirroring the available-card layout.
Implemented as _drawer_header partial; connect_form passes
custom_header: true to DS::Dialog so we own the row.
- Drawer footer trust statement: "Read-only — Sure can never move
money. Stored encrypted." A single-line reassurance covering all
panels.
- Sentence-case the hardcoded primary buttons that were Title Case:
"Save Configuration" -> "Save and connect"
"Update Configuration" -> "Update connection"
"Connect Bank" -> "Connect bank"
Affects simplefin, lunchflow, enable_banking, provider_form. The
i18n'd panels (binance, coinbase, coinstats, indexa_capital,
mercury, snaptrade, sophtron) keep their existing keys.
* chore(locales): drop unused provider-panel status strings
Footer "Configured / Not configured" status was removed from each
provider panel partial in the prior drawer-cleanup pass; the matching
i18n keys are no longer referenced. Removing them across every
locale to keep the catalogue clean.
Dropped (15 keys × varying locale coverage, 36 line removals across
24 files):
- coinstats_items.new.{status_configured_html, status_not_configured}
- indexa_capital_items.panel.{status_configured_html, status_not_configured}
- mercury_items.provider_panel.{configured_html, not_configured, accounts_link}
- sophtron_items.sophtron_panel.status.{configured_html, not_configured}
(parent `status:` removed where it became empty)
- providers.snaptrade.{status_needs_registration, status_not_configured}
(status_connected stays — still used by the lazy-load summary)
- settings.providers.{binance_panel, coinbase_panel}.{status_connected, status_not_connected}
* feat(settings/providers): connected-state polish per design §05 + Linked institutions rename
Building the next phase of the design review. Pulls forward the
slim health strip, denser connection rows, and "Linked institutions"
heading rename — the small Phase A lift the designer flagged in
§08 of the doc.
- New _health_strip partial: single-line at-a-glance pulse —
connected count + needs-attention count + accounts syncing +
last-synced timestamp. Renders only when at least one provider
is linked or needs action.
- New _connection_row partial replaces the generic settings_section
call for providers. Tighter rows: text-sm title (was text-lg),
px-4 py-3.5 padding, single-line summary (chevron + name +
maturity badge + meta + status pill + sync action). Warn/error
rows get a coloured outline (border-warning/25 or
border-destructive/25) so the at-risk row stands out without
shouting.
- "Sync all" button restyled to match the design's secondary
button: text-primary, alpha-black-100 border, rounded-[10px],
padding 7px 12px (was the broader px-3 py-1.5 ghost).
- "Your connections" → "Linked institutions" heading, lifted from
the designer's Phase-C reconciliation note. Primes users for the
Option-C institution-search wizard six months early; existing
i18n key stays as `groups.your_connections` for now to keep the
rename to a single value flip.
- Controller computes the new @health hash (connected,
needs_attention, accounts_syncing, last_synced_at) feeding the
strip; brings back the single accounts query that was removed
with the four-tile component.
System test updated for the new heading copy.
* fix(settings/providers): align connected state with the final design mock
Tightening the §05 polish to match the user-confirmed final design.
- Revert "Linked institutions" → "Your connections". The §08
designer note about the Phase-A heading rename didn't carry
forward to the final mock; keep the original wording.
- Drop the warn/err auto-open on connection rows. The design shows
Enable Banking collapsed with a warn-outline and a status pill —
no auto-expanded form. Single-connection auto-open kept (handy
when the page is otherwise empty).
- Hide the "accounts syncing" segment in the health strip when the
count is 0 — the design mock assumes a populated number; an
always-visible "0 accounts syncing" reads as a placeholder.
- Strip the leading "about " from `time_ago_in_words` everywhere
the result is shown to the user (health strip "Last synced %{time}
ago" plus per-row "Synced %{time} ago" meta). Matches the design's
shorter copy.
* refactor(settings/providers): tighten paddings, dedupe maturity badge, semantic + a11y fixes
Pixel-level alignment to the design's §05 mock + cleanup from a DS
audit pass.
Paddings, margins, font sizes
- Health strip: my-4 → mt-4 mb-5 to match the design's 16px / 20px
vertical breathing room.
- Search filters bar: gap-2 → gap-2.5; mt-2 → mt-5 mb-3 (was missing
the 12px bottom margin entirely).
- Search box: rounded-lg → rounded-[10px]; px-3 py-2 → px-[14px]
py-[9px]. Search icon downsized w-4 → w-3.5 to match.
- Chip group: p-1 → p-[3px]; rounded-lg → rounded-[10px].
- Chip: py-1 → py-[5px]; rounded-md → rounded-lg.
- Group heading: mt-2 → mt-[18px]; mb-1 → mb-1.5.
- Status pill: text-xs → text-[11px].
- Provider card: gap-3 → gap-2.5 (outer + top); name gets explicit
text-sm; tagline + foot 14px → 13px; arrow icon w-4 → w-3.5.
- Sync icon button: p-1 → fixed w-7 h-7 (28×28) so the row hit
target matches the design's column width.
- Connect drawer header logo glyph: text-[10px] → text-xs (matches
the available card's logo-glyph treatment).
Component / partial cleanup (DS audit follow-ups)
- New _maturity_badge partial replaces the inline span that was
duplicated in 3 places (_connection_row, _drawer_header,
provider_card.html.erb).
- Settings::ProviderCard.maturity_label class method centralizes the
MATURITY_LABELS lookup; callers no longer reach into the constant.
- _connection_row title: <h2> → <h3> (the row sits inside the
"Your connections" h2 group heading; nested h2s flattened the
outline).
- show.html.erb encryption error: <h3> → <h2> for the same reason.
Locale
- Drop orphaned keys: settings.providers.groups.connected and
groups.needs_attention (no view code uses them) plus the leftover
show.coinbase_title block.
- Health strip "needs reconsent" → "needs attention" so the strip
copy lines up with the per-row status pill ("Action needed") and
the original group heading wording.
A11y
- focus-visible:ring-2 on chip buttons, provider-card link, and
focus-within:ring-2 on the search input wrapper. Keyboard users
now get a visible focus state.
- Search input: explicit autocomplete="off" (erb_lint hint).
* fix(settings/providers): icons + search input height
- Icons were rendering at 20px because the application_helper's `icon`
default size (`md` = w-5 h-5) was beating the inline class override
in compiled CSS source order. Pass `size: "sm"` and use the project's
`!w-3.5 !h-3.5` important-prefix pattern (precedent: dashboard.html.erb)
so chevron, refresh-cw, search, check, circle-alert, and arrow-right
all render at the design's 14px.
- Search input was 54px tall because @tailwindcss/forms applies
`padding: 8px 12px` to bare `<input type="search">`. Override with
`!p-0 focus:ring-0 focus:shadow-none` so the wrapping div's padding
alone defines the box (38px total — matches the design).
* refactor(settings/providers): align Sync all + search input with DS, address review feedback
- Sync all: replace the hand-rolled `button_to` with `DS::Link.new(variant: "outline", method: :post)` — same component as the
"Identify Patterns" button on the recurring-transactions page.
- Search input: switch to the icon-overlay pattern used by the
Manage-currencies and transaction filter rows
(relative wrapper + absolutely positioned search icon +
bordered input with `focus:ring-gray-500`). Brings the keyboard
focus state in line with the rest of the app's filterable lists.
- SnapTrade panel: restore the "needs registration" status row that
the drawer-cleanup pass dropped along with the redundant
Configured/Not configured footer. The unregistered case is
meaningful state, not redundant chrome.
- Move the slim health-strip computation out of the controller and
into `SettingsHelper#provider_health_strip` (Convention 2: skinny
controllers).
- Extract `concise_time_ago` helper so the "drop leading 'about '"
trick stops being duplicated 3x.
- `Settings::ProviderCard#maturity_label` (instance) now delegates
to `.maturity_label` (class) instead of duplicating the lookup.
- Drop unused `warn_or_err` local in `_connection_row`.
- Replace the `data-controller` string-injection + html_safe in
`_connection_row` with `tag.details(data: ...)`; safer and more
idiomatic.
- Add a system test for the empty-filter message wiring.
* fix(settings/providers): drawer trust statement uses border-tertiary
`border-secondary/10` was reaching for the text-foreground token at
10% opacity for a divider. The project ships a dedicated divider
token (`border-tertiary`, ~8% black) used by DS::Menu, the holdings
page, and admin/sso forms. Switching to it makes the trust-statement
HR match every other thin divider in Sure and stops misusing the
text token as a border.
* refactor(settings/providers): swap arbitrary Tailwind values for scale tokens
Per the user's directive — DS-compliance over pixel-perfect alignment
with the design mock. Walked the design audit and applied every swap
that lands within ±2px of the original.
Swaps:
- _health_strip: gap-[18px] → gap-5 (+2), px-[14px] → px-3.5 (=),
text-[13px] → text-sm (+1).
- _search_filters: chip group p-[3px] → p-1, rounded-[10px] →
rounded-xl (concentric with rounded-lg inner pills), chip py-[5px]
→ py-1.
- _status_pill: text-[11px] → text-xs.
- _group_heading: mt-[18px] → mt-5.
- _maturity_badge: text-[10px] → text-xs.
- provider_card: tagline + foot text-[13px] → text-sm.
Kept arbitrary: `min-w-[200px]` in _search_filters — nearest scale
tokens are min-w-48 (192px) and min-w-52 (208px); both are noticeable
layout shifts for a one-off responsive guard. Worth keeping the
arbitrary here.
Net: 9 of 10 arbitrary values gone. Visual delta: max +2px on a
single value. Design mock and DS scale now agree.
* revert(settings/providers): drop the slim health strip
Per-row status pills already carry the at-a-glance signal (connected
/ action needed) at the scale this app sees (1–4 connections per
family). The strip was redundant chrome for almost every user; only
worth bringing back if the catalog grows to a point where the row
list itself stops fitting on a single screen.
- Delete _health_strip.html.erb partial.
- Drop @health controller assignment + provider_health_strip helper.
- Drop unused settings.providers.health_strip.* locale keys.
- concise_time_ago helper stays — still used by per-row meta text.
* refactor(settings/providers): align with DS conventions
Two consistency wins from the screenshot/DS audit pass.
Sync icon button now renders DS::Button (variant: icon, size: sm)
instead of a hand-rolled `button_to`. Same component used by other
icon-only actions across the app (settings/profiles, layouts/imports).
Visual delta: 28×28 → 32×32 (DS sm size). Accept the +4px for
consistency. `event.stopPropagation()` still wired via the form opt
so the row's <details> doesn't toggle when the user clicks the
button.
Group heading now follows the established Sure section-label style
(`text-xs font-medium text-secondary uppercase`) used by
`_settings_nav` and the imports/categories surfaces. The previous
sentence-case `text-sm text-primary` was a one-off that didn't
match the rest of the app. Locale strings stay sentence-case;
uppercase comes from CSS `text-transform`. Tests updated to
case-insensitively match the rendered heading text.
* fix(provider/metadata): add plaid_eu entry
`plaid_eu` is registered as a separate Provider::ConfigurationRegistry
entry but had no Provider::Metadata row, so its card in the
Available grid fell through to the gray-500 default and rendered
empty (no region, kind, tier, or tagline). The title also came out
as "Plaid Eu" because `titleize` doesn't know "EU" is an initialism.
- Add a `plaid_eu` row to Provider::Metadata::REGISTRY with the same
shape as `plaid` (US → EU, otherwise identical).
- Introduce an optional `name:` field in metadata; controller falls
back to it before titleizing the provider key. Lets `plaid_eu`
render as "Plaid EU".
- Add the missing `settings.providers.taglines.plaid_eu` translation.
* fix(settings/providers): center-align Sync all next to the lede
`items-start` made the button hug the first line when the lede wrapped;
on a single line the button sat at the top of the text bounding box
which read slightly off. Center matches the dominant convention
across the rest of settings (api_keys, securities, hostings, _section,
_settings_nav_link_large).
* fix(settings/providers): drop colour palette + filter polish + drawer warnings
Round of design-feedback fixes.
Provider chips
- Drop the per-provider raw Tailwind palette (bg-blue-600 etc.) from
Provider::Metadata. All cards + drawer logo lock-up now use
bg-surface-inset + text-primary, matching the design's §04 "drop
colour entirely" recommendation. Solves the long-standing §01
BLOCKER without externalising brand assets. Re-introducing logos
later just means an optional logo_svg: field on metadata.
- ProviderCard component drops the `logo_bg:` parameter; the chip
is now styled in the template.
Filter / search
- "Available · N" count and the empty-filter state now update
client-side as the chip filter and free-text search narrow the
grid (new `count` Stimulus target + dedicated update path).
- Empty-filter state now offers a Clear filters button that resets
both the search input and the active chip in one click.
- Search placeholder drops the drifting "Search 9 providers" count
for plain "Search providers" — the section heading carries the
number.
- Chip labels normalised to plural where natural: "Banks · Crypto ·
Investments" (Crypto stays as the mass noun).
Drawer copy / treatment
- "IP Whitelisting Required" → "IP whitelisting required" (DS
sentence-case).
- Binance "do NOT enable withdrawal permissions" lifted out of
inline red-text into a proper bg-warning-50 border-warning-200
alert block with an alert-triangle icon. Matches the api_keys /
hosting alert pattern.
- SnapTrade free-tier inline alert-triangle now uses `size: "sm"`
so the icon stops rendering at 20px next to 14px body text.
Spacing
- Group-heading margin top bumped 5 → 6 (20→24px) so the eyebrow
has more breathing room above the search bar.
* refactor(settings/providers): drawer alerts use DS::Alert; drop card-in-card
Two consistency fixes from a design-review pass.
DS::Alert adoption
- Replaces 9 hand-rolled error blocks across the provider panels
(`bg-destructive/10 text-destructive ... line-clamp-3`) with
`DS::Alert(variant: :error)` — the project's existing primitive.
- Replaces the just-shipped Binance no-withdraw warning block with
`DS::Alert(variant: :warning)` instead of a hand-rolled
`bg-warning-50 border-warning-200` card.
- Replaces the SnapTrade free-tier inline icon-prefixed warning
paragraph with `DS::Alert(variant: :warning)` — proper alert
treatment for an actual warning, not body copy.
- Replaces the Enable Banking "Configuration locked" inline
`bg-warning/10` two-paragraph block with `DS::Alert(variant: :warning)`
using `safe_join` for the title + body.
- Replaces the encryption-error block at the top of show.html.erb
with `DS::Alert(variant: :error)`, again via `safe_join`.
Mercury card-within-card
- The "Add another Mercury connection" form was wrapped in a
`<details>` `bg-container shadow-border-xs rounded-xl` card. In
the Connect drawer (always 0 existing connections), that wrapping
card-inside-the-drawer-card has no value — the form is the only
thing on the surface. Drop the wrapper when no connections exist;
keep the heading + form inline. When 1+ connections exist (the
section page) the heading hints "+ Add another connection"
without the disclosure indirection.
Trade-off: the error-alert blocks lose their `line-clamp-3` /
`title=` truncation. Acceptable for now — DS::Alert can grow a
truncate option as a follow-up if needed.
Open follow-up: DS::Alert itself uses raw Tailwind palette
(`bg-yellow-50` etc.) instead of semantic tokens, and only accepts
a single string `message:`. A separate issue tracks this.
* fix(settings/providers): hoist warning alerts to top of drawer
DS::Alert convention across the rest of the app: alerts sit at the
top of the form / page / section, not floating between content
blocks. The Binance no-withdraw warning and SnapTrade free-tier
warning were rendering between the setup-instructions list and the
form fields — visually wonky.
Move both to the top of their respective panels so the warning is
the first thing the user sees when the connect drawer opens.
Existing precedents this aligns with:
- accounts/_form.html.erb (error alert above form)
- valuations/new.html.erb (error alert above form)
- other_assets/new.html.erb (info alert above form)
- holdings/show.html.erb (warn alerts above content)
* fix(DS::Alert): align icon to cap-height of first text line
`items-start` on the container made the icon's top edge flush with
the text's top edge, leaving the icon's optical center sitting below
the text's first-line center. The hand-rolled alerts elsewhere in
the codebase (api_keys/new, hostings/_sync_settings, holdings/show)
all add `mt-0.5` to the icon for the same reason — fold that into
the primitive so every caller gets the cap-height alignment.
* copy(settings/providers): tighten alert messaging per voice review
Copy expert pass on the new provider drawer alerts. House style:
sentence case for titles, lead with the action, drop "Warning:" /
"Please" filler (the alert variant icon already signals tone),
prefer one short sentence + optional title-paragraph for emphasis.
- Binance no-withdraw warning: was a single line "Warning: do NOT
enable withdrawal permissions" — alarmist without context. Now
splits into "Read-only key only" (title) + "Don't enable
withdrawal permissions when creating your Binance API key — Sure
only needs read access." (body).
- SnapTrade free-tier note: "Free tier includes 5 brokerage
connections. Additional connections require a paid SnapTrade
plan." → "SnapTrade's free tier covers 5 brokerage connections.
Upgrade on SnapTrade for more."
- SnapTrade connection-limit-info inside the brokerage list: cut
entirely. The drawer already shows the cap; restating it in the
list was noise.
- SnapTrade needs-registration: "Credentials saved — finish
registration to connect a brokerage." → "Credentials saved.
Finish setup to connect a brokerage." ("registration" was
ambiguous — register where, with whom?)
- Enable Banking "Configuration locked" body: "Credentials cannot
be changed while you have active bank connections. Remove all
connections first to update credentials." → "Disconnect all
linked banks before changing these credentials." Same meaning,
half the words.
- Encryption-error block: title-cased "Encryption Configuration
Required" → "Encryption keys missing"; body strips "Please
ensure" filler and the parenthetical credential dump, leaving
the three credential names inline as a clean list. Self-hosters
still get exactly the names they need to set.
* feat(settings/providers): SetupSteps partial for connect-drawer instructions
Per the design's drawer-cleanup follow-up. Replaces the per-panel
"Setup instructions:" + ordered list + "Field descriptions:" block
with a shared boxed-step component.
The new partial — `_setup_steps.html.erb` — takes a `steps:` array
of strings (or html_safe strings for inline links / code) plus an
optional `help:` hash for a docs link below the steps. The eyebrow
label is "Setup" (uppercase, tracking-wider) matching Sure's other
section labels.
Applied across all eleven provider panels:
- _provider_form (Plaid + Plaid EU): field descriptions move to
per-field helper text below the input.
- _binance, _coinbase, _coinstats, _indexa_capital,
_lunchflow, _mercury, _simplefin, _snaptrade, _sophtron,
_enable_banking: ordered list + duplicate "Field descriptions"
block both replaced by the partial.
- Some panels' inline copy tightened in the same pass (Lunch Flow,
SimpleFIN, Enable Banking) — the design copy is shorter than the
current legacy strings; a copy-pass through every panel can
follow as a separate cleanup.
Token notes: uses scale tokens (`rounded-xl`, `text-xs`/`text-sm`,
`tracking-wider`) instead of the design mock's exact arbitrary
values, per the consistency-over-design-specs directive on this
branch.
* fix(settings/providers): tighten panel spacing + relocate per-panel notes
Read-flow audit on each connect drawer. The uniform `space-y-4`
treated every block (alert, steps, info card, fields, button) the
same — visually they were five sibling boxes with no grouping. The
fix is per panel; some notes belong as helper text on a specific
field, others as a tightly-grouped pre-fill primer.
Per panel:
- Binance: IP-whitelisting card now matches the setup_steps box
(`bg-surface-inset rounded-xl`) and is wrapped with setup_steps
in an inner `space-y-2` so they read as a single pre-fill primer
cluster. Same eyebrow treatment ("IP whitelisting required") so
the two boxes look like sister panels, not unrelated chrome.
- SnapTrade: drop the description paragraph above setup_steps. The
available-providers card grid already markets SnapTrade
("Connect brokerage accounts via the SnapTrade aggregation
network."); repeating in the drawer was duplication.
- Mercury: move the sandbox-API note out of its standalone <p>
below setup_steps and into per-field helper text under the
base_url field — the user only cares about the sandbox URL when
they're filling that field. Applied to both the per-item edit
form and the add-new form.
- _setup_steps partial: drop the now-pointless `mb-2` (outer
`space-y-4` already controls the gap; bottom-margin was dead
CSS thanks to margin-collapse rules with the next sibling's
margin-top).
* fix(settings/providers): plaid + indexa drawers join the SetupSteps look
Two unifying fixes after the panel-by-panel screenshots showed
mixed treatments.
Plaid + Plaid EU
- The registry-driven panel (_provider_form) was still rendering
each adapter's markdown `description` block as plain prose
("Setup instructions: 1. Visit the Plaid Dashboard ..."). Other
panels switched to the SetupSteps box; Plaid was the odd one out.
- Drop the markdown `description` block from both plaid_adapter
and plaid_eu_adapter. Render setup_steps in _provider_form for
these two provider keys via inline ERB (link helper handles the
Plaid Dashboard link cleanly; the regional differences fold to
the same dashboard URL with a different account scope).
- Other registry-based providers fall through to the previous
markdown description path — no behavior change for them.
Indexa Capital
- The API token field was wrapped in a `bg-surface border` "card"
that duplicated the field label inside as a heading and put the
description above the input. Same pattern the user flagged as
the "card within input" anti-shape.
- Drop the wrapper. The styled-form input renders its own label;
description moves to per-field helper text below the input,
matching the pattern used by Plaid (provider_form) and Mercury.
* fix(settings/providers): surface configured plaid_eu + dedup show context
provider_summary had no plaid_eu branch — configured plaid_eu was
falling through to status :off and rendering in Available even with
credentials set. Collapse plaid + plaid_eu into a single registry
check.
Drawer title for non-panel configurations was provider_key.titleize,
which produced "Plaid Eu" while the available card grid used
metadata[:name] = "Plaid EU". Read from metadata first.
While here:
- compute_provider_sync_health no longer relies on
instance_variable_get; pass family_panel_items explicitly so the
hash-key/ivar-name coupling is gone.
- drop unused .includes(:syncs, :mercury_accounts) and
.includes(:snaptrade_accounts) from prepare_show_context. The show
view only consults summary[:status]; the eager-loads were carried
over from connect_form (which has its own load_provider_items).
* i18n(settings/providers): localize plaid setup steps + drop dead defaults
The plaid + plaid_eu setup steps in _provider_form.html.erb were
hardcoded English strings. Move them to settings.providers.plaid_panel
(shared) + plaid_eu_panel (EU-specific step 1) so they can be
translated like every other panel.
_setup_steps.html.erb was passing default: "Setup" / "Need help?" to
t(), masking missing translations in non-EN locales. Both keys exist
in en.yml — drop the defaults so missing translations actually
surface.
* test(settings/providers): cover plaid_eu, clear filters, warn outline
Three system test additions:
- Configured plaid_eu surfaces in Your connections (regression guard
for the helper fix; previously fell through to Available).
- Clear filters button resets input + chip state and brings cards
back into view.
- :warn-state connection row carries the border-warning/25 outline
that distinguishes it from an :ok row.
* copy(settings/providers): drop em dashes, naturalize phrasing
Sweep through every string this branch added and replace em-dash
splices with full sentences or simple connectives.
en.yml:
- drawer_trust_statement now reads "Read-only access. Sure can never
move money, and your credentials are stored encrypted." instead
of em-dash splicing.
- sync_all_recently / recently_synced split into two sentences.
- binance_panel.no_withdraw_body, plaid_panel.step_1_html / step_2,
plaid_eu_panel.step_1_html same treatment.
Hardcoded panel steps (enable_banking, lunchflow, simplefin) become
"Go to <link> and …" or "Go to <link> for …" instead of the
"<link> — get …" splice. Same setup_steps comment cleaned up.
* fix(settings/providers): address CodeRabbit pass on PR #1717
Fixed:
- Localize the setup steps in _enable_banking_panel,
_lunchflow_panel, and _simplefin_panel. The em-dash sweep had
rewritten these into hardcoded English; they now route through
settings.providers.{enable_banking,lunchflow,simplefin}_panel
step_1_html / step_2 / step_3 keys, mirroring the plaid_panel
treatment.
- connect_form: silent redirect when provider_key is unknown now
carries an alert (settings.providers.not_found) so misrouted
links don't drop users on the page with no feedback.
- sync action: redirect notice now reflects whether anything was
actually scheduled — adds settings.providers.sync_provider_no_items
for the "all items already syncing or none exist" path.
- Family::Syncer test: count plaid_items via the .syncable scope to
match what Family::Syncer actually schedules (already done for
binance_items in the same test).
Skipped, with reasons:
- focus:ring-gray-500/-gray-900 in coinstats / coinbase / simplefin /
search_filters: tracked under issue #1715 as part of the raw-palette
→ DS-token sweep across the whole codebase.
- Coinbase #0052FF brand-color wrapper: tracked under PR #1710's
follow-up tracking comment as the deferred Provider::Metadata
colour-palette decision (designer §01).
- Sophtron submit-button extraction into DS::Button: same
deferred sweep — every panel hand-rolls this class string;
one-off extraction would just churn.
- Redundant .html_safe on _html keys in coinstats: tracked in #1715.
- _provider_form.html.erb env hint, "Optional" placeholder, "Save and
connect" submit: pre-existing strings not added on this branch.
- Renaming sync_health_for's :stale to :data_stale: pre-existing
shape, refactor scope.
- Plaid_eu using plaid_panel.step_2/step_3 keys: deliberate. Same
English copy across both providers; duplicating keys would just
give translators twice the work for identical strings.
- _enable_banking_panel / _lunchflow_panel / _simplefin_panel
alert + submit + button labels: pre-existing hardcoded strings
from before this branch. Setup steps were the strings actually
touched in the em-dash sweep, so those got localized; the rest
belong in a broader panel-i18n pass.
Verified:
- bundle exec erb_lint on the three panels: clean.
- bin/rubocop on controller + test: clean.
- bin/rails test test/models/family/syncer_test.rb
test/controllers/settings/providers_controller_test.rb:
23 runs, 85 assertions, 0 failures.
- DISABLE_PARALLELIZATION=true bin/rails test
test/system/settings/providers_test.rb:
15 runs, 38 assertions, 0 failures.
* fix(db): rename migration to clear collision with main's 20260508120000
Main's PR #1705 (Sophtron manual sync) shipped a migration with
the same 20260508120000 timestamp as our
add_last_sync_all_attempted_at_to_families migration. The merge
that brought main into this branch left both files at the same
prefix, which trips Rails' "Duplicate migration" guard at
db:schema:load time and broke CI.
Renaming our migration to 20260510120000 keeps the column it adds
intact (already in db/schema.rb) and bumps the schema version to
match. No DB-level change.
* fix(settings/providers): card + strip a11y polish
- Bring back the slim health strip; gate behind 10+ accounts
(HEALTH_STRIP_MIN_ACCOUNTS) so it stays out of the way for
small libraries where per-row pills already carry the signal.
- Status pill: drop the bg-{c}/10 text-{c} pattern (failed AA
on warn / err); switch to bg-surface-inset text-primary with
the dot still carrying semantic colour. Passes AA in both
themes; the dot is the only colourful affordance.
- Maturity badge: bg-alpha-black-50 was invisible against the
hovered card bg in light mode and against bg-container in
dark mode. Move to bg-surface-inset + border-tertiary so it
stays delineated through hover and dark theme.
- Provider card: keep the bg shift on hover (now bg-surface-inset
for a perceptible delta), focus ring promoted alpha-black-100
-> alpha-black-300 (visible to keyboard users), meta line
text-subdued -> text-secondary (text-subdued failed AA at
2.86:1 against bg-container).
- Restore the per-provider logo palette dropped in
|
||
|
Juan José Mata
|
81cdccb768 |
[codex] Complete Sophtron account mapping (#1698)
* Complete Sophtron account mapping * Clarify Sophtron login challenge flow * Add Sophtron connection UI timeout * Treat Sophtron timeout jobs as failed * Reset failed Sophtron connection state * Handle stale Sophtron connection jobs * Advance Sophtron polling timeout * Shorten Sophtron connection timeout * Fix Sophtron modal polling updates * Stabilize Sophtron MFA polling * Give Sophtron OTP challenges more time * Clarify Sophtron institution login failures * Extend Sophtron polling during login progress * Probe Sophtron accounts after completed MFA step * Align Sophtron dialogs with design system * Start Sophtron initial load after linking accounts * Fix Sophtron initial transaction load * Fail Sophtron sync without institution connection * Fix tests * Wrap Sophtron account linking in transaction * Wrap Sophtron provider responses * Fix Sophtron MFA security tests * Guard Sophtron MFA challenge arrays * Respect Sophtron initial load window * Use unique Sophtron MFA answer field ids * Address Sophtron review follow-ups * Fix Sophtron transaction sync refresh * Avoid blocking Sophtron refresh polling * Move Sophtron account helpers to model * Keep Sophtron grouping provider-level * Start new Sophtron institution links * Isolate Sophtron institution connections --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
GermanDZ
|
9cc52b9d35 |
fix: handle OpenAI Responses API stream errors instead of crashing (#1669)
The streaming code assumed every stream produced a `response.completed` event and dereferenced its data unconditionally, causing `undefined method 'data' for nil` whenever OpenAI emitted `response.failed`, `response.incomplete`, or a top-level `error` event (e.g. expired `previous_response_id`, context-window overflow, transient upstream failures). Surface a descriptive `Provider::Error` instead. - Extend `ChatStreamParser` to recognise `response.failed`, `response.incomplete`, and `error` events and emit an `error` chunk with a `StreamErrorData` payload (event, message, code, details). - In `Provider::Openai#native_chat_response`, detect the missing `response` chunk, build a user-facing error message from the collected error chunk, and raise `Provider::Error`. - Add unit tests for the parser (8 cases) and integration tests for the error path in the chat response flow. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Abhinav Dhiman
|
139c89d0f4 |
feat(investments): add India investment subtypes and exchange support (#1659)
* feat(investments): add India investment subtypes and exchange support * fix(yahoo-finance): scope Indian exchange de-duplication per company instead of globally Resolves feedback from Codex and CodeRabbit on #1413. prefer_indian_exchange previously collapsed all Indian securities into a single entry, silently dropping unrelated tickers. Now groups Indian listings by name and only de-duplicates within each group, so distinct companies (e.g. Reliance and Infosys) are preserved while NSE/BSE dual-listings still prefer NSE. - Derive India subtype keys dynamically from Investment::SUBTYPES in tests - Fix missing keyword arguments in Security.new test calls * refactor(yahoo-finance): generalize exchange config and dual-listing de-duplication Replaces hardcoded Indian exchange logic with a declarative EXCHANGE_CONFIG hash that maps ISO MIC codes to Yahoo-specific settings (symbol suffix, default currency, dual-listing group, and preference rank). This makes adding new markets a one-line hash entry instead of scattered conditionals. * fix(yahoo-finance): normalize security names for dual-listing de-duplication * fix(yahoo-finance): skip dual-listing de-duplication when filtering by exchange * fix: address PR review feedback for India market support - fix cache key mismatch in fetch_security_price by normalizing symbol before building cache key - remove dead YAHOO_EXCHANGE_CURRENCY constant - tighten normalize_symbol guard to use end_with?(suffix) instead of include?('.') - remove misleading '# India' comment from Property::SUBTYPES - remove 'rented' property subtype in favor of 'investment_property' - rename 'demat' to 'indian_stocks' for clarity - add INR to CURRENCY_REGION_MAP so India appears first for INR users - add dotted-symbol regression test for normalize_symbol * fix(investments): rename 'demat' subtype to 'indian_stocks' and remove trailing comma |
||
|
ghost
|
6c84fc760e |
fix(mercury): support named multiple API connections (#1627)
* fix(mercury): support named multiple connections * fix(mercury): address multi-connection review feedback * fix(mercury): localize connection labels * fix(mercury): strip API tokens before provider calls * test(mercury): localize provider config assertions * fix(mercury): address multi-connection review * refactor(mercury): simplify connection selection failure |
||
|
GermanDZ
|
c5503320af |
Fix IndexaCapital sync, account setup, and balance/type bugs (#1562)
* Add missing IndexaCapitalItem::SyncCompleteEvent
Syncable#sync_broadcaster instantiates self.class::SyncCompleteEvent,
which is implemented for every other provider (Plaid, Lunchflow,
Mercury, etc.) but was missing for IndexaCapitalItem. The error was
swallowed by Sync#perform_post_sync's rescue, so syncs appeared to
succeed but post-sync UI broadcasts never fired:
Error performing post-sync for IndexaCapitalItem (...):
uninitialized constant IndexaCapitalItem::SyncCompleteEvent
This adds the class, modeled on LunchflowItem::SyncCompleteEvent,
restoring per-account and per-item Turbo broadcasts after Indexa
Capital syncs.
* Fix IndexaCapital account setup never creating accounts
complete_account_setup read params[:accounts], but the form in
setup_accounts.html.erb submits account_ids[] (array) and
sync_start_dates[<id>] (hash). The hash was always empty, so every
submit hit the empty-config branch and bounced back with
"No accounts to set up." — accounts were never created.
The controller also branched on config[:account_type] / config[:subtype]
even though the form has no account-type picker (Indexa Capital is an
investment-only broker). Rewrote complete_account_setup to consume the
form's actual params and infer the accountable type as Investment from
indexa_capital_account.account_type.
* Fix IndexaCapital balance double-count and account type
Two more issues in the IndexaCapital flow that surfaced once accounts
could actually be created (see prior commit):
1. Accountable type was inferred from indexa_capital_account.account_type
("mutual" / "pension"), but infer_accountable_type doesn't recognize
those values and falls through to "Depository". The result: every
imported Indexa account showed up as a Cash depository account
instead of an Investment account, hiding holdings/trades surfaces.
Indexa Capital is investment-only, so hard-code the accountable
type to Investment.
2. Account::Processor#calculate_total_balance summed every row in
raw_holdings_payload. Indexa returns a time series — one row per
security per date — so the naive sum double-counts (observed:
reported €91,633 became stored balance €180,039). Trust the API's
current_balance when present, and if we have to fall back to a
computed total, dedupe by instrument and take the latest-dated
amount per security.
* Fix IndexaCapital holdings reflecting oldest snapshot per security
HoldingsProcessor#process iterated every row in raw_holdings_payload.
Indexa returns a time series (many rows per security across dates),
and each iteration upserts the same (account, security, today) holding
row, so the LAST row processed wins. The payload is ordered with
newer dates first, so the last row processed is the OLDEST snapshot —
the holdings shown in the UI reflected tiny early positions instead
of the current ones (e.g. 3.8 shares of US 500 stored vs 62.34 actual).
Reduce the payload to one row per security (latest date) before
processing. The cost-basis update is now also driven by the latest
snapshot for the same reason.
* Fix IndexaCapital holdings using per-lot detail instead of totals
Importer#normalize_holdings_response read data[:fiscal_results], which
the Indexa API returns as per-tax-lot detail — many rows per security
covering each subscription_date, plus virtual sell/buy rows generated
by rebalances. Iterating it produced wildly wrong stored holdings:
e.g. 9.61 shares stored for Vanguard US 500 vs 62.34 actual; total
weights summed to ~10% instead of 100%.
The same response also includes data[:total_fiscal_results] — one
aggregated row per security with current titles/amount/cost matching
the Indexa UI and the user-downloadable positions CSV. Prefer it,
falling back to the per-lot field only when the totals are absent.
* Address CodeRabbit review on IndexaCapital fixes
Four review items, all fixed:
* Share instrument-key extraction
HoldingsProcessor#extract_ticker and Processor#calculate_holdings_value
used different fallback orders (one looked at :isin, the other at
:isin_code), so they could disagree on which rows referred to the same
security. Moved a single extract_instrument_key helper into
IndexaCapitalAccount::DataHelpers and routed both callers through it.
* Simplify Processor#calculate_holdings_value
The date-based dedupe was a workaround for the bug already fixed in
the importer (which now stores total_fiscal_results — one row per
security). Replaced the date comparison with a per-security map
populated via the shared key extractor. Same end result, fewer
moving parts, no fragile string-date comparison.
* Drop dead config key passed to create_account_from_indexa_capital
create_account_from_indexa_capital only reads :subtype and :balance
from its config arg. Passing :sync_start_date there was inert.
* Don't mark created accounts as skipped on post-create errors
In complete_account_setup, ensure_account_provider! and
update!(sync_start_date:) ran inside the same begin/rescue as the
Account.create!. If either raised after the Account row was already
persisted, control jumped to the rescue with created_count not yet
incremented and the account was wrongly counted as skipped. Now:
parse the form-supplied sync_start_date up front (a malformed value
is silently dropped instead of bubbling out of the loop), bump
created_count immediately after persisted?, and isolate the post-
create steps in their own rescue so failures there are logged but
don't desync the success counter.
* Fall back to /portfolio so pension plans get holdings imported
Indexa's /accounts/{id}/fiscal-results endpoint returns
{fiscal_results: [], total_fiscal_results: []} for pension plan
accounts (e.g. type "pension"). The same positions are exposed via
/accounts/{id}/portfolio in instrument_accounts[].positions[] for
both mutual funds and pensions, so use it as a fallback when
fiscal-results is empty.
The portfolio response uses the same field names HoldingsProcessor
already understands (instrument, titles, price, amount, cost_amount)
plus a derived cost_price (cost_amount / titles) added during
adaptation. No HoldingsProcessor changes needed.
Verified against the user-downloadable "Posiciones" CSV for an
SH71ZPMY pension account: two positions (N5138 Acciones, N5137
Bonos) and balance €8,273.56 match exactly.
* Fix CI: update tests for new IndexaCapital flow + rubocop blank line
* Lint: drop trailing blank line before `end` in
IndexaCapitalAccount::Processor (Layout/EmptyLinesAroundClassBody).
* Controller test: complete_account_setup#creates was posting
params: { accounts: { id => { account_type:, subtype: } } } against
the old controller schema. The new endpoint reads
params[:account_ids] and infers Investment for Indexa Capital, so
switch the test to that shape (and update the matching skip-already-
linked / no-selected-accounts cases).
* Processor test: "updates account balance from holdings value" set
current_balance: 38905.21 alongside holdings summing to 27093.01
and asserted the latter wins. After the fix
(calculate_total_balance prefers the API-reported current_balance
when present), the API value is the right answer. Renamed to
"trusts API current_balance over holdings sum when present" and
added a sibling test that nils current_balance to exercise the
holdings-sum fallback path explicitly (still asserts 27093.01).
* Wrap account creation+linking in a transaction to avoid orphans
complete_account_setup created the Account row first, incremented
created_count, and only then called ensure_account_provider! / the
sync_start_date update inside an inner rescue. If the link or the
sync_start_date update raised after the Account was already persisted,
control fell into the inner rescue: the orphaned Account row stayed
in the database, the failure was silently logged, and the success
counter was inflated.
Wrap creation, ensure_account_provider!, and the optional
sync_start_date update in a single ActiveRecord::Base.transaction.
Increment created_count only after the transaction commits; on any
exception the outer rescue rolls the whole step into skipped_count
with a clear log line tagged with the indexa_capital_account id.
|
||
|
Sophtron Rocky
|
b32e9dbc45 |
Add Sophtron Provider (#596)
* Add Sophtron Provider * fix syncer test issue * fix schema wrong merge * sync #588 * sync code for #588 * fixed a view issue * modified by comment * modified * modifed * modified * modified * fixed a schema issue * use global subtypes * add some locales * fix a safe_return_to_path * fix exposing raw exception messages issue * fix a merged issue * update schema.rb * fix a schema issue * fix some issue * Update bank sync controller to reflect beta status Signed-off-by: Juan José Mata <jjmata@jjmata.com> * Rename settings section title to 'Sophtron (alpha)' Signed-off-by: Juan José Mata <jjmata@jjmata.com> * Consistency in alpha/beta for Sophtron * Good PR suggestions from CodeRabbit --------- Signed-off-by: soky srm <sokysrm@gmail.com> Signed-off-by: Sophtron Rocky <rocky@sophtron.com> Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Signed-off-by: Juan José Mata <jjmata@jjmata.com> Co-authored-by: soky srm <sokysrm@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <jjmata@jjmata.com> |
||
|
Juan José Mata
|
7b2b1dd367 |
Rebase PR #784 and fix OpenAI model/chat regressions (#1384)
* Wire conversation history through OpenAI responses API * Fix RuboCop hash brace spacing in assistant tests * Pipelock ignores * Batch fixes --------- Co-authored-by: sokiee <sokysrm@gmail.com> |
||
|
soky srm
|
e40811b1ee |
Add improvements from security providers to FX providers also (#1445)
* FIX prefer provider rate always - add debugging also * Move logic from securities over * FIXes * Review fixes * Update provided.rb --------- Signed-off-by: soky srm <sokysrm@gmail.com> |
||
|
Romain Brucker
|
16a0fa08f8 |
Add DeFi via Coinstats (#1417)
* feat: handle defi account with coinstats provider * chore: refactor to follow project conventions * fix: fixing codex/coderabbit findings * fix: fixing coderabbit findings * fix: fixing coderabbit findings * fix: fixing coderabbit findings * fix: fixing coderabbit findings * fix: fixing coderabbit findings |