* Prevent Yahoo crumb cache poisoning
* Make Yahoo Finance health status rate-limit aware
* Normalize Yahoo Colombian listings
Map Yahoo's BVC code to XBOG so search results derive Colombia from canonical exchange metadata and price requests use the .CL suffix with a COP fallback.
Spec: .scratch/normalize-yahoo-colombia-listings/spec.md
* Move Yahoo health checks to background
* Fix N+1 queries on categories index by batching lookups and removing partial fallbacks
* resolve Codex review suggestion about Guard subcategory against missing parents
* resolve coderabbitai review suggestion - Guard against empty categories when computing category_ids_with_transactions
* resolve coderabbitai review suggestion - Redundant pb-4 makes the conditional dead code
* resolve coderabbitai caution on PR review
* resolve sure-design review - DS Drift Patrol
* fix conflicting hidden/flex classes
* resolve jjmata review suggestion - schema.rb noise
* fix conflict and adjust related parts
* Ignore Brakeman EOLRails warning for Rails 7.2
Restore ignore entry lost during merge from main; documents upgrade
tracking and matches brakeman 7.1.2 in Gemfile.lock.
* db migration executed
* Remove deprecated focus-ring override from goals color picker.
The summary_class override was reintroduced during merge conflict
resolution and clobbered DS::Disclosure's canonical focus styling.
* fix merge conflict on disclosure.rb
* Restore parent-based semantics while keeping the performance win for root categories
* fix(ci): skip scheduled preview cleanup on forks
Only run the hourly Cloudflare preview cleanup on we-promise/sure,
where the required secrets exist.
* Revert schema.rb dump noise unrelated to categories N+1 fix
Restore db/schema.rb from main so PostgreSQL check-constraint and
column-order churn does not obscure the real PR changes.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Remove obsolete Rails 7.2 EOLRails Brakeman ignore
Main is already on Rails 8.1, so the 7.2.3.1 ignore entry is no longer needed.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Address review: bare disclosure docs, category picker aria label
* fix(test): wait for async exchange rate updates in system test
* fix(test): retry account edit submit around Turbo morph races
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(sync): allow EnableBanking credit card balances to be parsed as available credit
* feat(ui): expose Enable Banking balance interpretation toggle on credit card form
Adds a toggle to the credit card edit dialog, shown only when the account
is linked to an Enable Banking provider account. Toggling persists
treat_balance_as_available_credit and enqueues an item sync so the balance
is reinterpreted immediately.
* fix(sync): keep existing balance when credit limit is missing for available-credit cards
When treat_balance_as_available_credit is on and the API omits credit_limit,
the reported balance is known to be available credit, so recording it as
debt fabricates a liability. Skip the balance write in that case and only
update the available credit metadata.
Also extract the credit card branching into a helper and include
provider_key, account_provider and metadata in the debug log entries so
support can filter /settings/debug by the affected connection.
* refactor(ui): move provider lookup to Account and label the toggle for assistive tech
Adds Account#provider_account_for so the view no longer queries
account_providers directly, and wires aria-labelledby from the toggle to
its visible label.
* fix(ui): apply Enable Banking setting only after a successful account update
Runs the provider flag update after super and only on the redirect path,
so a failed account save no longer persists the flag or enqueues a sync.
Also permits the enable_banking params so malformed input is filtered
instead of raising.
* test(sync): extract relink helper in Enable Banking processor test
* feat(sync): fall back to manually set available credit as the credit limit
When the toggle is on, the accountable's available_credit field now holds
the credit limit (API-provided, or user-entered when the API omits it) and
is never overwritten with the reported balance. This lets users whose bank
reports available credit without a credit limit compute the outstanding
debt by entering their limit in the Available credit field, while keeping
the field stable across syncs so a stale reported balance can never be
mistaken for a limit.
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* feat(settings): super-admin background jobs console
Neither managed nor self-hosted production deployments have any view
into background job state (/sidekiq is only mounted outside
production), so a stuck sync, import, or export is invisible until a
user complains — and even then there is no way to act on it.
Adds /settings/background_jobs, gated by the same super-admin
Admin::BaseController as /settings/debug:
- Worker status header: Sidekiq processes, busy count, queue depths
and latency, retry/scheduled/dead set sizes. Read through a
fail-closed PORO (BackgroundJobConsole) — when Redis is unreachable
the console says so and disables actions instead of pretending
health (deliberate contrast to BackgroundJobHealth's fail-open).
- In-flight operations across all families: incomplete Syncs, Imports
importing/reverting, ImportSessions importing, FamilyExports
pending/processing, each with a liveness verdict derived from
Sidekiq::Workers (job GlobalIDs in running payloads).
- One mutation: mark a presumed-lost operation as such (Sync → stale,
Import → failed/revert_failed, PdfImport → claim released back to
pending, ImportSession/FamilyExport → failed).
Guard rails on the mutation, server-side re-checked:
- refused while Redis state is unknown (fail closed)
- refused while the record's job is visibly executing
- refused until the record has been idle past 30 minutes, so a merely
queued job cannot be shot down and then still run
- refused for parent Syncs with children still in flight (a parent
legitimately has no live job of its own while children run)
- applied inside with_lock with a status re-check, so a job finishing
between render and click wins
- audited as a DebugLogEntry (actor, prior status, family)
The cancel endpoint gets its own Rack Attack throttle since the
console deliberately lives under /settings rather than the throttled
/admin prefix (its 10 req/min limit would fight the page's polling).
* fix(jobs-console): count waiting jobs as live and harden cancel paths
Review feedback on #2682 (Codex, CodeRabbit):
- Liveness now covers jobs sitting in queues, the retry set, and the
scheduled set, not just visibly-executing workers — a job waiting out
a backlog or retry backoff WILL run later, and most affected job
classes don't abort on a flipped status, so cancelling invited
duplicate work. The backlog scan is bounded (5k entries); a truncated
scan fails closed like redis_error?. The liveness column shows
"Queued" for these
- find_record! resolves STI subclass names (TransactionImport,
PdfImport, …) against a base-class whitelist via safe_constantize
instead of a fixed name map, so non-UI callers naming the subclass
don't 404
- A PdfImport stuck in reverting goes to revert_failed like every other
import instead of being released to pending — pending presented a
possibly half-reverted import as publishable again; only the AI
extraction claim (importing) is released to pending
- Redis-unreachable warning renders via DS::Alert; operation id cast
to_s before splitting; admin-cancel error copy moved to i18n
* fix(jobs-console): re-check the stuck window inside the cancel row lock
CodeRabbit round-2: cancellable? evaluates Sidekiq liveness outside the
with_lock transaction (re-running Redis calls under a row lock would be
worse), so a worker picking the job up between the liveness check and
the lock acquisition was invisible. Repeating the updated_at staleness
check inside the lock closes that window — a freshly-started job
touches the record, and the re-check refuses the cancel.
* fix(jobs-console): resolve record_type without reflection, i18n nits
Brakeman flagged safe_constantize on params[:record_type] as a
High-confidence UnsafeReflection (ci/scan_ruby). Replace the
constant lookup with a reverse lookup: find the id in each
cancellable base table and require the claimed type to match the
found record's class or its base class. STI subclass names still
resolve; unknown types still 404.
Also from DS Drift Patrol:
- "Sync · " label in _operation.html.erb now goes through
t(".sync_label", type:)
- drop the redundant default: on background_jobs_label now that
the key exists in the locale file
* Add send_email_notification rule action
Adds a new rule action that emails a digest of transactions matching a
rule. Re-syncs re-apply every active rule to all in-window matches, so a
notification_deliveries table (unique on rule_id + transaction_id) backs
deduplication and a per-rule watermark:
- Rule::ActionExecutor::SendEmailNotification plucks candidate ids, drops
ones already in notification_deliveries, records the remainder BEFORE
enqueuing (fail-safe: a crash suppresses rather than double-sends), and
returns the count of newly-notified transactions.
- RuleEmailNotificationJob loads the rule + transactions and delivers the
digest via RuleNotificationMailer.
- Creating the action pre-seeds all currently-matching transactions as
already-delivered (after_create_commit), so the rule only emails about
transactions appearing after the action exists.
Dedup keys on the DB row id, not provider identity, so a re-ingested
transaction (new id) may re-notify; accepted as benign.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Add view transactions link to rule notification digest email
Include a "View transactions" CTA (APP_DOMAIN/transactions) in both the
HTML and text versions of the rule notification digest email.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Address review feedback on rule email notifications
- Restrict digest delivery to family admins only (drop non-admin fallback)
- Make NotificationDelivery.record_for return only inserted ids and enqueue
off that result, preventing duplicate digests under concurrent rule runs
- Seed the notification baseline when an existing action is changed to
send_email_notification, so historical matches are not emailed
- Strengthen tests: assert the transactions CTA/link in the digest and the
enqueued job's transaction-id args
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Include super_admin owner as rule digest recipient
The admin-only recipient lookup used find_by(role: :admin), which excluded
super_admin owners. A self-hosted family is commonly a single super_admin, so
the digest was silently skipped (NullMail no-op) and no email was sent.
Match the recipient on %w[admin super_admin] (the same pattern used elsewhere,
and consistent with User#admin?), while still excluding regular members/guests.
Add tests for the super_admin recipient and the no-admin skip path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Add mixed-role digest recipient test
Cover the case where a family has both an admin and a super_admin. The recipient
lookup uses find_by(role: %w[admin super_admin]) with no ORDER BY, so which one
is returned is non-deterministic; the contract is only that the recipient is an
admin-level user (never a member). Assert recipient.admin? rather than a brittle
precedence between the two roles.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Deliver rule digest via deliver_later and order in DB
Use deliver_later so a slow/flaky SMTP connection doesn't tie up the
Sidekiq worker, and push the entry-date sort into the query instead of
materializing and sorting the result set in Ruby.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Replace raw hex colors with email-safe design tokens in digest template
The rule digest email hardcoded Tailwind slate-100/200 hex values for
table borders, which aren't part of this project's design system.
Resolve to the actual border-primary/border-secondary token values and
centralize them as a reusable .email-table class in the mailer layout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(lunchflow): prune orphaned accounts deleted upstream (#1861)
Accounts deleted in Lunch Flow lingered in Sure as unlinked
LunchflowAccount records, permanently pinning the item to "Need setup".
The importer created/updated accounts returned by the API but never
removed records for accounts that disappeared upstream.
Add prune_orphaned_lunchflow_accounts, mirroring SimpleFin's
prune_orphaned_simplefin_accounts: after importing, delete LunchflowAccount
records whose account_id is no longer returned upstream and that are not
linked to an Account via AccountProvider. Linked accounts are kept so the
prune never cascade-destroys a user's Account. The prune is guarded to a
non-empty upstream list so a transient empty/failed response can't wipe out
all accounts.
Fixes#1861
* fix(lunchflow): prune NULL-id orphans + consistent import return shape (#1886)
* fix(lunchflow): make orphan prune resilient to destroy failures
Wrap the per-record destroy in begin/rescue so a single failed prune
doesn't abort the whole import. Pruning runs before transaction fetch,
so an unhandled error would have blocked transaction syncing entirely.
Matches the importer's existing continue-on-error convention.
* fix(lunchflow): only count pruned accounts when destroy succeeds
destroy returns false (without raising) when a before_destroy callback
halts deletion; the prior code incremented the pruned count regardless,
which could inflate it. Increment only on success and log when halted.
* fix(enable-banking): handle overdraft balance sync
* fix(enable-banking): skip nil provider balances
* test(enable-banking): cover nil provider balance
* fix(enable-banking): avoid stale and unsafe balance logs
* fix(enable-banking): guard unavailable balance writes
* test(enable-banking): stub unsaved account api id
* style(enable-banking): remove extra test blank line
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Add per-account toggle to disable automatic transaction categorization
Adds an `enable_category_matcher` boolean (default: true) to accounts so
users can opt out of Plaid's automatic category suggestions on a per-account
basis. When disabled, newly synced transactions arrive uncategorized so
rules or manual assignment take precedence.
- New migration adds `enable_category_matcher` column (default true, null: false)
- `PlaidEntry::Processor#matched_category` gates the CategoryMatcher call on the account flag
- Toggle rendered in the account edit modal for linked accounts (saves via main form submit)
- `AccountableResource#account_params` permits the new field
- `AccountsController#toggle_category_matcher` action added for potential API use
- i18n strings added for label and hint text
- Unit test covers the disabled-matcher path
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Only show category matcher toggle for Plaid-linked accounts
Only PlaidEntry::Processor honors enable_category_matcher, but the toggle
rendered for every linked account, silently doing nothing for other
providers. Adds Account::Linkable#supports_category_matcher? (covering both
the legacy plaid_account_id link and AccountProvider rows) and gates the
form toggle on it. The SimpleFIN TODO now also points at this helper so the
toggle appears once SimpleFIN matching lands.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add controller tests for category matcher toggle persistence and rendering
Covers the two paths flagged in review as untested: the flag persisting
through the shared AccountableResource#update action via account_params
(both disable and re-enable), and the edit form rendering the toggle only
for accounts where supports_category_matcher? is true.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Signed-off-by: Mike Lloyd <49411532+mike-lloyd03@users.noreply.github.com>
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* feat(sync): family-facing cancellation for syncs, imports, and exports
Users had no way to stop or recover any background operation: a
mistaken "Sync all" runs to completion, and an import or export whose
job died (hard worker kills lose in-flight Sidekiq jobs) wedges with a
spinner forever.
Sync cancellation (cooperative — nothing is ever killed):
- New syncs.cancel_requested_at column. Only the cancelled sync
carries the flag: pending descendants are marked stale immediately
(their queued jobs no-op via the existing may_start? guard), while
descendants whose jobs are already executing finish their work
honestly.
- Family::Syncer stops fanning out child syncs once the flag is set
(fresh read per iteration — the flag comes from the web process).
- Finalization resolves a cancel-requested sync to stale instead of
completed, which also skips post-sync (transfer matching, rules,
broadcasts) via the existing stale gate.
- The `visible` scope excludes cancel-requested syncs, so spinners
clear immediately and — fixing a latent bug this feature would have
amplified — sync_later no longer piggybacks a new sync request onto
a dying sync it would silently swallow.
- Cancel button appears next to "Sync all" on the accounts page while
a family sync is visible. SyncsController#cancel scopes through
Sync.for_family with resource_owner, so cross-family ids 404 and
account-level syncs respect per-user account access.
Stuck import/export self-service:
- Import#force_fail! / FamilyExport#force_fail!: allowed only once the
record has been idle past PRESUMED_LOST_AFTER (1 hour — dwarfs any
legitimate run), and applied inside with_lock with a status
re-check, so a job finishing between page render and button click
wins. Imports fail into the existing retry path (reverting ->
revert_failed keeps the revert retryable); PdfImports release their
processing claim back to pending; exports fail so a new one can be
created.
- "Mark as failed" buttons appear on the imports/exports index rows
only when a record is presumed lost, behind the pages' existing
permission gates (statement-import permission for imports, admin
for exports).
* fix(sync-cancel): cascade pending cancels, guard late finalizers, scope provider syncs
Review feedback on #2685 (CodeRabbit, Codex):
- request_cancel! now cascades finalization for pending syncs too: a
pending child resolved to stale never runs its job, so nothing else
would ever call finalize_if_all_children_finalized — its waiting
parent hung in syncing until the 24h sweep (CodeRabbit critical)
- SimplefinItem::Syncer#mark_completed re-reads the sync under a row
lock and skips finalization once cancellation was requested or the
row went terminal — its in-memory copy predates the cancel, and the
unguarded complete! (plus the raw status fallback) resurrected a
cancelled sync and re-ran post-sync (Codex)
- Cancelling provider-item syncs now requires admin: for_family's
resource_owner only scopes the Account branch, so a restricted member
could cancel admin-managed provider syncs spanning accounts they
cannot see. Family- and account-level syncs stay member-cancellable,
matching the buttons the UI shows (Codex)
- Lost-import error copy moved behind i18n
(imports.errors.presumed_lost), resolved at call time (CodeRabbit)
- Tests: pending-child cancel finalizes the parent; late provider
complete! cannot resurrect a cancelled sync; provider-sync
cancellation is admin-only
* fix(sync-cancel): capture the skipped-finalization case via DebugLogEntry
CodeRabbit round-2: the mark_completed skip (cancelled/terminal sync)
is support-relevant — record it in the super-admin debug UI with the
family and provider attached instead of a raw Rails.logger line.
category: provider_sync, matching the other provider syncers.
* fix(provider): resolve Tiingo security currency from countryCode
Tiingo's search API never returns the priceCurrency field the code was
reading, so currency detection silently failed. Currency is now derived
from countryCode via the countries gem's ISO 4217 data, with a
best-match tie-break for tickers that collide across countries so the
currency shown in search results always matches what fetch_security_prices
later returns.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(provider): guard tiingo currency cache against non-US downgrade
Addresses PR review feedback (jjmata): the per-ticker currency cache
written in search_securities was unconditionally overwritten on every
search, keyed only by ticker. A later search whose result set doesn't
happen to include a ticker's US cross-listing could silently downgrade
a previously-cached USD (the currency actually backing daily price
data) to a foreign currency. Only overwrite when the current search's
match is US, or nothing is cached yet.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
CoinbaseAccount::Processor uses the compact class form, so its lexical
nesting is only [CoinbaseAccount::Processor]. The bare `HoldingsProcessor`
reference in process_holdings resolved against that nesting and raised
`uninitialized constant CoinbaseAccount::Processor::HoldingsProcessor`
instead of finding CoinbaseAccount::HoldingsProcessor.
The error was swallowed by the rescue around process_holdings, so every
Coinbase sync logged the failure and skipped the holdings step while the
balance path still completed. Holdings and value never refreshed.
Qualify the reference to CoinbaseAccount::HoldingsProcessor so it resolves
absolutely, matching every sibling processor (Binance, Kraken, Snaptrade,
Questrade, Indexa).
Fixes#2412
Co-authored-by: agentloop <agentloop@localhost>
A sync marked stale by SyncCleanerJob while its job is still running
hits two lost-update paths when that job finishes:
- success path: finalize skipped the status transition but still ran
perform_post_sync, re-applying transfer matching, rules, and
broadcasts for a sync the system had already written off
- failure path: the rescue's unguarded fail! silently overwrote the
terminal stale status with failed (the in-memory record still read
syncing, so the AASM guard never fired)
Fix: re-check state under a row lock (with_lock reloads) before
failing, and skip post-sync for stale syncs in finalize. Post-sync
still runs for failed syncs — that behavior is intentional and
covered by existing tests.
* feat(wise): add Wise integration with JAR savings account and activity support
- Add WiseItem/WiseAccount models with full sync pipeline (importer, syncer, processor)
- Detect income vs expense using targetAccount == recipientId from borderless accounts API
- Support JAR (SAVINGS) accounts with totalWorth balance and savings subtype
- Fetch JAR activity via profile activities API (INTERBALANCE, BALANCE_CASHBACK, BALANCE_ASSET_FEE)
- Route INTERBALANCE activities to both JAR and STANDARD accounts and link as Transfer records
- Add provider connection status registration, routes, views, and i18n
- Add migration for wise_items and wise_accounts tables
- Add tests for WiseAccount, WiseEntry::Processor, WiseActivity::Processor, WiseItem::Importer, and WiseItem#link_jar_transfers!
* chore(lint): add ignore to security scan (false positive)
* fix(wise): address PR review feedback on activity routing, HTML stripping, rate limiting, and scope extraction
- Replace title string matching in activity_for_account? with resource.id vs balance_id comparison to avoid breakage when users rename JAR accounts on Wise
- Replace gsub(/<[^>]+>/, "") with ActionController::Base.helpers.strip_tags to safely handle user-controlled HTML-like content
- Wrap paginated API calls in with_rate_limit_retry (up to 3 attempts, exponential backoff) to handle 429 responses during fetch_jar_activities and fetch_transfers
- Extract WiseAccount.unlinked scope and remove duplicated left_joins query from controller
* fix(wise): address PR #2433 review feedback
- Wire @wise_items into AccountsController#index so linked accounts appear on /accounts
- Fix find_wise_account_for_linking to preserve .active scope via .merge instead of .then
- Fix cross-currency incoming transfer amount to use targetValue instead of sourceValue
- Add missing select_profiles.session_expired locale key
- Add missing account_name interpolation to link_existing_account success notice
- Use i18n for profile type labels in profile_display_name
- Trigger wise_item.sync_later after account linking in all three linking actions
- Isolate fee transaction failure so it no longer aborts the main transfer import
- Use bare raise in WiseActivity/WiseEntry processors to preserve original backtrace
- Re-raise in WiseItem::Unlinking to prevent silently orphaned Holdings
- Fix avatar badge to use design system tokens (bg-container-inset, text-primary)
* fix(wise): fix INTERBALANCE routing and encrypt pending token in session
* fix(wise): replace hand-rolled buttons/links with DS::Button and DS::Link
Address repeated sure-design DS drift findings: migrate all manual
Tailwind button_to and link_to calls in Wise views to DS::Button
(outline/outline_destructive variants) and DS::Link (secondary variant).
Add missing provider_panel.disconnect locale key for the disconnect button text.
* fix(wise): use render_provider_panel_error in create instead of missing new template
* fix(wise): add missing comma in PROVIDERS array
CI failed with a SyntaxError because the questrade entry wasn't
comma-terminated before the wise entry.
* chore(wise): re-add pipelock:ignore for pending token param
Lost when the token source moved from session to an encrypted params
field in 0b5dc886, causing the CI secret scanner to flag a false positive.
* fix(test): widen random ticker suffix to avoid rare collision flake
hex(2) only yields 65536 possible tickers, so create_trade's 4 calls per
test run had a small but nonzero chance of colliding on the unique
ticker+exchange index. hex(8) makes collisions practically impossible.
* feat(exchange-rates): add Frankfurter as an exchange-rate provider
Frankfurter (frankfurter.dev) is a free, keyless FX rates API backed by
ECB daily reference rates, with no published rate limit and no auth
flow to maintain (unlike Yahoo Finance's reverse-engineered cookie/
crumb auth or TwelveData's fast-exhausting free tier).
Follows the Provider::MoexPublic template: Faraday client with retry
middleware, SslConfigurable for self-hosted CA support, a light
RateLimitable throttle, and a FRANKFURTER_URL env escape hatch for
self-hosters. Registered as exchange-rates-only (no security/stock
data) and added to the hosting settings dropdown.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* refactor(provider): switch Frankfurter to the v2 API
v1 is explicitly marked "frozen" on Frankfurter's own root endpoint; v2
is "current" and covers 201 currencies across 84 central banks vs v1's
~30 ECB-only. Confirmed via the v2 OpenAPI spec and live requests:
- Single-date lookups now use GET /rate/{base}/{quote}?date=..., which
carries weekends/holidays forward server-side (a Saturday returns a
real rate directly), so the provider no longer needs its own
lookback-window logic.
- Range lookups now use GET /rates?base=..."es=...&from=...&to=...,
a flat array of { date, base, quote, rate } records (v2's shape)
instead of v1's { "rates": { date => currencies } } hash.
- Every calendar day in a range is present (v2 gapfills itself), rather
than v1's omit-non-trading-days behavior.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(provider): sanitize currency codes before URL path interpolation
from/to were only upcased before being interpolated directly into the
URL path in fetch_exchange_rate (GET /rate/{from}/{to}). Low risk since
currency codes come from validated internal sources, but adds cheap
defense-in-depth: strip anything that isn't A-Z, matching the ISO 4217
format real currency codes always take.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* feat(insights): proactive financial intelligence feed
Adds a nightly job that analyzes each family's finances in pure Ruby and
surfaces typed, stateful insights on the dashboard and a new /insights page.
- Insight model (active/read/dismissed) with a per-family dedup_key unique
index so nightly re-runs refresh rows instead of duplicating them
- Seven generators (spending anomaly, cash-flow warning, net worth milestone,
subscription audit, savings rate change, idle cash, budget health) built on
IncomeStatement, BalanceSheet, RecurringTransaction, and BudgetCategory
- LLM used as a writer, not a reasoner: Insight::BodyWriter narrates
pre-computed facts via the configured provider, with an i18n template
fallback so self-hosted installs without API keys work identically; bodies
are only (re)written when an insight is new or its numbers changed
- GenerateInsightsJob: cron fan-out per family, per-family advisory lock,
metadata-diff upsert that preserves read/dismissed state for unchanged
signals and reactivates on material change
- Dashboard insights_feed section (top 3, collapsible/reorderable) and an
/insights feed page with turbo-stream dismissal; viewing the feed marks
insights read
- Tests for the model, job upsert semantics, and controller flows
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6ZRA6cgCRM4UdFKct3wm4
* fix(insights): address Codex review — expiry, budget metadata, LLM usage
- Expire visible insights whose condition cleared: generators declare the
insight types they produce, the registry reports which generators ran to
completion, and the job expires visible insights of those types whose
dedup_key was not regenerated. A crashing generator can't wipe out its
healthy insights, and an expired insight reactivates when its condition
returns — unlike a user-dismissed one, which stays dismissed.
- Include a bucketed budget-spent percent in budget_at_risk metadata so the
body refreshes when overall usage moves >=10 points even if the same
categories remain flagged.
- Pass family to chat_response so LLM narration is recorded in llm_usages.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6ZRA6cgCRM4UdFKct3wm4
* fix(insights): address CodeRabbit review
- Skip the mark-as-read write for Turbo hover-prefetch requests
(X-Sec-Purpose) so unread badges don't clear before a real visit
- Show the New pill on the dashboard feed (active = unread there; the
feed never marks insights read)
- Filter idle accounts in SQL instead of a per-account exists? loop
- Eager-load merchants in the subscription audit query
- Widen the advisory-lock key to the signed-bigint range and log when
acquisition fails so a skipped nightly run is observable
- Mirror the dedup_key unique index as a model validation
- Assert the refresh action enqueues for the signed-in family
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6ZRA6cgCRM4UdFKct3wm4
* fix(insights): clear stale lifecycle timestamps on reactivation
When an insight resurfaces, the row now leaves no contradictory state
behind: the material-change path clears both read_at and dismissed_at,
and the expired-recovery path clears read_at. Tests assert the contract,
and the dashboard feed test now also locks in the unread "New" badge.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6ZRA6cgCRM4UdFKct3wm4
* fix(insights): drop redundant standalone family_id index
Every composite index on insights already leads with family_id, so the
auto-created single-column index was pure write overhead.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6ZRA6cgCRM4UdFKct3wm4
* fix(insights): stop nightly drift from resurrecting dismissed insights
Three generators stored continuously drifting values (projected amounts,
starting balance, current net worth) in metadata, and any metadata diff
counts as a material change: the body is rewritten (an LLM call when a
provider is configured) and read/dismissed state is cleared. Dismissing a
cash-flow warning was undone at the next 6:00 run.
Metadata now carries only the signal identity plus a coarse bucket, the
same damping the budget generators already use; display values live in
facts. Also orders the idle-cash pick by balance so the selected accounts
don't flip between runs, and documents the dollar-scale threshold
assumption on the relevant constants.
* fix(ds): map DS::Button aria_label into the aria hash
A bare aria_label: option reaches the tag helpers as a literal aria_label
attribute (Rails only dasherizes the nested aria: hash), so the icon-only
fallback overrode it and screen readers announced the insight dismiss
button and the popover trigger as "X".
* fix(insights): gate LLM narration behind AI consent
The nightly job runs unprompted for every family, so narration now
requires someone in the family to have AI enabled: consent to share
financial facts with the provider, and a cost cap in managed mode. The
template fallback keeps behavior identical otherwise. Narration failures
are captured via DebugLogEntry so support can see them.
Adds BodyWriter coverage, including a template-interpolation test for
every generator template key.
* feat(insights): rework the dashboard feed
The feed rendered full insight cards inside the section shell — the only
widget nesting card-on-card — and appended below the fold for every
family with a saved section order. It now mirrors the outflows and
balance-sheet list idiom (inset well with an uppercase mini-header, white
row block, 28px sentiment-tinted icon circles via color-mix on the DS
CSS variables, right-aligned key figures) and leads the dashboard for
saved orders that predate it.
Icon color comes from sentiment, not priority — a savings-rate
improvement is high priority AND good news, and must not render red; red
is reserved for a projected-negative balance. Rows have no hover wash
(cursor plus a gentle icon scale, like the sibling widgets), links to
/insights disable Turbo prefetch so the mark-as-read actually fires for
mouse users, and the standard widget-size popover offers Half/Full. Full
stays the default: the feed is far shorter than any other single-width
widget, so a half default leaves a grid hole the masonry cannot backfill.
* feat(insights): actionable cards, dismiss undo, live refresh
Each row now persists its display facts (new jsonb column) alongside the
change-detection metadata. Facts refresh every run without touching the
body or user state, which is exactly why they are not part of the
material-change comparison.
The card gains what the stored data always supported: a type-and-period
meta line replacing "x minutes ago", a right-aligned key figure (green
only for good news), and a contextual link resolved from the subject ids
in metadata — category, account, recurring transaction, budget month —
omitted when the subject no longer exists.
Dismiss is forgiving: a toast in the notification tray offers undo, and
undismissing restores the row as read rather than re-badging it.
Milestone insights can never regenerate once dismissed, so this closes a
real loss path. Upsert failures are captured via DebugLogEntry.
Manual refresh gets feedback: the button swaps to a disabled checking
state, the page subscribes to a family-scoped stream, and the job
broadcasts the refreshed list and the idle button when it finishes (also
after a lock-skipped run, so the button cannot stay stuck).
Savings copy is sign-aware: a negative rate reads "you spent more than
you earned" with true minus signs. The empty state swaps Lucide sparkles
for the brand assistant glyph (DS::EmptyState learns icon_custom:).
* feat(insights): top-bar entry with unread count
The dashboard feed hides at zero insights and nothing else linked to
/insights, so the manual refresh (and the empty state) were unreachable
for exactly the families who need them: fresh setups before the first
6:00 run. The sticky top bar travels to every screen and its right
cluster had room, so insights get an icon entry there with a monochrome
unread badge. Prefetch is disabled on the link for the same
mark-as-read reason as the feed.
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Guillem Arias <accounts@gariasf.com>
* fix: share family accounts with invited members on every sign-up path
A user who joined an existing family through an invitation was added to the
family but saw none of its accounts, even when the family's default sharing is
"share with all members". Only Invitation#accept_for created the AccountShare
records; the OIDC just-in-time sign-up, invite-token registration, and mobile
SSO onboarding paths all skipped it, so invitees landed in the family with an
empty account list. Signups routed into an invite-only default family had the
same gap.
Extract the sharing into Family#auto_share_existing_accounts_with, the single
entry point for "a member just joined, apply the family's sharing policy". It
honors default_account_sharing, only shares with a persisted member of the
family, grants read_only to guests and read_write to everyone else, excludes
accounts the user already owns, and is idempotent. accept_for and every
sign-up path call it, so no current or future join path can reintroduce the
empty-account bug or share accounts across families.
Also wrap the two SSO account-creation paths (OIDC JIT and mobile SSO) in a
transaction covering the user save, invitation acceptance, account sharing, and
identity creation, so a failure partway through can no longer leave a
half-onboarded user with no linked identity.
* address review: guest read_only symmetric in auto_share_with_family!, load-bearing guard comment, mobile SSO private-no-op test
Income trades created via Trade::CreateForm#create_income_trade set
price: 0 on the Trade record. This zero price was ingested by
PortfolioCache#load_prices as a valid price source, causing
ForwardCalculator#build_holdings to compute holding amount = qty * 0 = $0,
and overriding the security's market price on that date.
Additionally, Balance::BaseCalculator#flows_for_date misclassified income
trades (qty=0) as sells, producing spurious non_cash_outflows.
Fixes:
- PortfolioCache#load_prices: filter out zero-price trades from trade
price sources using .select { |t| t.entryable.price&.positive? }
- Balance::BaseCalculator#flows_for_date: separate income trades from
regular trades using their qty (income trades have qty=0), so they
contribute only to cash flows, not non-cash flows
- Balance::SyncCache#converted_entries: preserve the entryable association
target on duped entries so that e.entryable.qty access doesn't N+1
Fixes#2672
* Add native Questrade brokerage provider integration
Adds a per-family Questrade provider so users can sync their Questrade
investment accounts (TFSA, FHSA, RRSP, margin, etc.) directly via
Questrade's free personal API, with no paid aggregator.
- OAuth2 refresh-token flow with single-use token rotation, persisted
under a row lock. Tokens self-renew on each sync; the connected panel
lets users paste a fresh token if a connection goes stale (no need to
disconnect and re-link).
- Imports accounts, balances, positions and activities; multi-currency
holdings with per-currency cash holdings; Norbert's Gambit journals.
- New-account and link-existing-account flows, settings card with
desktop-only setup steps, and connect/update/disconnect.
- Restricted to Investment account types. Registered in the provider
connection-status registry with a syncable scope so it participates in
nightly family sync.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: linting error
* fix: refresh token encrypted
The OAuth token exchange passed the single-use refresh token as a GET
query parameter, so it could leak into URL-based logs (Sentry
breadcrumbs, APM spans, debug output). Switch to POST with a
form-encoded body (RFC 6749 3.2) so the credential stays out of URLs.
Verified Questrade's token endpoint accepts POST (returns 400 for a bad
token, not 405). Adds a test asserting the token travels in the body.
* Address PR review: authz, data integrity, retries, logging
Batch of fixes from the automated PR review:
- Require admin for all mutating/linking Questrade actions, and gate
existing-account linking through accessible_accounts + write permission
(was only Current.family scoped).
- Clear requires_update when a fresh token is accepted; use a real 302
redirect (not 422) on full-page failures.
- Require refresh_token on all saves (not just create) unless the item is
scheduled for deletion.
- Migrations target Rails 7.2; questrade_items state columns are NOT NULL.
- Background activity dedup keys on Questrade fields (matches the importer)
so multiple activities no longer collapse to one.
- Persist the normalized account payload; date-scope synthetic cash
holdings so daily history is not overwritten.
- Retry 429/5xx via a RetryableResponseError instead of hard-failing.
- Route provider error bodies to DebugLogEntry instead of Rails.logger /
exception messages, so payloads do not leak into application logs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Address PR review: atomic linking, sync health, retry loop, USD cash
- Wrap account creation + provider linking (+ sync_start_date) in a
transaction in both link paths so a link failure rolls back the orphan
account.
- Surface per-account process/schedule failures in the item sync health
instead of always reporting healthy.
- Always stamp last_activities_sync once the background fetch completes,
so legitimately empty accounts stop being re-queued every sync.
- Treat only the account-currency (CAD) balance as primary cash; other
currencies (e.g. USD) now surface as separate cash holdings instead of
being hidden as primary.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Address PR review: serialize token exchange, real processor tests
- Single-use token race: the SDK now wraps every token exchange (initial
and 401 re-auth) in a model-supplied lock that reloads and spends the
freshest persisted token (provided.rb#synchronize_exchange). Two
concurrent syncs/jobs can no longer double-spend the same refresh token.
Adds a test asserting the exchange runs inside the lock with the fresh
token.
- Replace the all-skipped QuestradeAccount processor test stubs with real
fixture-backed tests covering balance anchoring, holdings import, and
Buy-trade import (plus blank-symbol / blank-type guards).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Fix indentation of spliced Questrade schema blocks
The manually added questrade_accounts/questrade_items create_table blocks
sat at column 0 instead of the file 2-space indent, so rubocop flagged
them as inconsistent. Re-indent to match the rest of the schema.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Include currency and type in the Questrade activity merge key
Two activities that differ only by currency or type could collapse to a
single row in merge_activities. Add both fields to activity_key in the
importer and the background fetch job so multi-currency imports dedup
correctly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Address review: infer account currency, Encryptable, safer flag clear
From @jjmata's review:
- Currency: QuestradeAccount#upsert_from_questrade! no longer hardcodes CAD
for every account. upsert_balances! now infers the home currency from the
per-currency balances (the currency holding the cash wins, ties broken by
total equity, default CAD) so USD-denominated accounts are labelled USD and
match the right combinedBalances anchor. Adds tests for USD and CAD cases.
- QuestradeItem now includes the shared Encryptable concern instead of
reimplementing encryption_ready? inline.
- QuestradeActivitiesFetchJob#clear_pending_flag is now best-effort so it can
never mask (and swallow) the original error in perform's rescue.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Fix review issues: safe_return_to_path, DebugLogEntry, financial reset, turbo_prefetch, N+1 counts
- Add safe_return_to_path to QuestradeItemsController (blocks //evil.com
protocol-relative open redirect; same 3-check guard as PR #2591 Wise provider)
- Pass return_to through select_accounts and complete_account_setup so
users land back on the account they were linking from
- Replace Rails.logger.error/warn with DebugLogEntry.capture in controller
and unlinking concern (surface errors in the app debug log UI)
- Add questrade_items to Family::FinancialDataReset::PROVIDER_ITEM_ASSOCIATIONS
so Reset Financial Data actually removes Questrade data
- Add when "questrade" case to load_provider_items in providers_controller
so the settings panel lazy-load refresh works
- Fix turbo_prefetch: false on non-lunchflow provider links in
_method_selector.html.erb and select_provider.html.erb (prevents
prefetch-cache blank-modal bug for all generic sync providers)
- Preload questrade_accounts: :account_provider and build
@questrade_account_counts_map in AccountsController; read from map in
partial instead of calling .count on associations (eliminates N+1)
- Localize default connection name via I18n.t(questrade_items.default_name)
- Add default_name key to questrade_items locale
Patterns and bugs surfaced during review of PR #2591 (Wise provider).
* Cross-apply Wise learnings to Questrade provider
Encryption (matched convention from Wise/jjmata review):
- Add deterministic: true to QuestradeItem#refresh_token
- Add encrypts :raw_payload + :raw_institution_payload to QuestradeItem
- Add Encryptable + encrypts :raw_payload, :raw_holdings_payload,
:raw_activities_payload, :raw_balances_payload to QuestradeAccount
(brokerage-specific columns; matches MercuryAccount/UpAccount pattern)
Bug fix:
- Add missing RetryableResponseError class to Provider::Questrade
(used in with_retries rescue clause but never defined — would cause
NameError on any rate-limited or 5xx response)
Logging:
- Replace Rails.logger.error with DebugLogEntry.capture in
QuestradeItem#import_latest_questrade_data, #process_accounts,
and #schedule_account_syncs to surface errors in the support UI
Consistency:
- Extract update_sync_status(sync, key, **i18n_options) helper in
QuestradeItem::Syncer, replacing 5 inline sync.update! guard calls
- Use blank? instead of ||= for default name fallback in create action
Tests:
- Add QuestradeItemsControllerTest (18 tests: CRUD, sync, account
linking/setup flows, admin guard enforcement)
- Add questrade fixtures: questrade_items.yml, questrade_accounts.yml
- Add retry/backoff tests to Provider::QuestradeTest (network error,
429, 5xx — all verify MAX_RETRIES exhaustion raises Error)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Signed-off-by: Jestin Palamuttam <34907800+jestinjoshi@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(invitations): allow re-inviting after an unaccepted invite expires
An expired, never-accepted invitation still occupies the partial unique
index `index_invitations_on_email_and_family_id_pending` (predicate
`accepted_at IS NULL`), but the `pending` scope excludes it via the
`expires_at > now` clause. The duplicate-guard validation therefore
passes and the INSERT collides with the index, raising
ActiveRecord::RecordNotUnique — surfaced to the admin as a 500 with no
way to re-invite that address through the UI.
Remove the stale expired row inside the create transaction
(before_create) so the unique slot is freed and the re-invite succeeds.
It runs only after validations pass, so a still-pending (non-expired)
invitation can never be removed here. Add a controller-level rescue of
ActiveRecord::RecordNotUnique as a safety net against a concurrent
double-submit race.
Closes#2535
* refactor(invitations): scope RecordNotUnique rescue to the save
The method-level `rescue ActiveRecord::RecordNotUnique` wrapped the whole
create action, so any future write after @invitation.save (e.g.
accept_for) would silently be reported as a generic invite failure.
Extract save_invitation to scope the rescue to the save alone, and add a
regression test that drives the raced unique-index path directly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(sync): don't persist zero balances when a provider balance fetch fails
A nil current_balance means the sync's balance fetch did not succeed
(the snapshot upsert clears it; only a successful fetch repopulates it).
The Lunchflow and Enable Banking processors coerced nil to 0 and write
it (plus a currency fallback) onto the linked account, so any transient
provider failure persists wrong data with no user-visible signal.
Instead, treat nil as no-data: skip the account update. Also stop the
Lunchflow snapshot upsert resetting an established account's currency to
USD when the accounts endpoint omits currency.
* fix(lunchflow): normalize the preserved currency in snapshot upsert
Addresses the review comment on #2617: the preserved fallback reused the
record's raw in-memory currency, so a blank value would fail the presence
validation and break import, and an invalid code would persist instead of
falling back to USD. Run it through parse_currency like the payload value.
Regression test added.
* fix(lunchflow,eb): review round 2 — failure visibility and currency parity
Addresses the three review findings on #2617:
1. Capture the Lunch Flow balance-fetch failure via DebugLogEntry (the sync
otherwise reports success with no mention of the skip).
This is Lunch Flow only: Enable Banking's importer already surfaces the
failure through transactions_failed/@sync_error.
2. Apply the currency-preservation fix to Enable Banking's snapshot upsert
(same shape as the Lunch Flow fix: parity/safety). Regression test added.
3. Label the Enable Banking processor currency assertion as a parity check —
it also passes on main, since the reset defect was Lunch Flow-specific.
Trend#percent divided the delta by the signed previous value, so a negative
base inverted the sign of the result and contradicted #direction. This showed
an up arrow next to a negative percentage (and vice versa) for anything that
can go below zero — most visibly net worth in reports and balance sparklines.
Divide by the magnitude of the base instead, and carry the sign of current
when the base is zero so an all-negative move reports -Infinity rather than
+Infinity.
Fixes#2579
test "totals aggregate directly from trade entries" builds a month-to-date
period (beginning_of_month..Date.current) but places a trade at
start_date + 1.day. On the 1st of the month the period collapses to a single
day, so that trade falls outside the range and withdrawals aggregate to 0,
failing the assertion (expected 40, got 0). The test passes every other day.
Use a fixed multi-day period so the test is date-independent.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(up): flag internal transfers and round-ups as funds_movement
Up populates relationships.transferAccount on transactions that move money
between the user's own accounts (including round-ups swept into a Saver), but
flatten_transaction dropped it, so these imported as ordinary income/expense
and distorted budgets and cashflow.
- Provider::Up#flatten_transaction: lift transfer_account_id from
relationships.transferAccount.data.id.
- UpEntry::Processor: import transfers as funds_movement and persist
transfer_account_id in extra["up"].
- Account::ProviderImportAdapter#import_transaction: optional kind: param; an
explicit provider kind takes precedence over account-type auto-detection and
is applied after the sync-protection check, so user re-categorisations
survive re-sync.
Complementary to Family#auto_match_transfers!: two-sided transfers between
linked accounts are still paired into a Transfer (the matcher does not filter
on kind); one-sided movements and round-ups, which the matcher cannot pair,
are the cases this fixes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(up): account-type kind wins over provider transfer hint
Codex review caught that Up HOME_LOAN accounts map to a Loan account, so a
repayment carrying transferAccount would be reclassified from loan_payment to
funds_movement (budget-excluded). Make the provider kind: a fallback:
activity-label and account-type classification now take precedence, so
loan_payment and cc_payment survive. Adds a regression test (loan repayment
stays loan_payment), a depository-applies test, and a note that the up_test
stub ignores query: intentionally.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Gavin Matthews <matthews.gav@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(mercury): pending transactions, kind/counterpartyId metadata, and test coverage
- Transaction::PENDING_PROVIDERS: add "mercury" so the existing pending
reconciliation pipeline (pending→posted amount matching in
ProviderImportAdapter) activates for Mercury entries
- MercuryEntry::Processor: pass extra: to import_transaction with
extra["mercury"]["pending"] = true/false (status == "pending")
extra["mercury"]["kind"] = ACH / Wire / Card / etc.
extra["mercury"]["counterparty_id"] = Mercury counterparty UUID
Pending transactions are now imported with the flag set rather than
being silently ignored; failed transactions continue to be skipped
- Tests (new files, 30 cases):
- test/models/mercury_entry/processor_test.rb — sign convention,
date fallback, name priority, notes concat, pending flag, kind,
counterpartyId, failed skip, idempotency, merchant creation,
no-linked-account guard
- test/models/mercury_item/importer_test.rb — account discovery,
no-duplicate unlinked records, balance update on linked accounts,
transaction dedup (append-only new ids), sync window (90-day first
sync, last_synced_at-7d subsequent), 401 marks requires_update
- test/models/mercury_account/processor_test.rb — balance update,
CreditCard sign negation, cash_balance parity, no-linked-account
no-op, transaction processing delegation
* fix(mercury): address review findings — pending SQL, dedup upsert, N+1, nil assertion
- provider_import_adapter: add mercury to all three find_pending_transaction*
SQL predicates so Mercury pending entries are found and claimed when the
posted version arrives (exact, fuzzy, and low-confidence paths)
- mercury_item/importer: replace append-only dedup with an upsert-by-id
that replaces the stored raw payload when status changes from pending to
non-pending; prevents pending flag from persisting indefinitely when Mercury
reuses the same transaction ID for the posted version
- kraken_account/ledger_processor: preload all existing kraken ledger
external_ids into a Set before the loop; replaces per-entry exists? query
(N+1) with an in-memory Set#include? lookup
- test/models/mercury_account/processor_test: capture return value from
process and add assert_nil to enforce the nil contract stated in the test name
* fix(mercury): route transaction-count diagnostics through DebugLogEntry
* feat(kraken): fetch and import Ledgers API for deposits, withdrawals, staking, fees
Closes#2450
Kraken TradesHistory only returns spot buy/sell trades. The Ledgers API
(/0/private/Ledgers) covers deposits, withdrawals, staking rewards, Earn
income, and standalone fees — everything that was missing from syncs.
Changes:
- Provider::Kraken#get_ledgers — new method forwarding start/type/offset params
- KrakenItem::Importer#fetch_ledgers — paginated fetch (up to 200 pages) with
graceful fallback if the API key lacks Query Ledger Entries permission
- Importer#upsert_kraken_account — stores "ledgers" alongside "trades" in
raw_transactions_payload
- KrakenAccount::LedgerProcessor — new class; maps each supported ledger type
(deposit, withdrawal, staking, earn, fee) to a Transaction entry with the
correct investment_activity_label, kind, and sign convention; skips trade/
transfer/margin types to avoid double-counting with TradesHistory
- KrakenAccount::Processor#process — calls LedgerProcessor after process_trades
- Multi-currency: fiat amounts converted via ExchangeRate (non-USD fiat bridged
through USD); crypto amounts use the spot price cached in raw_payload["assets"]
with a price_missing flag when no price is available
- Dedup guard: external_id "kraken_ledger_<id>" + source "kraken" prevents
re-importing on repeated syncs
* fix(kraken): correct sign convention, fee inclusion, and earn subtype filtering
- Sign convention: deposits/staking/earn → negative (inflow), withdrawals/fees
→ positive (outflow), matching Sure's global convention (inflow is negative)
- Fee inclusion: use (amount - fee).abs as abs_impact so withdrawal fees are
counted in the total outflow rather than discarded
- Earn subtypes: skip allocation/deallocation ledger entries (internal fund
movements); only import rewardallocation/bonusallocation as Interest income
- DebugLogEntry: replace Rails.logger.warn/error with DebugLogEntry.capture
throughout LedgerProcessor and the Ledgers permission fallback in Importer,
so support-relevant incidents surface in /settings/debug
- Importer test: stub get_ledgers in setup so existing tests do not error
on the new fetch_ledgers call
* fix(kraken): route duplicate ledger-id warning through DebugLogEntry
* perf(kraken): batch ledger idempotency check; strengthen tests
Address review feedback (jjmata):
- N+1: LedgerProcessor#process_ledger_entry ran `account.entries.exists?(...)`
per ledger entry (up to ~10k per sync). Load the existing Kraken external IDs
once into a Set and test membership in memory (newly created IDs are added so
the same run stays idempotent) — same pattern as #2452.
- Tests: the idempotency test now asserts the first pass actually creates the
entry (assert_difference) before asserting the second is a no-op; add a guard
asserting the second (all-skipped) pass issues a single bulk external_id pluck,
not one query per entry.
No behavior change to imported entries.
* perf(kraken): scope ledger idempotency pluck to kraken_ledger_ prefix
Only load existing ledger external IDs (not trade entries) into the idempotency
Set, matching the reviewed approach. No behavior change.
* feat(goals): earmark a portion of an account toward a goal
Goals currently count each linked account's whole balance, so an account
shared across goals double-counts and one account can't fund several goals
in distinct slices. Add a per-account earmark — the "GoalBacking" the v1
model already foreshadowed (goal.rb).
- goal_accounts.allocated_amount (nullable). NULL = "dedicate the whole
balance" (the v1 default: no backfill, existing goals unchanged); a set
amount reserves a fixed slice.
- Goal#current_balance is now the single chokepoint computing each account's
backing under a family-wide shared pool: fixed earmarks take their slice,
an unallocated link takes the remainder, and when fixed earmarks exceed the
balance every slice is scaled down pro-rata so the goals' shares can never
sum past the account balance (no double-counting).
- Account#free_to_earmark / #goal_earmarked_total (mirror Budget's
available_to_allocate) back a soft, non-blocking over-allocation hint.
- GoalsController threads a goal[allocations] hash through create/update.
Phase 1 of the goals earmarking work; investment-backed goals follow.
* feat(goals): earmark UI on the goal form + backing-aware funding breakdown
- Goal form: a per-account "earmark amount" input (blank = whole balance)
next to each funding-account checkbox, prefilled from the saved
allocation on edit.
- Goal#account_backing exposes a single linked account's share so the
funding-accounts breakdown shows each account's earmarked contribution
and percent instead of its whole balance — keeping the show page
consistent with the (now allocation-aware) progress ring.
- English strings for the earmark controls and the "earmarked of balance"
breakdown line.
* fix(goals): address review on the earmark shared-pool math
- Overdrawn (<= 0 balance) accounts now back nothing on both the fixed and
whole-balance paths. The fixed path previously produced negative backing and
let a goal claim money the account doesn't hold.
- An archived goal reads its OWN earmark from its own goal_accounts instead of
the shared pool (which excludes archived goals), so it no longer mis-reports
the whole account balance for itself.
- goals#index injects one family-wide earmark pool into every card
(Goal.pooled_allocations_for) instead of querying once per goal (N+1), and
preloads goal_accounts.
- The projection chart scales its whole-account historical series by the
backing ratio so the saved line meets current_balance at "today" rather than
dropping off a cliff for earmarked goals.
- Honest comments: free_to_earmark no longer claims a form warning that doesn't
exist yet; pace documents its deliberate whole-account basis.
* fix(goals): widen the earmark input so the 'Whole balance' placeholder isn't clipped
* fix(goals): address review on #2490
- autosave: true on goal_accounts so earmark edits to already-linked accounts
persist through goal.save! (Rails only auto-saves newly built children, so
changing/clearing an existing earmark was silently dropped). + test.
- Reset the balance/progress memos on AASM transitions, not just the status
memos, so a same-instance render after complete!/archive! isn't stale. + test.
- backing_ratio is 0 (not 1) when the linked-account total is non-positive, so
the projection saved series ends at 0 to match the forced-zero current_balance.
- Localize the funding-row subtype label via goals.form.subtypes.*.
- Add the earmark strings to zh-CN (the maintained second locale; goals has no
ca locale, so Catalan keeps falling back to en like the rest of goals).
* feat(goals): investment-backed goals (Phase 2)
Goals can now be funded by investment accounts, not just depository.
- Relax linked_accounts_must_be_depository -> _must_be_fundable
(depository || investment); the funding picker + counts include investment
accounts.
- Add goals.progress_basis ('balance' | 'contributions', default 'balance').
Investment-backed goals default to 'contributions' so a market swing doesn't
move the goal: current_balance = value - cumulative market gain
(Sum of balances.net_market_flows); depository accounts have zero
net_market_flows, so they're unchanged. Goal#market_value_money shows what
it's worth today next to the contributed figure on the show page.
- Pledge false-match guard: investment accounts never use manual_save /
valuation-delta matching (a market move isn't a deposit) - they resolve on
transfer (cash-inflow) entries only. Guarded in both
Account#default_pledge_kind and GoalPledge#matches?.
- Add a `reopen` AASM event (completed -> active) + route/action/menu item so a
manually-completed goal whose value later dips can be reopened.
Stacked on the earmarking branch (#2490). Full suite green; +6 goal tests.
* fix(goals): address Phase 2 review — allocation-aware contributions, N+1, basis-on-update
- Contributions basis now goes through the same earmark/shared-pool logic as
the balance basis: backing_balance_for -> backing_share_for(account, base),
where base is the live balance (balance basis) or net contributions
(contributions basis). Earmarks are respected and shared accounts no longer
double-count on contributions goals; market_value_money stays consistent.
- Batch the per-account net_market_flows sum (Goal.market_flows_for) and inject
it on index like pooled_allocations, killing the N+1 for contributions goals.
- Default the basis on update too (not just create), so adding an investment
account to an existing depository goal flips it to contributions instead of
silently tracking market value.
- Fix the stale reconciliation_manager comment (renamed validation) and the
orphaned zh-CN must_be_depository key.
* fix(goals): address review on #2491
- before_save (not before_validation) for the progress_basis default, so a goal
can be inspected via valid? without its basis flipping as a side effect (jjmata).
- Pledge copy keys off default_pledge_kind, not manual?, so a manual investment
account — which pledges via transfer — shows the transfer prompt instead of the
"update your manual balance" flow (codex). pledge_action_label_key and the
pledge modal's per-account helper flag both use it.
- Add the Phase 2 strings (reopen success/invalid_transition, show.reopen,
ring.market_value) to zh-CN.
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* feat(goals): earmark a portion of an account toward a goal
Goals currently count each linked account's whole balance, so an account
shared across goals double-counts and one account can't fund several goals
in distinct slices. Add a per-account earmark — the "GoalBacking" the v1
model already foreshadowed (goal.rb).
- goal_accounts.allocated_amount (nullable). NULL = "dedicate the whole
balance" (the v1 default: no backfill, existing goals unchanged); a set
amount reserves a fixed slice.
- Goal#current_balance is now the single chokepoint computing each account's
backing under a family-wide shared pool: fixed earmarks take their slice,
an unallocated link takes the remainder, and when fixed earmarks exceed the
balance every slice is scaled down pro-rata so the goals' shares can never
sum past the account balance (no double-counting).
- Account#free_to_earmark / #goal_earmarked_total (mirror Budget's
available_to_allocate) back a soft, non-blocking over-allocation hint.
- GoalsController threads a goal[allocations] hash through create/update.
Phase 1 of the goals earmarking work; investment-backed goals follow.
* feat(goals): earmark UI on the goal form + backing-aware funding breakdown
- Goal form: a per-account "earmark amount" input (blank = whole balance)
next to each funding-account checkbox, prefilled from the saved
allocation on edit.
- Goal#account_backing exposes a single linked account's share so the
funding-accounts breakdown shows each account's earmarked contribution
and percent instead of its whole balance — keeping the show page
consistent with the (now allocation-aware) progress ring.
- English strings for the earmark controls and the "earmarked of balance"
breakdown line.
* fix(goals): address review on the earmark shared-pool math
- Overdrawn (<= 0 balance) accounts now back nothing on both the fixed and
whole-balance paths. The fixed path previously produced negative backing and
let a goal claim money the account doesn't hold.
- An archived goal reads its OWN earmark from its own goal_accounts instead of
the shared pool (which excludes archived goals), so it no longer mis-reports
the whole account balance for itself.
- goals#index injects one family-wide earmark pool into every card
(Goal.pooled_allocations_for) instead of querying once per goal (N+1), and
preloads goal_accounts.
- The projection chart scales its whole-account historical series by the
backing ratio so the saved line meets current_balance at "today" rather than
dropping off a cliff for earmarked goals.
- Honest comments: free_to_earmark no longer claims a form warning that doesn't
exist yet; pace documents its deliberate whole-account basis.
* fix(goals): widen the earmark input so the 'Whole balance' placeholder isn't clipped
* fix(goals): address review on #2490
- autosave: true on goal_accounts so earmark edits to already-linked accounts
persist through goal.save! (Rails only auto-saves newly built children, so
changing/clearing an existing earmark was silently dropped). + test.
- Reset the balance/progress memos on AASM transitions, not just the status
memos, so a same-instance render after complete!/archive! isn't stale. + test.
- backing_ratio is 0 (not 1) when the linked-account total is non-positive, so
the projection saved series ends at 0 to match the forced-zero current_balance.
- Localize the funding-row subtype label via goals.form.subtypes.*.
- Add the earmark strings to zh-CN (the maintained second locale; goals has no
ca locale, so Catalan keeps falling back to en like the rest of goals).
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
#2356 build the accountable inside Account#subtype= so a top-level subtype
survives create. That fix assumes accountable_type is already set when
subtype= runs, but the real controller path violates it: strong-params
permit preserves filter order, and account_params lists :subtype before
:accountable_type. So on create subtype= runs while accountable_type (and
accountable_class) is still blank, the build is skipped, and the chosen
subtype is silently dropped — the account renders with the type's fallback
label (e.g. a Depository shows 'Cash' instead of 'Savings').
The existing regression test only covered the accountable_type-first order,
so it never caught this.
Make the writer order-independent: when subtype arrives before the type,
stash it and apply it from an accountable_type= override once the type is
known. Add regression tests for the permit order and for create_and_sync.
- Show principal-only transfer amounts on both sides with separate fee and total lines (fixes inconsistent gross/net convention)
- Derive displayed fee amounts from fee_transactions entries (single source of truth) instead of stored columns
- Remove stored source_fee_amount/destination_fee_amount columns from transfers table
- Add foreign key for transactions.transfer_id -> transfers.id (replaces invalid CHECK subquery)
- Move destination fee line inside destination side div for consistent layout
- Remove orphaned view_fee_transaction locale keys from 7 locale files
- Rebuild schema.rb from origin/main to eliminate unrelated column reordering churn
Entries now hold principal only (no fee baked into amounts). Fee transactions created as standard kind with Fees category. Transfer#amount_abs returns principal from new amount column. Update handler recomputes entries and fee transactions on edit. Remove dead source_principal/destination_principal helpers. Schema regenerated cleanly with only transfer fee columns.
* Fix QIF import for dd mmm yyyy dates (#2498)
American Express QIF exports use dd mmm yyyy D-fields (e.g. D26 Jan 2026).
Import previously failed with "Unable to detect date format" for two reasons:
1. QifParser.normalize_qif_date stripped all internal whitespace, turning
26 Jan 2026 into the unparseable 26Jan2026. For month-name dates the
space IS the separator, so collapse multiples to a single space instead
of removing them. Numeric dates keep the existing strip-all behavior.
2. Family::DATE_FORMATS had no candidate mapping to dd mmm yyyy, so
detect_date_format could not match it. Add "%d %b %Y" (DD MMM YYYY).
Extend the 2-digit-year expansion separator class to include space so
month-name dates with 2-digit years also normalize (26 Jan 26 -> 26 Jan 2026).
Covered by normalize/parse/detect and Amex-style row-generation regression
tests in qif_import_test.rb.
* test(qif): cover month-name 2-digit year normalization & parse
Addresses CodeRabbit nitpick on #2500: the production change extends
the 2-digit-year expansion regex separator class to include a space
so month-name dates like "26 Jan 26" normalize to "26 Jan 2026".
Add normalize_qif_date and parse_qif_date regressions for that branch.
