* fix(simplefin): treat Vanguard/Fidelity cost_basis as total when needed
PR #1692 normalized SimpleFIN holdings cost_basis under the assumption
that the `cost_basis` / `basis` keys carry a per-share value (per the
SimpleFIN spec) and only `total_cost` / `value` carry a total position
cost. Vanguard and Fidelity violate the spec — they populate
`cost_basis` with the *total* (see the payload in #1182). After PR
#1692 those holdings get stored with cost_basis = total, and
Holding#calculate_trend then computes previous = qty × avg_cost, so the
"previous" value is inflated by a factor of qty and an entire
investment account renders a phantom return of roughly -(1 − 1/qty),
i.e. -97% to -99%.
Fix: sanity-check raw cost_basis against the holding's market share
price. Let share_price = market_value / qty; the geometric midpoint
between "raw is per-share" (raw ≈ share_price) and "raw is total"
(raw ≈ qty × share_price) is share_price × √qty. If raw is above the
midpoint it is divided by qty; otherwise it is kept as per-share.
Falls back to the pre-fix behaviour (trust the spec) when market_value
or qty is unavailable, so confidently-correct readings are never made
worse.
Verified against the reported Vanguard payload (qty=139, cost_basis=
22004.40, market_value=22626.42): normalize_cost_basis now returns
$158.31/share, matching 22004.40 / 139, and the phantom -99% return
collapses to a realistic ~+2.8%. Per-share readings ($45 cost on a $50
share price) remain untouched.
Closes#1718. Refs #1182, #1692.
* fixup: replace cost_basis heuristic with institution allowlist
Codex and @EdeAbreu23 flagged a real false-positive in the previous
geometric-midpoint heuristic: a legitimate per-share `cost_basis` on a
holding with a large unrealized loss (e.g. 100 shares with $100/share
basis now worth $5/share) trips `share_price × √qty` and gets divided
to $1/share — corrupting any standards-compliant brokerage with a big
loss.
Adopt @EdeAbreu23's safer shape:
- total_cost / value: always divide by qty (unchanged from #1692).
- cost_basis / basis: keep as-is by default.
- Only divide cost_basis / basis when the holding's SimpleFIN account
is connected to a known-misbehaving institution. Allowlist starts
with `vanguard` and `fidelity`, matched case-insensitively against
the account's stored org name and domain. Easy to extend as more
brokerages turn up.
Trades a small maintenance cost (curated list) for zero risk of
corrupting compliant providers.
Verified against five scenarios (all expected):
Vanguard total in cost_basis (allowlist) → +2.83%
Fidelity total in basis (allowlist) → +33.33%
Big-loss per-share (Codex case) → -95.0% (preserved)
Honest per-share, small loss → +11.11% (unchanged)
total_cost on any institution → +11.11% (unchanged)
---------
Co-authored-by: plind-junior <plind-junior@users.noreply.github.com>
* SimpleFIN: setup UX + same-provider relink + card-replacement detection
Fixes three bugs and adds auto-detection for credit-card fraud replacement.
Bugs:
- Importer: per-institution auth errors no longer flip the whole item to
requires_update. Partial errors stay on sync_stats so other institutions
keep syncing.
- Setup page: new activity badges (recent / dormant / empty / likely-closed)
via SimplefinAccount::ActivitySummary. Likely-closed (dormant + near-zero
balance + prior history) defaults to "skip" in the type picker.
- Relink: link_existing_account allows SimpleFIN to SimpleFIN swaps by
atomically detaching the old AccountProvider inside a transaction. Adds
"Change SimpleFIN account" menu item on linked-account dropdowns.
Feature (credit-card scope only):
- SimplefinItem::ReplacementDetector runs post-sync. Pairs a linked dormant
zero-balance sfa with an unlinked active sfa at the same institution and
account type. Persists suggestions on Sync#sync_stats.
- Inline banner on the SimpleFIN item card prompts relink via CustomConfirm.
Per-pair dismiss button scoped to the current sync (resurfaces on next
sync if still applicable). Auto-suppresses once the relink has landed.
Dev tooling:
- bin/rails simplefin:seed_fraud_scenario[email] creates a realistic broken
pair for manual QA; cleanup_fraud_scenario reverses it.
* Address review feedback on #1493
- ReplacementDetector: symmetric one-to-one matching. Two dormant cards
pointing at the same active card are now both skipped — previously the
detector could emit two suggestions that would clobber each other if
the user accepted both.
- ReplacementDetector: require non-blank institution names on both sides
before matching. Blank-vs-blank was accidentally treated as equal,
risking cross-provider false matches when SimpleFIN omitted org_data.
- ActivitySummary: fall back to "posted" when "transacted_at" is 0
(SimpleFIN's "unknown" sentinel). Integer 0 is truthy in Ruby, so the
previous `|| fallback` short-circuited and ignored posted.
- Controller: dismiss key is now the (dormant, active) pair so dismissing
one candidate for a dormant card doesn't suppress others.
- Helper test: freeze time around "6.hours.ago" and "5.days.ago"
assertions so they don't flake when the suite runs before 06:00.
* Address second review pass on #1493
- ReplacementDetector: canonicalize account_type in one place so filtering
(supported_type?) and matching (type_matches?) agree on "credit card"
vs "credit_card" variants.
- ReplacementDetector: skip candidates with nil current_balance. nil is
"unknown," not "zero" — previously fell back to 0 and passed the near-
zero gate, allowing suggestions without balance evidence.
* Add liability balance normalization logic with comprehensive tests
- Updated `SimplefinAccount::Processor` to normalize liability balances based on observed values, ensuring correct handling of debts and overpayments.
- Enhanced `SimplefinItem::Importer` to apply similar normalization rules during imports, improving consistency.
- Added multiple test cases in `SimplefinAccountProcessorTest` to validate edge cases for liabilities and mixed-sign scenarios.
- Introduced helper methods (`to_decimal`, `same_sign?`) to simplify numeric operations in normalization logic.
* Add overpayment detection for liabilities with heuristic-based classification
- Introduced `SimplefinAccount::Liabilities::OverpaymentAnalyzer` to classify liability balances as credit, debt, or unknown using transaction history.
- Updated `SimplefinAccount::Processor` and `SimplefinItem::Importer` to integrate heuristic-based balance normalization with fallback logic for ambiguous cases.
- Added comprehensive unit tests in `OverpaymentAnalyzerTest` to validate classification logic and edge cases.
- Enhanced logging and observability around classification results and fallback scenarios.
* Refactor liability handling for better fallback consistency
- Updated `sticky_key` method in `OverpaymentAnalyzer` to handle missing `@sfa.id` with a default value.
- Enhanced `SimplefinAccount::Processor` to use `with_indifferent_access` for `raw_payload` and `org_data`, improving robustness in liability type inference.
* Extract numeric helper methods into `SimplefinNumericHelpers` concern and apply across models
- Moved `to_decimal` and `same_sign?` methods into a new `SimplefinNumericHelpers` concern for reuse.
- Updated `OverpaymentAnalyzer`, `Processor`, and `Importer` to include the concern and remove redundant method definitions.
- Added empty fixtures for `simplefin_accounts` and `simplefin_items` to ensure test isolation.
- Refactored `OverpaymentAnalyzerTest` to reduce fixture dependencies and ensure cleanup of created records.
* Refactor overpayment detection logic for clarity and fallback consistency
- Simplified `enabled?` method in `OverpaymentAnalyzer` for clearer precedence order (Setting > ENV > default).
- Added `parse_bool` helper to streamline boolean parsing.
- Enhanced error handling with detailed logging for transaction gathering failures.
- Improved `sticky_key` method to use a temporary object ID fallback when `@sfa.id` is missing.
---------
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
* Add tests and update logic for processing SimpleFIN investment transactions
- Added `SimplefinAccount::Transactions::ProcessorInvestmentTest` to validate dividend transaction processing, transaction linking, and stale linkage repairs.
- Enhanced `SimplefinItem#process_accounts` with stale linkage repair logic and detailed logging for unlinked accounts with transactions.
- Updated `SimplefinAccount::Transactions::Processor` for improved logging and error handling during transaction processing.
- Adjusted `SimplefinItem::Importer` to log detailed account and transaction information and use extended sync windows for investment accounts.
* Refactor `SimplefinItem#process_accounts` to use direct queries for fresh data and streamline stale linkage repair logic; update tests for improved coverage and clarity.
* Improve stale linkage repair logic in `SimplefinItem#repair_stale_linkages`
- Updated to handle multiple linked accounts matching the same unlinked account by selecting the first match.
- Added detailed logging to warn about multiple matches for easier debugging.
* Include `:linked_account` in `SimplefinItem#process_accounts` queries for more comprehensive account data processing.
* Expand `merge_transactions` logic with composite key fallback for deduplication; document edge cases.
---------
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
