Files
sure/app/models/account/provider_import_adapter.rb
LPW 61eb611529 Simplefin enhancements v2 (#267)
* SimpleFin: metadata + merge fixes; holdings (incl. crypto) + Day Change; Sync Summary; ops rakes; lint

# Conflicts:
#	db/schema.rb

# Conflicts:
#	app/controllers/simplefin_items_controller.rb

* fix testing

* fix linting

* xfix linting x2

* Review PR #267 on we-promise/sure (SimpleFin enhancements v2). Address all 15 actionable CodeRabbit comments: Add UUID validations in rakes (e.g., simplefin_unlink), swap Ruby pattern matching/loops for efficient DB queries (e.g., where LOWER(name) LIKE ?), generate docstrings for low-coverage areas (31%), consolidate routes for simplefin_items, move view logic to helpers (e.g., format_transaction_extra), strengthen tests with exact assertions/fixtures for dedup/relink failures. Also, check for overlaps with merged #262 (merchants fix): Ensure merchant creation in simplefin_entry/processor.rb aligns with new payee-based flow and MD5 IDs; add tests for edge cases like empty payees or over-merging pendings. Prioritize security (PII redaction in logs, no hardcoded secrets).

* SimpleFin: address CodeRabbit comments (batch 1)

- Consolidate simplefin_items routes under a single resources block; keep URLs stable
- Replace inline JS with Stimulus auto-relink controller; auto-load relink modal via global modal frame
- Improve a11y in relink modal by wrapping rows in labels
- Harden unlink rake: default dry_run=true, UUID validation, redact PII in outputs, clearer errors
- Backfill rake: default dry_run=true, UUID validation; groundwork for per-SFA counters
- Fix-was-merged rake: default dry_run=true, UUID validation; clearer outputs
- Idempotent transfer auto-match (find_or_create_by! + RecordNotUnique rescue)
- Extract SimpleFin error tooltip assembly into helper and use it in view

RuboCop: maintain 2-space indentation, spaces inside array brackets, spaces after commas, and no redundant returns

* Linter noise

* removed filed commited by mistake.

* manual relink flow and tighten composite matching

* enforce manual relink UI; fix adapter keywords; guarantee extra.simplefin hash

* refactor(simplefin): extract relink service; enforce manual relink UI; tighten composite match; migration 7.2

* add provider date parser; refactor rake; move view queries; partial resilience

* run balances-only import in background job. make update flow enqueue balances-only job

* persists across all update redirects and initialize
used_manual_ids to prevent NameError in relink candidate computation.

* SimpleFin: metadata + merge fixes; holdings (incl. crypto) + Day Change; Sync Summary; ops rakes; lint

* Fixed failed test after rebase.

* scan_ruby fix

* Calming the rabbit:
Fix AccountProvider linking when accounts change
Drop the legacy unique index instead of duplicating it
Fix dynamic constant assignment
Use fixtures consistently; avoid rescue for control flow.
Replace bare rescue with explicit exception class.
Move business logic out of the view.
Critical: Transaction boundary excludes recompute phase, risking data loss.
Inconsistency between documentation and implementation for zero-error case.
Refactor to use the compute_unlinked_count helper for consistency.
Fix cleanup task default: it deletes by default.
Move sync stats computation to controller to avoid N+1 queries.
Consolidate duplicate sync query.
Clarify the intent of setting flash notice on the error path.
Fix Date/Time comparison in should_be_inactive?.
Move stats retrieval logic to controller.
Remove duplicate Sync summary section.
Remove the unnecessary sleep statement; use Capybara's built-in waiting.
Add label wrappers for accessibility and consistency.

* FIX SimpleFIN new account modal

Now new account properly loads as a Modal, instead of new page.
Fixes also form showing dashboard instead of settings page.

* Remove SimpleFin legacy UI components, migrate schema, and refine linking behavior.

# Conflicts:
#	app/helpers/settings_helper.rb

* Extract SimpleFin-related logic to `prepare_show_context` helper and refactor for consistency. Adjust conditional checks and ensure controller variables are properly initialized.

* Remove unused SimpleFin maps from prepare_show_context; select IDs to avoid N+1
Replace Tailwind bg-green-500 with semantic bg-success in _simplefin_panel/_provider_form
Add f.label :setup_token in simplefin_items/new for a11y
Remove duplicate require in AccountsControllerSimplefinCtaTest

* Remove unnecessary blank lines

* Reduce unnecessary changes

This reduces the diff against main

* Simplefin Account Setup: Display in modal

This fixes an issue with the `X` dismiss button in the top right corner

* Removed unnecessary comment.

* removed unnecessary function.

* fixed broken links

* Removed unnecessary file

* changed to database query

* set to use UTC and gaurd against null

* set dry_run=true

* Fixed comment

* Changed to use a database-level query

* matched test name to test behavior.

* Eliminate code duplication and Time.zone dependency

* make final summary surface failures

* lint fix

* Revised timezone comment. better handle missing selectors.

* sanitized LIKE wildcards

* Fixed SimpleFin import to avoid “Currency can’t be blank” validation failures when providers return an empty currency string.

* Added helper methods for admin and self-hosted checks

* Specify exception types in rescue clauses.

* Refined logic to determine transaction dates for credit accounts.

* Refined stats calculation for `total_accounts` to track the maximum unique accounts per run instead of accumulating totals.

* Moved `unlink_all!` logic to `SimplefinItem::Unlinking` concern and deprecated `SimplefinItem::Unlinker`. Updated related references.

* Refined legacy unlinking logic, improved `current_holdings` formatting, and added ENV-based overrides for self-hosted checks.

* Enhanced `unlink_all!` with explicit error handling, improved transaction safety, and refined ENV-based self-hosted checks. Adjusted exception types and cleaned up private method handling.

* Improved currency assignment logic by adding fallback to `current_account` and `family` currencies.

* Enhanced error tracking during SimpleFin account imports by adding categorized error buckets, limiting stored errors to the last 5, and improving `stats` calculations.

* typo fix

* Didn't realize rabbit was still mad...
Refactored SimpleFin error handling and CTA logic: centralized duplicate detection and relink visibility into controller, improved task counters, adjusted redirect notices, and fixed form indexing.

* Dang rabbit never stops... Centralized SimpleFin maps logic into `MapsHelper` concern and integrated it into relevant controllers and rake tasks. Optimized queries, reduced redundancy, and improved unlinked counts and manual account checks with batch processing. Adjusted task arguments for clarity.

* Persistent rabbit. Optimized SimpleFin maps logic by implementing batch queries for manual account and unlinked count checks, reducing N+1 issues. Improved clarity of rake task argument descriptions and error messages for better usability.

* Lost a commit somehow, resolved here. Refactored transaction extra details logic by introducing `build_transaction_extra_details` helper to improve clarity, reusability, and reduce view complexity. Enhanced rake tasks with strict dry-run validation and better error handling. Updated schema to allow nullable `merchant_id` and added conditional unique indexes for recurring transactions.

* Refactored sensitive data redaction in `simplefin_unlink` task for recursive handling, optimized SQL sanitization in `simplefin_holdings_backfill`, improved error handling in `transactions_helper`, and streamlined day change calculation logic in `Holding` model.

* Lint fix

* Removed per PR comments.

* Also removing per PR comment.

* git commit -m "SimpleFIN polish: preserve #manual-accounts wrapper, unify \"manual\" scope, and correct unlinked counts
- Preserve #manual-accounts wrapper: switch non-empty updates to turbo_stream.update and background broadcast_update_to; keep empty-path replace to render <div id=\"manual-accounts\"></div>
- Unify definition of manual accounts via Account.visible_manual (visible + legacy-nil + no AccountProvider); reuse in controllers, jobs, and helper
- Correct setup/unlinked counts: SimplefinItem::Syncer#finalize_setup_counts and maps now consider AccountProvider links (legacy account AND provider must be absent)
Deleted:
- app/models/simplefin_item/relink_service.rb
- app/controllers/concerns/simplefin_items/relink_helpers.rb
- app/javascript/controllers/auto_relink_controller.js
- app/views/simplefin_items/_relink_modal.html.erb
- app/views/simplefin_items/manual_relink.html.erb
- app/views/simplefin_items/relink.html.erb
- test/services/simplefin_item/relink_service_test.rb
Refs: PR #318 unified link/unlink; PR #267 SimpleFIN; follow-up to fix wrapper ID loss and counting drift."

* Extend unlinked account check to include "Investment" type

* set SimpleFIN item for `balances`, remove redundant unpacking, and improve holdings task error

* SimpleFIN: add `errors` action + modal; do not reintroduce legacy relink actions; removed dead helper

* FIX simpleFIN linking

* Add delay back, tests benefit from it

* Put cache back in

* Remove empty `rake` task

* Small spelling fixes.

---------

Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: sokie <sokysrm@gmail.com>
Co-authored-by: Dylan Corrales <deathcamel58@gmail.com>
2025-11-17 21:51:37 +01:00

320 lines
13 KiB
Ruby

class Account::ProviderImportAdapter
attr_reader :account
def initialize(account)
@account = account
end
# Imports a transaction from a provider
#
# @param external_id [String] Unique identifier from the provider (e.g., "plaid_12345", "simplefin_abc")
# @param amount [BigDecimal, Numeric] Transaction amount
# @param currency [String] Currency code (e.g., "USD")
# @param date [Date, String] Transaction date
# @param name [String] Transaction name/description
# @param source [String] Provider name (e.g., "plaid", "simplefin")
# @param category_id [Integer, nil] Optional category ID
# @param merchant [Merchant, nil] Optional merchant object
# @param notes [String, nil] Optional transaction notes/memo
# @param extra [Hash, nil] Optional provider-specific metadata to merge into transaction.extra
# @return [Entry] The created or updated entry
def import_transaction(external_id:, amount:, currency:, date:, name:, source:, category_id: nil, merchant: nil, notes: nil, extra: nil)
raise ArgumentError, "external_id is required" if external_id.blank?
raise ArgumentError, "source is required" if source.blank?
Account.transaction do
# Find or initialize by both external_id AND source
# This allows multiple providers to sync same account with separate entries
entry = account.entries.find_or_initialize_by(external_id: external_id, source: source) do |e|
e.entryable = Transaction.new
end
# If this is a new entry, check for potential duplicates from manual/CSV imports
# This handles the case where a user manually created or CSV imported a transaction
# before linking their account to a provider
# Note: We don't pass name here to allow matching even when provider formats names differently
if entry.new_record?
duplicate = find_duplicate_transaction(date: date, amount: amount, currency: currency)
if duplicate
# "Claim" the duplicate by updating its external_id and source
# This prevents future duplicate checks from matching it again
entry = duplicate
entry.assign_attributes(external_id: external_id, source: source)
end
end
# Validate entryable type matches to prevent external_id collisions
if entry.persisted? && !entry.entryable.is_a?(Transaction)
raise ArgumentError, "Entry with external_id '#{external_id}' already exists with different entryable type: #{entry.entryable_type}"
end
entry.assign_attributes(
amount: amount,
currency: currency,
date: date
)
# Use enrichment pattern to respect user overrides
entry.enrich_attribute(:name, name, source: source)
# Enrich transaction-specific attributes
if category_id
entry.transaction.enrich_attribute(:category_id, category_id, source: source)
end
if merchant
entry.transaction.enrich_attribute(:merchant_id, merchant.id, source: source)
end
if notes.present? && entry.respond_to?(:enrich_attribute)
entry.enrich_attribute(:notes, notes, source: source)
end
# Persist extra provider metadata on the transaction (non-enriched; always merged)
if extra.present? && entry.entryable.is_a?(Transaction)
existing = entry.transaction.extra || {}
incoming = extra.is_a?(Hash) ? extra.deep_stringify_keys : {}
entry.transaction.extra = existing.deep_merge(incoming)
end
entry.save!
entry
end
end
# Finds or creates a merchant from provider data
#
# @param provider_merchant_id [String] Provider's merchant ID
# @param name [String] Merchant name
# @param source [String] Provider name (e.g., "plaid", "simplefin")
# @param website_url [String, nil] Optional merchant website
# @param logo_url [String, nil] Optional merchant logo URL
# @return [ProviderMerchant, nil] The merchant object or nil if data is insufficient
def find_or_create_merchant(provider_merchant_id:, name:, source:, website_url: nil, logo_url: nil)
return nil unless provider_merchant_id.present? && name.present?
ProviderMerchant.find_or_create_by!(
provider_merchant_id: provider_merchant_id,
source: source
) do |m|
m.name = name
m.website_url = website_url
m.logo_url = logo_url
end
end
# Updates account balance from provider data
#
# @param balance [BigDecimal, Numeric] Total balance
# @param cash_balance [BigDecimal, Numeric] Cash balance (for investment accounts)
# @param source [String] Provider name (for logging/debugging)
def update_balance(balance:, cash_balance: nil, source: nil)
account.update!(
balance: balance,
cash_balance: cash_balance || balance
)
end
# Imports or updates a holding (investment position) from a provider
#
# @param security [Security] The security object
# @param quantity [BigDecimal, Numeric] Number of shares/units
# @param amount [BigDecimal, Numeric] Total value in account currency
# @param currency [String] Currency code
# @param date [Date, String] Holding date
# @param price [BigDecimal, Numeric, nil] Price per share (optional)
# @param cost_basis [BigDecimal, Numeric, nil] Cost basis (optional)
# @param external_id [String, nil] Provider's unique ID (optional, for deduplication)
# @param source [String] Provider name
# @param account_provider_id [String, nil] The AccountProvider ID that owns this holding (optional)
# @param delete_future_holdings [Boolean] Whether to delete holdings after this date (default: false)
# @return [Holding] The created or updated holding
def import_holding(security:, quantity:, amount:, currency:, date:, price: nil, cost_basis: nil, external_id: nil, source:, account_provider_id: nil, delete_future_holdings: false)
raise ArgumentError, "security is required" if security.nil?
raise ArgumentError, "source is required" if source.blank?
Account.transaction do
# Two strategies for finding/creating holdings:
# 1. By external_id (SimpleFin approach) - tracks each holding uniquely
# 2. By security+date+currency (Plaid approach) - overwrites holdings for same security/date
holding = if external_id.present?
account.holdings.find_or_initialize_by(external_id: external_id) do |h|
h.security = security
h.date = date
h.currency = currency
end
else
account.holdings.find_or_initialize_by(
security: security,
date: date,
currency: currency
)
end
holding.assign_attributes(
security: security,
date: date,
currency: currency,
qty: quantity,
price: price,
amount: amount,
cost_basis: cost_basis,
account_provider_id: account_provider_id
)
holding.save!
# Optionally delete future holdings for this security (Plaid behavior)
# Only delete if ALL providers allow deletion (cross-provider check)
if delete_future_holdings
unless account.can_delete_holdings?
Rails.logger.warn(
"Skipping future holdings deletion for account #{account.id} " \
"because not all providers allow deletion"
)
return holding
end
# Build base query for future holdings
future_holdings_query = account.holdings
.where(security: security)
.where("date > ?", date)
# If account_provider_id is provided, only delete holdings from this provider
# This prevents deleting positions imported by other providers
if account_provider_id.present?
future_holdings_query = future_holdings_query.where(account_provider_id: account_provider_id)
end
future_holdings_query.destroy_all
end
holding
end
end
# Imports a trade (investment transaction) from a provider
#
# @param security [Security] The security object
# @param quantity [BigDecimal, Numeric] Number of shares (negative for sells, positive for buys)
# @param price [BigDecimal, Numeric] Price per share
# @param amount [BigDecimal, Numeric] Total trade value
# @param currency [String] Currency code
# @param date [Date, String] Trade date
# @param name [String, nil] Optional custom name for the trade
# @param external_id [String, nil] Provider's unique ID (optional, for deduplication)
# @param source [String] Provider name
# @return [Entry] The created entry with trade
def import_trade(security:, quantity:, price:, amount:, currency:, date:, name: nil, external_id: nil, source:)
raise ArgumentError, "security is required" if security.nil?
raise ArgumentError, "source is required" if source.blank?
Account.transaction do
# Generate name if not provided
trade_name = if name.present?
name
else
trade_type = quantity.negative? ? "sell" : "buy"
Trade.build_name(trade_type, quantity, security.ticker)
end
# Use find_or_initialize_by with external_id if provided, otherwise create new
entry = if external_id.present?
# Find or initialize by both external_id AND source
# This allows multiple providers to sync same account with separate entries
account.entries.find_or_initialize_by(external_id: external_id, source: source) do |e|
e.entryable = Trade.new
end
else
account.entries.new(
entryable: Trade.new,
source: source
)
end
# Validate entryable type matches to prevent external_id collisions
if entry.persisted? && !entry.entryable.is_a?(Trade)
raise ArgumentError, "Entry with external_id '#{external_id}' already exists with different entryable type: #{entry.entryable_type}"
end
# Always update Trade attributes (works for both new and existing records)
entry.entryable.assign_attributes(
security: security,
qty: quantity,
price: price,
currency: currency
)
entry.assign_attributes(
date: date,
amount: amount,
currency: currency,
name: trade_name
)
entry.save!
entry
end
end
# Updates accountable-specific attributes (e.g., credit card details, loan details)
#
# @param attributes [Hash] Hash of attributes to update on the accountable
# @param source [String] Provider name (for logging/debugging)
# @return [Boolean] Whether the update was successful
def update_accountable_attributes(attributes:, source:)
return false unless account.accountable.present?
return false if attributes.blank?
# Filter out nil values and only update attributes that exist on the accountable
valid_attributes = attributes.compact.select do |key, _|
account.accountable.respond_to?("#{key}=")
end
return false if valid_attributes.empty?
account.accountable.update!(valid_attributes)
true
rescue => e
Rails.logger.error("Failed to update #{account.accountable_type} attributes from #{source}: #{e.message}")
false
end
# Finds a potential duplicate transaction from manual entry or CSV import
# Matches on date, amount, currency, and optionally name
# Only matches transactions without external_id (manual/CSV imported)
#
# @param date [Date, String] Transaction date
# @param amount [BigDecimal, Numeric] Transaction amount
# @param currency [String] Currency code
# @param name [String, nil] Optional transaction name for more accurate matching
# @param exclude_entry_ids [Set, Array, nil] Entry IDs to exclude from the search (e.g., already claimed entries)
# @return [Entry, nil] The duplicate entry or nil if not found
def find_duplicate_transaction(date:, amount:, currency:, name: nil, exclude_entry_ids: nil)
# Convert date to Date object if it's a string
date = Date.parse(date.to_s) unless date.is_a?(Date)
# Look for entries on the same account with:
# 1. Same date
# 2. Same amount (exact match)
# 3. Same currency
# 4. No external_id (manual/CSV imported transactions)
# 5. Entry type is Transaction (not Trade or Valuation)
# 6. Optionally same name (if name parameter is provided)
# 7. Not in the excluded IDs list (if provided)
query = account.entries
.where(entryable_type: "Transaction")
.where(date: date)
.where(amount: amount)
.where(currency: currency)
.where(external_id: nil)
# Add name filter if provided
query = query.where(name: name) if name.present?
# Exclude already claimed entries if provided
query = query.where.not(id: exclude_entry_ids) if exclude_entry_ids.present?
query.order(created_at: :asc).first
end
end