Files
sure/app/models/simplefin_account/investments/holdings_processor.rb
LPW c12c585a0e Harden SimpleFin sync: retries, safer imports, manual relinking, and data-quality reconciliation (#544)
* Add tests and enhance logic for SimpleFin account synchronization and reconciliation

- Added retry logic with exponential backoff for network errors in `Provider::Simplefin`.
- Introduced tests to verify retry functionality and error handling for rate-limit, server errors, and stale data.
- Updated `SimplefinItem` to detect stale sync status and reconciliation issues.
- Enhanced UI to display stale sync warnings and data integrity notices.
- Improved SimpleFin account matching during updates with multi-tier strategy (ID, fingerprint, fuzzy match).
- Added transaction reconciliation logic to detect data gaps, transaction count drops, and duplicate transaction IDs.

* Introduce `SimplefinConnectionUpdateJob` for asynchronous SimpleFin connection updates

- Moved SimpleFin connection update logic to `SimplefinConnectionUpdateJob` to improve response times by offloading network retries, data fetching, and reconciliation tasks.
- Enhanced SimpleFin account matching with a multi-tier strategy (ID, fingerprint, fuzzy name match).
- Added retry logic and bounded latency for token claim requests in `Provider::Simplefin`.
- Updated tests to cover the new job flow and ensure correct account reconciliation during updates.

* Remove unused SimpleFin account matching logic and improve error handling in `SimplefinConnectionUpdateJob`

- Deleted the multi-tier account matching logic from `SimplefinItemsController` as it is no longer used.
- Enhanced error handling in `SimplefinConnectionUpdateJob` to gracefully handle import failures, ensuring orphaned items can be manually resolved.
- Updated job flow to conditionally set item status based on the success of import operations.

* Fix SimpleFin sync: check both legacy FK and AccountProvider for linked accounts

* Add crypto, checking, savings, and cash account detection; refine subtype selection and linking

- Enhanced `Simplefin::AccountTypeMapper` to include detection for crypto, checking, savings, and standalone cash accounts.
- Improved subtype selection UI with validation and warning indicators for missing selections.
- Updated SimpleFin account linking to handle both legacy FK and `AccountProvider` associations consistently.
- Refined job flow and importer logic for better handling of linked accounts and subtype inference.

* Improve `SimplefinConnectionUpdateJob` and holdings processing logic

- Fixed race condition in `SimplefinConnectionUpdateJob` by moving `destroy_later` calls outside of transactions.
- Updated fuzzy name match logic to use Levenshtein distance for better accuracy.
- Enhanced synthetic ticker generation in holdings processor with hash suffix for uniqueness.

* Refine SimpleFin entry processing logic and ensure `extra` data persistence

- Simplified pending flag determination to rely solely on provider-supplied values.
- Fixed potential stale values in `extra` by ensuring deep merge overwrite with `entry.transaction.save!`.

* Replace hardcoded fallback transaction description with localized string

* Refine pending flag logic in SimpleFin processor tests

- Adjust test to prevent falsely inferring pending status from missing posted dates.
- Ensure provider explicitly sets pending flag for transactions.

* Add `has_many :holdings` association to `AccountProvider` with `dependent: :nullify`

---------

Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
2026-01-05 22:11:47 +01:00

187 lines
7.2 KiB
Ruby

class SimplefinAccount::Investments::HoldingsProcessor
def initialize(simplefin_account)
@simplefin_account = simplefin_account
end
def process
return if holdings_data.empty?
return unless [ "Investment", "Crypto" ].include?(account&.accountable_type)
holdings_data.each do |simplefin_holding|
begin
symbol = simplefin_holding["symbol"].presence
holding_id = simplefin_holding["id"]
description = simplefin_holding["description"].to_s.strip
Rails.logger.debug({ event: "simplefin.holding.start", sfa_id: simplefin_account.id, account_id: account&.id, id: holding_id, symbol: symbol, raw: simplefin_holding }.to_json)
unless holding_id.present?
Rails.logger.debug({ event: "simplefin.holding.skip", reason: "missing_id", id: holding_id, symbol: symbol }.to_json)
next
end
# If symbol is missing but we have a description, create a synthetic ticker
# This allows tracking holdings like 401k funds that don't have standard symbols
# Append a hash suffix to ensure uniqueness for similar descriptions
if symbol.blank? && description.present?
normalized = description.gsub(/[^a-zA-Z0-9]/, "_").upcase.truncate(24, omission: "")
hash_suffix = Digest::MD5.hexdigest(description)[0..4].upcase
symbol = "CUSTOM:#{normalized}_#{hash_suffix}"
Rails.logger.info("SimpleFin: using synthetic ticker #{symbol} for holding #{holding_id} (#{description})")
end
unless symbol.present?
Rails.logger.debug({ event: "simplefin.holding.skip", reason: "no_symbol_or_description", id: holding_id }.to_json)
next
end
security = resolve_security(symbol, simplefin_holding["description"])
unless security.present?
Rails.logger.debug({ event: "simplefin.holding.skip", reason: "unresolved_security", id: holding_id, symbol: symbol }.to_json)
next
end
# Parse provider data with robust fallbacks across SimpleFin sources
qty = parse_decimal(any_of(simplefin_holding, %w[shares quantity qty units]))
market_value = parse_decimal(any_of(simplefin_holding, %w[market_value value current_value]))
cost_basis = parse_decimal(any_of(simplefin_holding, %w[cost_basis basis total_cost]))
# Derive price from market_value when possible; otherwise fall back to any price field
fallback_price = parse_decimal(any_of(simplefin_holding, %w[purchase_price price unit_price average_cost avg_cost]))
price = if qty > 0 && market_value > 0
market_value / qty
else
fallback_price || 0
end
# Compute an amount we can persist (some providers omit market_value)
computed_amount = if market_value > 0
market_value
elsif qty > 0 && price > 0
qty * price
else
0
end
# Use best-known date: created -> updated_at -> as_of -> date -> today
holding_date = parse_holding_date(any_of(simplefin_holding, %w[created updated_at as_of date])) || Date.current
# Skip zero positions with no value to avoid invisible rows
next if qty.to_d.zero? && computed_amount.to_d.zero?
saved = import_adapter.import_holding(
security: security,
quantity: qty,
amount: computed_amount,
currency: simplefin_holding["currency"].presence || "USD",
date: holding_date,
price: price,
cost_basis: cost_basis,
external_id: "simplefin_#{holding_id}",
account_provider_id: simplefin_account.account_provider&.id,
source: "simplefin",
delete_future_holdings: false # SimpleFin tracks each holding uniquely
)
Rails.logger.debug({ event: "simplefin.holding.saved", account_id: account&.id, holding_id: saved.id, security_id: saved.security_id, qty: saved.qty.to_s, amount: saved.amount.to_s, currency: saved.currency, date: saved.date, external_id: saved.external_id }.to_json)
rescue => e
ctx = (defined?(symbol) && symbol.present?) ? " #{symbol}" : ""
Rails.logger.error "Error processing SimpleFin holding#{ctx}: #{e.message}"
end
end
end
private
attr_reader :simplefin_account
def import_adapter
@import_adapter ||= Account::ProviderImportAdapter.new(account)
end
def account
simplefin_account.current_account
end
def holdings_data
# Use the dedicated raw_holdings_payload field
simplefin_account.raw_holdings_payload || []
end
def resolve_security(symbol, description)
# Normalize crypto tickers to a distinct namespace so they don't collide with equities
sym = symbol.to_s.upcase
is_crypto_account = account&.accountable_type == "Crypto" || simplefin_account.name.to_s.downcase.include?("crypto")
is_crypto_symbol = %w[BTC ETH SOL DOGE LTC BCH].include?(sym)
mentions_crypto = description.to_s.downcase.include?("crypto")
if !sym.include?(":") && (is_crypto_account || is_crypto_symbol || mentions_crypto)
sym = "CRYPTO:#{sym}"
end
# Custom tickers (from holdings without symbols) should always be offline
is_custom = sym.start_with?("CUSTOM:")
# Use Security::Resolver to find or create the security, but be resilient
begin
if is_custom
# Skip resolver for custom tickers - create offline security directly
raise "Custom ticker - skipping resolver"
end
Security::Resolver.new(sym).resolve
rescue => e
# If provider search fails or any unexpected error occurs, fall back to an offline security
Rails.logger.warn "SimpleFin: resolver failed for symbol=#{sym}: #{e.class} - #{e.message}; falling back to offline security" unless is_custom
Security.find_or_initialize_by(ticker: sym).tap do |sec|
sec.offline = true if sec.respond_to?(:offline) && sec.offline != true
sec.name = description.presence if sec.name.blank? && description.present?
sec.save! if sec.changed?
end
end
end
def parse_holding_date(created_timestamp)
return nil unless created_timestamp
case created_timestamp
when Integer
Time.at(created_timestamp).to_date
when String
Date.parse(created_timestamp)
else
nil
end
rescue ArgumentError => e
Rails.logger.error "Failed to parse SimpleFin holding date #{created_timestamp}: #{e.message}"
nil
end
# Returns the first non-empty value for any of the provided keys in the given hash
def any_of(hash, keys)
return nil unless hash.respond_to?(:[])
Array(keys).each do |k|
# Support symbol or string keys
v = hash[k]
v = hash[k.to_s] if v.nil?
v = hash[k.to_sym] if v.nil?
return v if !v.nil? && v.to_s.strip != ""
end
nil
end
def parse_decimal(value)
return 0 unless value.present?
case value
when String
BigDecimal(value)
when Numeric
BigDecimal(value.to_s)
else
BigDecimal("0")
end
rescue ArgumentError => e
Rails.logger.error "Failed to parse SimpleFin decimal value #{value}: #{e.message}"
BigDecimal("0")
end
end