Files
sure/app/models/simplefin_account/processor.rb
LPW 78aa064bb0 Add overpayment detection for SimpleFIN liabilities (default ON) with heuristic-based classification and robust fallbacks (#412)
* Add liability balance normalization logic with comprehensive tests

- Updated `SimplefinAccount::Processor` to normalize liability balances based on observed values, ensuring correct handling of debts and overpayments.
- Enhanced `SimplefinItem::Importer` to apply similar normalization rules during imports, improving consistency.
- Added multiple test cases in `SimplefinAccountProcessorTest` to validate edge cases for liabilities and mixed-sign scenarios.
- Introduced helper methods (`to_decimal`, `same_sign?`) to simplify numeric operations in normalization logic.

* Add overpayment detection for liabilities with heuristic-based classification

- Introduced `SimplefinAccount::Liabilities::OverpaymentAnalyzer` to classify liability balances as credit, debt, or unknown using transaction history.
- Updated `SimplefinAccount::Processor` and `SimplefinItem::Importer` to integrate heuristic-based balance normalization with fallback logic for ambiguous cases.
- Added comprehensive unit tests in `OverpaymentAnalyzerTest` to validate classification logic and edge cases.
- Enhanced logging and observability around classification results and fallback scenarios.

* Refactor liability handling for better fallback consistency

- Updated `sticky_key` method in `OverpaymentAnalyzer` to handle missing `@sfa.id` with a default value.
- Enhanced `SimplefinAccount::Processor` to use `with_indifferent_access` for `raw_payload` and `org_data`, improving robustness in liability type inference.

* Extract numeric helper methods into `SimplefinNumericHelpers` concern and apply across models

- Moved `to_decimal` and `same_sign?` methods into a new `SimplefinNumericHelpers` concern for reuse.
- Updated `OverpaymentAnalyzer`, `Processor`, and `Importer` to include the concern and remove redundant method definitions.
- Added empty fixtures for `simplefin_accounts` and `simplefin_items` to ensure test isolation.
- Refactored `OverpaymentAnalyzerTest` to reduce fixture dependencies and ensure cleanup of created records.

* Refactor overpayment detection logic for clarity and fallback consistency

- Simplified `enabled?` method in `OverpaymentAnalyzer` for clearer precedence order (Setting > ENV > default).
- Added `parse_bool` helper to streamline boolean parsing.
- Enhanced error handling with detailed logging for transaction gathering failures.
- Improved `sticky_key` method to use a temporary object ID fallback when `@sfa.id` is missing.

---------

Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
2026-01-10 17:24:23 +01:00

193 lines
7.3 KiB
Ruby
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
class SimplefinAccount::Processor
include SimplefinNumericHelpers
attr_reader :simplefin_account
def initialize(simplefin_account)
@simplefin_account = simplefin_account
end
# Each step represents different SimpleFin data processing
# Processing the account is the first step and if it fails, we halt
# Each subsequent step can fail independently, but we continue processing
def process
# If the account is missing (e.g., user deleted the connection and relinked later),
# do not autolink. Relinking is now a manual, userconfirmed flow via the Relink modal.
unless simplefin_account.current_account.present?
return
end
process_account!
# Ensure provider link exists after processing the account/balance
begin
simplefin_account.ensure_account_provider!
rescue => e
Rails.logger.warn("SimpleFin provider link ensure failed for #{simplefin_account.id}: #{e.class} - #{e.message}")
end
process_transactions
process_investments
process_liabilities
end
private
def process_account!
# This should not happen in normal flow since accounts are created manually
# during setup, but keeping as safety check
if simplefin_account.current_account.blank?
Rails.logger.error("SimpleFin account #{simplefin_account.id} has no associated Account - this should not happen after manual setup")
return
end
# Update account balance and cash balance from latest SimpleFin data
account = simplefin_account.current_account
# Extract raw values from SimpleFIN snapshot
bal = to_decimal(simplefin_account.current_balance)
avail = to_decimal(simplefin_account.available_balance)
# Choose an observed value prioritizing posted balance first
observed = bal.nonzero? ? bal : avail
# Determine if this should be treated as a liability for normalization
is_linked_liability = [ "CreditCard", "Loan" ].include?(account.accountable_type)
raw = (simplefin_account.raw_payload || {}).with_indifferent_access
org = (simplefin_account.org_data || {}).with_indifferent_access
inferred = Simplefin::AccountTypeMapper.infer(
name: simplefin_account.name,
holdings: raw[:holdings],
extra: simplefin_account.extra,
balance: bal,
available_balance: avail,
institution: org[:name]
) rescue nil
is_mapper_liability = inferred && [ "CreditCard", "Loan" ].include?(inferred.accountable_type)
is_liability = is_linked_liability || is_mapper_liability
if is_mapper_liability && !is_linked_liability
Rails.logger.warn(
"SimpleFIN liability normalization: linked account #{account.id} type=#{account.accountable_type} " \
"appears to be liability via mapper (#{inferred.accountable_type}). Normalizing as liability; consider relinking."
)
end
balance = observed
if is_liability
# 1) Try transaction-history heuristic when enabled
begin
result = SimplefinAccount::Liabilities::OverpaymentAnalyzer
.new(simplefin_account, observed_balance: observed)
.call
case result.classification
when :credit
balance = -observed.abs
Rails.logger.info(
"SimpleFIN overpayment heuristic: classified as credit for sfa=#{simplefin_account.id}, " \
"observed=#{observed.to_s('F')} metrics=#{result.metrics.slice(:charges_total, :payments_total, :tx_count).inspect}"
)
Sentry.add_breadcrumb(Sentry::Breadcrumb.new(
category: "simplefin",
message: "liability_sign=credit",
data: { sfa_id: simplefin_account.id, observed: observed.to_s("F") }
)) rescue nil
when :debt
balance = observed.abs
Rails.logger.info(
"SimpleFIN overpayment heuristic: classified as debt for sfa=#{simplefin_account.id}, " \
"observed=#{observed.to_s('F')} metrics=#{result.metrics.slice(:charges_total, :payments_total, :tx_count).inspect}"
)
Sentry.add_breadcrumb(Sentry::Breadcrumb.new(
category: "simplefin",
message: "liability_sign=debt",
data: { sfa_id: simplefin_account.id, observed: observed.to_s("F") }
)) rescue nil
else
# 2) Fall back to existing sign-only logic (log unknown for observability)
begin
obs = {
reason: result.reason,
tx_count: result.metrics[:tx_count],
charges_total: result.metrics[:charges_total],
payments_total: result.metrics[:payments_total],
observed: observed.to_s("F")
}.compact
Rails.logger.info("SimpleFIN overpayment heuristic: unknown; falling back #{obs.inspect}")
rescue
# no-op
end
balance = normalize_liability_balance(observed, bal, avail)
end
rescue NameError
# Analyzer not loaded; keep legacy behavior
balance = normalize_liability_balance(observed, bal, avail)
rescue => e
Rails.logger.warn("SimpleFIN overpayment heuristic error for sfa=#{simplefin_account.id}: #{e.class} - #{e.message}")
balance = normalize_liability_balance(observed, bal, avail)
end
end
# Calculate cash balance correctly for investment accounts
cash_balance = if account.accountable_type == "Investment"
calculator = SimplefinAccount::Investments::BalanceCalculator.new(simplefin_account)
calculator.cash_balance
else
balance
end
account.update!(
balance: balance,
cash_balance: cash_balance,
currency: simplefin_account.currency
)
end
def process_transactions
SimplefinAccount::Transactions::Processor.new(simplefin_account).process
rescue => e
report_exception(e, "transactions")
end
def process_investments
return unless simplefin_account.current_account&.accountable_type == "Investment"
SimplefinAccount::Investments::TransactionsProcessor.new(simplefin_account).process
SimplefinAccount::Investments::HoldingsProcessor.new(simplefin_account).process
rescue => e
report_exception(e, "investments")
end
def process_liabilities
case simplefin_account.current_account&.accountable_type
when "CreditCard"
SimplefinAccount::Liabilities::CreditProcessor.new(simplefin_account).process
when "Loan"
SimplefinAccount::Liabilities::LoanProcessor.new(simplefin_account).process
end
rescue => e
report_exception(e, "liabilities")
end
def report_exception(error, context)
Sentry.capture_exception(error) do |scope|
scope.set_tags(
simplefin_account_id: simplefin_account.id,
context: context
)
end
end
# Helpers
# to_decimal and same_sign? provided by SimplefinNumericHelpers concern
def normalize_liability_balance(observed, bal, avail)
both_present = bal.nonzero? && avail.nonzero?
if both_present && same_sign?(bal, avail)
if bal.positive? && avail.positive?
return -observed.abs
elsif bal.negative? && avail.negative?
return observed.abs
end
end
-observed
end
end