mirror of
https://github.com/we-promise/sure.git
synced 2026-04-19 12:04:08 +00:00
Add overpayment detection for SimpleFIN liabilities (default ON) with heuristic-based classification and robust fallbacks (#412)
* Add liability balance normalization logic with comprehensive tests - Updated `SimplefinAccount::Processor` to normalize liability balances based on observed values, ensuring correct handling of debts and overpayments. - Enhanced `SimplefinItem::Importer` to apply similar normalization rules during imports, improving consistency. - Added multiple test cases in `SimplefinAccountProcessorTest` to validate edge cases for liabilities and mixed-sign scenarios. - Introduced helper methods (`to_decimal`, `same_sign?`) to simplify numeric operations in normalization logic. * Add overpayment detection for liabilities with heuristic-based classification - Introduced `SimplefinAccount::Liabilities::OverpaymentAnalyzer` to classify liability balances as credit, debt, or unknown using transaction history. - Updated `SimplefinAccount::Processor` and `SimplefinItem::Importer` to integrate heuristic-based balance normalization with fallback logic for ambiguous cases. - Added comprehensive unit tests in `OverpaymentAnalyzerTest` to validate classification logic and edge cases. - Enhanced logging and observability around classification results and fallback scenarios. * Refactor liability handling for better fallback consistency - Updated `sticky_key` method in `OverpaymentAnalyzer` to handle missing `@sfa.id` with a default value. - Enhanced `SimplefinAccount::Processor` to use `with_indifferent_access` for `raw_payload` and `org_data`, improving robustness in liability type inference. * Extract numeric helper methods into `SimplefinNumericHelpers` concern and apply across models - Moved `to_decimal` and `same_sign?` methods into a new `SimplefinNumericHelpers` concern for reuse. - Updated `OverpaymentAnalyzer`, `Processor`, and `Importer` to include the concern and remove redundant method definitions. - Added empty fixtures for `simplefin_accounts` and `simplefin_items` to ensure test isolation. - Refactored `OverpaymentAnalyzerTest` to reduce fixture dependencies and ensure cleanup of created records. * Refactor overpayment detection logic for clarity and fallback consistency - Simplified `enabled?` method in `OverpaymentAnalyzer` for clearer precedence order (Setting > ENV > default). - Added `parse_bool` helper to streamline boolean parsing. - Enhanced error handling with detailed logging for transaction gathering failures. - Improved `sticky_key` method to use a temporary object ID fallback when `@sfa.id` is missing. --------- Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
This commit is contained in:
22
app/models/concerns/simplefin_numeric_helpers.rb
Normal file
22
app/models/concerns/simplefin_numeric_helpers.rb
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
module SimplefinNumericHelpers
|
||||||
|
extend ActiveSupport::Concern
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def to_decimal(value)
|
||||||
|
return BigDecimal("0") if value.nil?
|
||||||
|
case value
|
||||||
|
when BigDecimal then value
|
||||||
|
when String then BigDecimal(value) rescue BigDecimal("0")
|
||||||
|
when Numeric then BigDecimal(value.to_s)
|
||||||
|
else
|
||||||
|
BigDecimal("0")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def same_sign?(a, b)
|
||||||
|
(a.positive? && b.positive?) || (a.negative? && b.negative?)
|
||||||
|
end
|
||||||
|
end
|
||||||
225
app/models/simplefin_account/liabilities/overpayment_analyzer.rb
Normal file
225
app/models/simplefin_account/liabilities/overpayment_analyzer.rb
Normal file
@@ -0,0 +1,225 @@
|
|||||||
|
# frozen_string_literal: true
|
||||||
|
|
||||||
|
# Classifies a SimpleFIN liability balance as :debt (owe, show positive)
|
||||||
|
# or :credit (overpaid, show negative) using recent transaction history.
|
||||||
|
#
|
||||||
|
# Notes:
|
||||||
|
# - Preferred signal: already-imported Entry records for the linked Account
|
||||||
|
# (they are in Maybe's convention: expenses/charges > 0, payments < 0).
|
||||||
|
# - Fallback signal: provider raw transactions payload with amounts converted
|
||||||
|
# to Maybe convention by negating SimpleFIN's banking convention.
|
||||||
|
# - Returns :unknown when evidence is insufficient; callers should fallback
|
||||||
|
# to existing sign-only normalization.
|
||||||
|
class SimplefinAccount::Liabilities::OverpaymentAnalyzer
|
||||||
|
include SimplefinNumericHelpers
|
||||||
|
Result = Struct.new(:classification, :reason, :metrics, keyword_init: true)
|
||||||
|
|
||||||
|
DEFAULTS = {
|
||||||
|
window_days: 120,
|
||||||
|
min_txns: 10,
|
||||||
|
min_payments: 2,
|
||||||
|
epsilon_base: BigDecimal("0.50"),
|
||||||
|
statement_guard_days: 5,
|
||||||
|
sticky_days: 7
|
||||||
|
}.freeze
|
||||||
|
|
||||||
|
def initialize(simplefin_account, observed_balance:, now: Time.current)
|
||||||
|
@sfa = simplefin_account
|
||||||
|
@observed = to_decimal(observed_balance)
|
||||||
|
@now = now
|
||||||
|
end
|
||||||
|
|
||||||
|
def call
|
||||||
|
return unknown("flag disabled") unless enabled?
|
||||||
|
return unknown("no-account") unless (account = @sfa.current_account)
|
||||||
|
|
||||||
|
# Only applicable for liabilities
|
||||||
|
return unknown("not-liability") unless %w[CreditCard Loan].include?(account.accountable_type)
|
||||||
|
|
||||||
|
# Near-zero observed balances are too noisy to infer
|
||||||
|
return unknown("near-zero-balance") if @observed.abs <= epsilon_base
|
||||||
|
|
||||||
|
# Sticky cache via Rails.cache to avoid DB schema changes
|
||||||
|
sticky = read_sticky
|
||||||
|
if sticky && sticky[:expires_at] > @now
|
||||||
|
return Result.new(classification: sticky[:value].to_sym, reason: "sticky_hint", metrics: {})
|
||||||
|
end
|
||||||
|
|
||||||
|
txns = gather_transactions(account)
|
||||||
|
return unknown("insufficient-txns") if txns.size < min_txns
|
||||||
|
|
||||||
|
metrics = compute_metrics(txns)
|
||||||
|
cls, reason = classify(metrics)
|
||||||
|
|
||||||
|
if %i[credit debt].include?(cls)
|
||||||
|
write_sticky(cls)
|
||||||
|
end
|
||||||
|
|
||||||
|
Result.new(classification: cls, reason: reason, metrics: metrics)
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def enabled?
|
||||||
|
# Setting override takes precedence, then ENV, then default enabled
|
||||||
|
setting_val = Setting["simplefin_cc_overpayment_detection"]
|
||||||
|
return parse_bool(setting_val) unless setting_val.nil?
|
||||||
|
|
||||||
|
env_val = ENV["SIMPLEFIN_CC_OVERPAYMENT_HEURISTIC"]
|
||||||
|
return parse_bool(env_val) if env_val.present?
|
||||||
|
|
||||||
|
true # Default enabled
|
||||||
|
end
|
||||||
|
|
||||||
|
def parse_bool(value)
|
||||||
|
case value
|
||||||
|
when true, false then value
|
||||||
|
when String then %w[1 true yes on].include?(value.downcase)
|
||||||
|
else false
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def window_days
|
||||||
|
val = Setting["simplefin_cc_overpayment_window_days"]
|
||||||
|
v = (val.presence || DEFAULTS[:window_days]).to_i
|
||||||
|
v > 0 ? v : DEFAULTS[:window_days]
|
||||||
|
end
|
||||||
|
|
||||||
|
def min_txns
|
||||||
|
val = Setting["simplefin_cc_overpayment_min_txns"]
|
||||||
|
v = (val.presence || DEFAULTS[:min_txns]).to_i
|
||||||
|
v > 0 ? v : DEFAULTS[:min_txns]
|
||||||
|
end
|
||||||
|
|
||||||
|
def min_payments
|
||||||
|
val = Setting["simplefin_cc_overpayment_min_payments"]
|
||||||
|
v = (val.presence || DEFAULTS[:min_payments]).to_i
|
||||||
|
v > 0 ? v : DEFAULTS[:min_payments]
|
||||||
|
end
|
||||||
|
|
||||||
|
def epsilon_base
|
||||||
|
val = Setting["simplefin_cc_overpayment_epsilon_base"]
|
||||||
|
d = to_decimal(val.presence || DEFAULTS[:epsilon_base])
|
||||||
|
d > 0 ? d : DEFAULTS[:epsilon_base]
|
||||||
|
end
|
||||||
|
|
||||||
|
def statement_guard_days
|
||||||
|
val = Setting["simplefin_cc_overpayment_statement_guard_days"]
|
||||||
|
v = (val.presence || DEFAULTS[:statement_guard_days]).to_i
|
||||||
|
v >= 0 ? v : DEFAULTS[:statement_guard_days]
|
||||||
|
end
|
||||||
|
|
||||||
|
def sticky_days
|
||||||
|
val = Setting["simplefin_cc_overpayment_sticky_days"]
|
||||||
|
v = (val.presence || DEFAULTS[:sticky_days]).to_i
|
||||||
|
v > 0 ? v : DEFAULTS[:sticky_days]
|
||||||
|
end
|
||||||
|
|
||||||
|
def gather_transactions(account)
|
||||||
|
start_date = (@now.to_date - window_days.days)
|
||||||
|
|
||||||
|
# Prefer materialized entries
|
||||||
|
entries = account.entries.where("date >= ?", start_date).select(:amount, :date)
|
||||||
|
txns = entries.map { |e| { amount: to_decimal(e.amount), date: e.date } }
|
||||||
|
return txns if txns.size >= min_txns
|
||||||
|
|
||||||
|
# Fallback: provider raw payload
|
||||||
|
raw = Array(@sfa.raw_transactions_payload)
|
||||||
|
raw_txns = raw.filter_map do |tx|
|
||||||
|
h = tx.with_indifferent_access
|
||||||
|
amt = convert_provider_amount(h[:amount])
|
||||||
|
d = (
|
||||||
|
Simplefin::DateUtils.parse_provider_date(h[:posted]) ||
|
||||||
|
Simplefin::DateUtils.parse_provider_date(h[:transacted_at])
|
||||||
|
)
|
||||||
|
next nil unless d
|
||||||
|
next nil if d < start_date
|
||||||
|
{ amount: amt, date: d }
|
||||||
|
end
|
||||||
|
raw_txns
|
||||||
|
rescue => e
|
||||||
|
Rails.logger.debug("SimpleFIN transaction gathering failed for sfa=#{@sfa.id}: #{e.class} - #{e.message}")
|
||||||
|
[]
|
||||||
|
end
|
||||||
|
|
||||||
|
def compute_metrics(txns)
|
||||||
|
charges = BigDecimal("0")
|
||||||
|
payments = BigDecimal("0")
|
||||||
|
payments_count = 0
|
||||||
|
recent_payment = false
|
||||||
|
guard_since = (@now.to_date - statement_guard_days.days)
|
||||||
|
|
||||||
|
txns.each do |t|
|
||||||
|
amt = to_decimal(t[:amount])
|
||||||
|
if amt.positive?
|
||||||
|
charges += amt
|
||||||
|
elsif amt.negative?
|
||||||
|
payments += -amt
|
||||||
|
payments_count += 1
|
||||||
|
recent_payment ||= (t[:date] >= guard_since)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
net = charges - payments
|
||||||
|
{
|
||||||
|
charges_total: charges,
|
||||||
|
payments_total: payments,
|
||||||
|
payments_count: payments_count,
|
||||||
|
tx_count: txns.size,
|
||||||
|
net: net,
|
||||||
|
observed: @observed,
|
||||||
|
window_days: window_days,
|
||||||
|
recent_payment: recent_payment
|
||||||
|
}
|
||||||
|
end
|
||||||
|
|
||||||
|
def classify(m)
|
||||||
|
# Boundary guard: a single very recent payment may create temporary credit before charges post
|
||||||
|
if m[:recent_payment] && m[:payments_count] <= 2
|
||||||
|
return [ :unknown, "statement-guard" ]
|
||||||
|
end
|
||||||
|
|
||||||
|
eps = [ epsilon_base, (@observed.abs * BigDecimal("0.005")) ].max
|
||||||
|
|
||||||
|
# Overpayment (credit): payments exceed charges by at least the observed balance (within eps)
|
||||||
|
if (m[:payments_total] - m[:charges_total]) >= (@observed.abs - eps)
|
||||||
|
return [ :credit, "payments>=charges+observed-eps" ]
|
||||||
|
end
|
||||||
|
|
||||||
|
# Debt: charges exceed payments beyond epsilon
|
||||||
|
if (m[:charges_total] - m[:payments_total]) > eps && m[:payments_count] >= min_payments
|
||||||
|
return [ :debt, "charges>payments+eps" ]
|
||||||
|
end
|
||||||
|
|
||||||
|
[ :unknown, "ambiguous" ]
|
||||||
|
end
|
||||||
|
|
||||||
|
def convert_provider_amount(val)
|
||||||
|
amt = case val
|
||||||
|
when String then BigDecimal(val) rescue BigDecimal("0")
|
||||||
|
when Numeric then BigDecimal(val.to_s)
|
||||||
|
else BigDecimal("0")
|
||||||
|
end
|
||||||
|
# Negate to convert banking convention (expenses negative) -> Maybe convention
|
||||||
|
-amt
|
||||||
|
end
|
||||||
|
|
||||||
|
def read_sticky
|
||||||
|
Rails.cache.read(sticky_key)
|
||||||
|
end
|
||||||
|
|
||||||
|
def write_sticky(value)
|
||||||
|
Rails.cache.write(sticky_key, { value: value.to_s, expires_at: @now + sticky_days.days }, expires_in: sticky_days.days)
|
||||||
|
end
|
||||||
|
|
||||||
|
def sticky_key
|
||||||
|
id = @sfa.id || "tmp:#{@sfa.object_id}"
|
||||||
|
"simplefin:sfa:#{id}:liability_sign_hint"
|
||||||
|
end
|
||||||
|
|
||||||
|
# numeric coercion handled by SimplefinNumericHelpers#to_decimal
|
||||||
|
|
||||||
|
def unknown(reason)
|
||||||
|
Result.new(classification: :unknown, reason: reason, metrics: {})
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -1,4 +1,5 @@
|
|||||||
class SimplefinAccount::Processor
|
class SimplefinAccount::Processor
|
||||||
|
include SimplefinNumericHelpers
|
||||||
attr_reader :simplefin_account
|
attr_reader :simplefin_account
|
||||||
|
|
||||||
def initialize(simplefin_account)
|
def initialize(simplefin_account)
|
||||||
@@ -39,15 +40,90 @@ class SimplefinAccount::Processor
|
|||||||
|
|
||||||
# Update account balance and cash balance from latest SimpleFin data
|
# Update account balance and cash balance from latest SimpleFin data
|
||||||
account = simplefin_account.current_account
|
account = simplefin_account.current_account
|
||||||
balance = simplefin_account.current_balance || simplefin_account.available_balance || 0
|
|
||||||
|
|
||||||
# Normalize balances for liabilities (SimpleFIN typically uses opposite sign)
|
# Extract raw values from SimpleFIN snapshot
|
||||||
# App convention:
|
bal = to_decimal(simplefin_account.current_balance)
|
||||||
# - Liabilities: positive => you owe; negative => provider owes you (overpayment/credit)
|
avail = to_decimal(simplefin_account.available_balance)
|
||||||
# Since providers often send the opposite sign, ALWAYS invert for liabilities so
|
|
||||||
# that both debt and overpayment cases are represented correctly.
|
# Choose an observed value prioritizing posted balance first
|
||||||
if [ "CreditCard", "Loan" ].include?(account.accountable_type)
|
observed = bal.nonzero? ? bal : avail
|
||||||
balance = -balance
|
|
||||||
|
# Determine if this should be treated as a liability for normalization
|
||||||
|
is_linked_liability = [ "CreditCard", "Loan" ].include?(account.accountable_type)
|
||||||
|
raw = (simplefin_account.raw_payload || {}).with_indifferent_access
|
||||||
|
org = (simplefin_account.org_data || {}).with_indifferent_access
|
||||||
|
inferred = Simplefin::AccountTypeMapper.infer(
|
||||||
|
name: simplefin_account.name,
|
||||||
|
holdings: raw[:holdings],
|
||||||
|
extra: simplefin_account.extra,
|
||||||
|
balance: bal,
|
||||||
|
available_balance: avail,
|
||||||
|
institution: org[:name]
|
||||||
|
) rescue nil
|
||||||
|
is_mapper_liability = inferred && [ "CreditCard", "Loan" ].include?(inferred.accountable_type)
|
||||||
|
is_liability = is_linked_liability || is_mapper_liability
|
||||||
|
|
||||||
|
if is_mapper_liability && !is_linked_liability
|
||||||
|
Rails.logger.warn(
|
||||||
|
"SimpleFIN liability normalization: linked account #{account.id} type=#{account.accountable_type} " \
|
||||||
|
"appears to be liability via mapper (#{inferred.accountable_type}). Normalizing as liability; consider relinking."
|
||||||
|
)
|
||||||
|
end
|
||||||
|
|
||||||
|
balance = observed
|
||||||
|
if is_liability
|
||||||
|
# 1) Try transaction-history heuristic when enabled
|
||||||
|
begin
|
||||||
|
result = SimplefinAccount::Liabilities::OverpaymentAnalyzer
|
||||||
|
.new(simplefin_account, observed_balance: observed)
|
||||||
|
.call
|
||||||
|
|
||||||
|
case result.classification
|
||||||
|
when :credit
|
||||||
|
balance = -observed.abs
|
||||||
|
Rails.logger.info(
|
||||||
|
"SimpleFIN overpayment heuristic: classified as credit for sfa=#{simplefin_account.id}, " \
|
||||||
|
"observed=#{observed.to_s('F')} metrics=#{result.metrics.slice(:charges_total, :payments_total, :tx_count).inspect}"
|
||||||
|
)
|
||||||
|
Sentry.add_breadcrumb(Sentry::Breadcrumb.new(
|
||||||
|
category: "simplefin",
|
||||||
|
message: "liability_sign=credit",
|
||||||
|
data: { sfa_id: simplefin_account.id, observed: observed.to_s("F") }
|
||||||
|
)) rescue nil
|
||||||
|
when :debt
|
||||||
|
balance = observed.abs
|
||||||
|
Rails.logger.info(
|
||||||
|
"SimpleFIN overpayment heuristic: classified as debt for sfa=#{simplefin_account.id}, " \
|
||||||
|
"observed=#{observed.to_s('F')} metrics=#{result.metrics.slice(:charges_total, :payments_total, :tx_count).inspect}"
|
||||||
|
)
|
||||||
|
Sentry.add_breadcrumb(Sentry::Breadcrumb.new(
|
||||||
|
category: "simplefin",
|
||||||
|
message: "liability_sign=debt",
|
||||||
|
data: { sfa_id: simplefin_account.id, observed: observed.to_s("F") }
|
||||||
|
)) rescue nil
|
||||||
|
else
|
||||||
|
# 2) Fall back to existing sign-only logic (log unknown for observability)
|
||||||
|
begin
|
||||||
|
obs = {
|
||||||
|
reason: result.reason,
|
||||||
|
tx_count: result.metrics[:tx_count],
|
||||||
|
charges_total: result.metrics[:charges_total],
|
||||||
|
payments_total: result.metrics[:payments_total],
|
||||||
|
observed: observed.to_s("F")
|
||||||
|
}.compact
|
||||||
|
Rails.logger.info("SimpleFIN overpayment heuristic: unknown; falling back #{obs.inspect}")
|
||||||
|
rescue
|
||||||
|
# no-op
|
||||||
|
end
|
||||||
|
balance = normalize_liability_balance(observed, bal, avail)
|
||||||
|
end
|
||||||
|
rescue NameError
|
||||||
|
# Analyzer not loaded; keep legacy behavior
|
||||||
|
balance = normalize_liability_balance(observed, bal, avail)
|
||||||
|
rescue => e
|
||||||
|
Rails.logger.warn("SimpleFIN overpayment heuristic error for sfa=#{simplefin_account.id}: #{e.class} - #{e.message}")
|
||||||
|
balance = normalize_liability_balance(observed, bal, avail)
|
||||||
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
# Calculate cash balance correctly for investment accounts
|
# Calculate cash balance correctly for investment accounts
|
||||||
@@ -98,4 +174,19 @@ class SimplefinAccount::Processor
|
|||||||
)
|
)
|
||||||
end
|
end
|
||||||
end
|
end
|
||||||
|
|
||||||
|
# Helpers
|
||||||
|
# to_decimal and same_sign? provided by SimplefinNumericHelpers concern
|
||||||
|
|
||||||
|
def normalize_liability_balance(observed, bal, avail)
|
||||||
|
both_present = bal.nonzero? && avail.nonzero?
|
||||||
|
if both_present && same_sign?(bal, avail)
|
||||||
|
if bal.positive? && avail.positive?
|
||||||
|
return -observed.abs
|
||||||
|
elsif bal.negative? && avail.negative?
|
||||||
|
return observed.abs
|
||||||
|
end
|
||||||
|
end
|
||||||
|
-observed
|
||||||
|
end
|
||||||
end
|
end
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
require "set"
|
require "set"
|
||||||
class SimplefinItem::Importer
|
class SimplefinItem::Importer
|
||||||
|
include SimplefinNumericHelpers
|
||||||
class RateLimitedError < StandardError; end
|
class RateLimitedError < StandardError; end
|
||||||
attr_reader :simplefin_item, :simplefin_provider, :sync
|
attr_reader :simplefin_item, :simplefin_provider, :sync
|
||||||
|
|
||||||
@@ -117,9 +118,91 @@ class SimplefinItem::Importer
|
|||||||
# Only update balance for already-linked accounts (if any), to avoid creating duplicates in setup.
|
# Only update balance for already-linked accounts (if any), to avoid creating duplicates in setup.
|
||||||
if (acct = sfa.current_account)
|
if (acct = sfa.current_account)
|
||||||
adapter = Account::ProviderImportAdapter.new(acct)
|
adapter = Account::ProviderImportAdapter.new(acct)
|
||||||
|
|
||||||
|
# Normalize balances for SimpleFIN liabilities so immediate UI is correct after discovery
|
||||||
|
bal = to_decimal(account_data[:balance])
|
||||||
|
avail = to_decimal(account_data[:"available-balance"])
|
||||||
|
observed = bal.nonzero? ? bal : avail
|
||||||
|
|
||||||
|
is_linked_liability = [ "CreditCard", "Loan" ].include?(acct.accountable_type)
|
||||||
|
inferred = begin
|
||||||
|
Simplefin::AccountTypeMapper.infer(
|
||||||
|
name: account_data[:name],
|
||||||
|
holdings: account_data[:holdings],
|
||||||
|
extra: account_data[:extra],
|
||||||
|
balance: bal,
|
||||||
|
available_balance: avail,
|
||||||
|
institution: account_data.dig(:org, :name)
|
||||||
|
)
|
||||||
|
rescue
|
||||||
|
nil
|
||||||
|
end
|
||||||
|
is_mapper_liability = inferred && [ "CreditCard", "Loan" ].include?(inferred.accountable_type)
|
||||||
|
is_liability = is_linked_liability || is_mapper_liability
|
||||||
|
|
||||||
|
normalized = observed
|
||||||
|
if is_liability
|
||||||
|
# Try the overpayment analyzer first (feature-flagged)
|
||||||
|
begin
|
||||||
|
result = SimplefinAccount::Liabilities::OverpaymentAnalyzer
|
||||||
|
.new(sfa, observed_balance: observed)
|
||||||
|
.call
|
||||||
|
|
||||||
|
case result.classification
|
||||||
|
when :credit
|
||||||
|
normalized = -observed.abs
|
||||||
|
when :debt
|
||||||
|
normalized = observed.abs
|
||||||
|
else
|
||||||
|
# Fallback to existing normalization when unknown/disabled
|
||||||
|
begin
|
||||||
|
obs = {
|
||||||
|
reason: result.reason,
|
||||||
|
tx_count: result.metrics[:tx_count],
|
||||||
|
charges_total: result.metrics[:charges_total],
|
||||||
|
payments_total: result.metrics[:payments_total],
|
||||||
|
observed: observed.to_s("F")
|
||||||
|
}.compact
|
||||||
|
Rails.logger.info("SimpleFIN overpayment heuristic (balances-only): unknown; falling back #{obs.inspect}")
|
||||||
|
rescue
|
||||||
|
# no-op
|
||||||
|
end
|
||||||
|
both_present = bal.nonzero? && avail.nonzero?
|
||||||
|
if both_present && same_sign?(bal, avail)
|
||||||
|
if bal.positive? && avail.positive?
|
||||||
|
normalized = -observed.abs
|
||||||
|
elsif bal.negative? && avail.negative?
|
||||||
|
normalized = observed.abs
|
||||||
|
end
|
||||||
|
else
|
||||||
|
normalized = -observed
|
||||||
|
end
|
||||||
|
end
|
||||||
|
rescue NameError
|
||||||
|
# Analyzer missing; use legacy path
|
||||||
|
both_present = bal.nonzero? && avail.nonzero?
|
||||||
|
if both_present && same_sign?(bal, avail)
|
||||||
|
if bal.positive? && avail.positive?
|
||||||
|
normalized = -observed.abs
|
||||||
|
elsif bal.negative? && avail.negative?
|
||||||
|
normalized = observed.abs
|
||||||
|
end
|
||||||
|
else
|
||||||
|
normalized = -observed
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
cash = if acct.accountable_type == "Investment"
|
||||||
|
# Leave investment cash to investment calculators in full run
|
||||||
|
normalized
|
||||||
|
else
|
||||||
|
normalized
|
||||||
|
end
|
||||||
|
|
||||||
adapter.update_balance(
|
adapter.update_balance(
|
||||||
balance: account_data[:balance],
|
balance: normalized,
|
||||||
cash_balance: account_data[:"available-balance"],
|
cash_balance: cash,
|
||||||
source: "simplefin"
|
source: "simplefin"
|
||||||
)
|
)
|
||||||
end
|
end
|
||||||
@@ -1062,4 +1145,20 @@ class SimplefinItem::Importer
|
|||||||
|
|
||||||
ids.group_by(&:itself).select { |_, v| v.size > 1 }.keys
|
ids.group_by(&:itself).select { |_, v| v.size > 1 }.keys
|
||||||
end
|
end
|
||||||
|
|
||||||
|
# --- Simple helpers for numeric handling in normalization ---
|
||||||
|
def to_decimal(value)
|
||||||
|
return BigDecimal("0") if value.nil?
|
||||||
|
case value
|
||||||
|
when BigDecimal then value
|
||||||
|
when String then BigDecimal(value) rescue BigDecimal("0")
|
||||||
|
when Numeric then BigDecimal(value.to_s)
|
||||||
|
else
|
||||||
|
BigDecimal("0")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def same_sign?(a, b)
|
||||||
|
(a.positive? && b.positive?) || (a.negative? && b.negative?)
|
||||||
|
end
|
||||||
end
|
end
|
||||||
|
|||||||
2
test/fixtures/simplefin_accounts.yml
vendored
Normal file
2
test/fixtures/simplefin_accounts.yml
vendored
Normal file
@@ -0,0 +1,2 @@
|
|||||||
|
# Empty fixture to ensure the simplefin_accounts table is truncated during tests.
|
||||||
|
# Tests create SimplefinAccount records explicitly in setup.
|
||||||
2
test/fixtures/simplefin_items.yml
vendored
Normal file
2
test/fixtures/simplefin_items.yml
vendored
Normal file
@@ -0,0 +1,2 @@
|
|||||||
|
# Empty fixture to ensure the simplefin_items table is truncated during tests.
|
||||||
|
# Tests create SimplefinItem records explicitly in setup.
|
||||||
@@ -0,0 +1,99 @@
|
|||||||
|
require "test_helper"
|
||||||
|
|
||||||
|
class SimplefinAccount::Liabilities::OverpaymentAnalyzerTest < ActiveSupport::TestCase
|
||||||
|
# Limit fixtures to only what's required to avoid FK validation on unrelated tables
|
||||||
|
fixtures :families
|
||||||
|
setup do
|
||||||
|
@family = families(:dylan_family)
|
||||||
|
@item = SimplefinItem.create!(family: @family, name: "SimpleFIN", access_url: "https://example.com/token")
|
||||||
|
@sfa = SimplefinAccount.create!(
|
||||||
|
simplefin_item: @item,
|
||||||
|
name: "Test Credit Card",
|
||||||
|
account_id: "cc_txn_window_1",
|
||||||
|
currency: "USD",
|
||||||
|
account_type: "credit",
|
||||||
|
current_balance: BigDecimal("-22.72")
|
||||||
|
)
|
||||||
|
|
||||||
|
# Avoid cross‑suite fixture dependency by creating a fresh credit card account
|
||||||
|
@acct = Account.create!(
|
||||||
|
family: @family,
|
||||||
|
name: "Test CC",
|
||||||
|
balance: 0,
|
||||||
|
cash_balance: 0,
|
||||||
|
currency: "USD",
|
||||||
|
accountable: CreditCard.new
|
||||||
|
)
|
||||||
|
# Create explicit provider link to ensure FK validity in isolation
|
||||||
|
AccountProvider.create!(account: @acct, provider: @sfa)
|
||||||
|
|
||||||
|
# Enable heuristic
|
||||||
|
Setting["simplefin_cc_overpayment_detection"] = "true"
|
||||||
|
# Loosen thresholds for focused unit tests
|
||||||
|
Setting["simplefin_cc_overpayment_min_txns"] = "1"
|
||||||
|
Setting["simplefin_cc_overpayment_min_payments"] = "1"
|
||||||
|
Setting["simplefin_cc_overpayment_statement_guard_days"] = "0"
|
||||||
|
end
|
||||||
|
|
||||||
|
teardown do
|
||||||
|
# Disable heuristic to avoid bleeding into other tests
|
||||||
|
Setting["simplefin_cc_overpayment_detection"] = nil
|
||||||
|
Setting["simplefin_cc_overpayment_min_txns"] = nil
|
||||||
|
Setting["simplefin_cc_overpayment_min_payments"] = nil
|
||||||
|
Setting["simplefin_cc_overpayment_statement_guard_days"] = nil
|
||||||
|
begin
|
||||||
|
Rails.cache.delete_matched("simplefin:sfa:#{@sfa.id}:liability_sign_hint") if @sfa&.id
|
||||||
|
rescue
|
||||||
|
# ignore cache backends without delete_matched
|
||||||
|
end
|
||||||
|
# Ensure created records are removed to avoid FK validation across examples in single-file runs
|
||||||
|
AccountProvider.where(account_id: @acct.id).destroy_all rescue nil
|
||||||
|
@acct.destroy! rescue nil
|
||||||
|
@sfa.destroy! rescue nil
|
||||||
|
@item.destroy! rescue nil
|
||||||
|
end
|
||||||
|
|
||||||
|
test "classifies credit when payments exceed charges roughly by observed amount" do
|
||||||
|
# Create transactions in Maybe convention for liabilities:
|
||||||
|
# charges/spend: positive; payments: negative
|
||||||
|
# Observed abs is 22.72; make payments exceed charges by ~22.72
|
||||||
|
@acct.entries.delete_all
|
||||||
|
@acct.entries.create!(date: 10.days.ago.to_date, name: "Store A", amount: 50, currency: "USD", entryable: Transaction.new)
|
||||||
|
# Ensure payments exceed charges by at least observed.abs (~22.72)
|
||||||
|
@acct.entries.create!(date: 8.days.ago.to_date, name: "Payment", amount: -75, currency: "USD", entryable: Transaction.new)
|
||||||
|
|
||||||
|
result = SimplefinAccount::Liabilities::OverpaymentAnalyzer.new(@sfa, observed_balance: @sfa.current_balance).call
|
||||||
|
assert_equal :credit, result.classification, "expected classification to be credit"
|
||||||
|
end
|
||||||
|
|
||||||
|
test "classifies debt when charges exceed payments" do
|
||||||
|
@acct.entries.delete_all
|
||||||
|
@acct.entries.create!(date: 12.days.ago.to_date, name: "Groceries", amount: 120, currency: "USD", entryable: Transaction.new)
|
||||||
|
@acct.entries.create!(date: 11.days.ago.to_date, name: "Coffee", amount: 10, currency: "USD", entryable: Transaction.new)
|
||||||
|
@acct.entries.create!(date: 9.days.ago.to_date, name: "Payment", amount: -50, currency: "USD", entryable: Transaction.new)
|
||||||
|
|
||||||
|
result = SimplefinAccount::Liabilities::OverpaymentAnalyzer.new(@sfa, observed_balance: BigDecimal("-80")).call
|
||||||
|
assert_equal :debt, result.classification, "expected classification to be debt"
|
||||||
|
end
|
||||||
|
|
||||||
|
test "returns unknown when insufficient transactions" do
|
||||||
|
@acct.entries.delete_all
|
||||||
|
@acct.entries.create!(date: 5.days.ago.to_date, name: "Small", amount: 1, currency: "USD", entryable: Transaction.new)
|
||||||
|
|
||||||
|
result = SimplefinAccount::Liabilities::OverpaymentAnalyzer.new(@sfa, observed_balance: BigDecimal("-5")).call
|
||||||
|
assert_equal :unknown, result.classification
|
||||||
|
end
|
||||||
|
|
||||||
|
test "fallback to raw payload when no entries present" do
|
||||||
|
@acct.entries.delete_all
|
||||||
|
# Provide raw transactions in provider convention (expenses negative, income positive)
|
||||||
|
# We must negate in analyzer to convert to Maybe convention.
|
||||||
|
@sfa.update!(raw_transactions_payload: [
|
||||||
|
{ id: "t1", amount: -100, posted: (10.days.ago.to_date.to_s) }, # charge (-> +100)
|
||||||
|
{ id: "t2", amount: 150, posted: (8.days.ago.to_date.to_s) } # payment (-> -150)
|
||||||
|
])
|
||||||
|
|
||||||
|
result = SimplefinAccount::Liabilities::OverpaymentAnalyzer.new(@sfa, observed_balance: BigDecimal("-50")).call
|
||||||
|
assert_equal :credit, result.classification
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -81,4 +81,101 @@ class SimplefinAccountProcessorTest < ActiveSupport::TestCase
|
|||||||
|
|
||||||
assert_equal BigDecimal("-75.00"), acct.reload.balance
|
assert_equal BigDecimal("-75.00"), acct.reload.balance
|
||||||
end
|
end
|
||||||
|
|
||||||
|
test "liability debt with both fields negative becomes positive (you owe)" do
|
||||||
|
sfin_acct = SimplefinAccount.create!(
|
||||||
|
simplefin_item: @item,
|
||||||
|
name: "BofA Visa",
|
||||||
|
account_id: "cc_bofa_1",
|
||||||
|
currency: "USD",
|
||||||
|
account_type: "credit",
|
||||||
|
current_balance: BigDecimal("-1200"),
|
||||||
|
available_balance: BigDecimal("-5000")
|
||||||
|
)
|
||||||
|
|
||||||
|
acct = accounts(:credit_card)
|
||||||
|
acct.update!(simplefin_account: sfin_acct)
|
||||||
|
|
||||||
|
SimplefinAccount::Processor.new(sfin_acct).send(:process_account!)
|
||||||
|
|
||||||
|
assert_equal BigDecimal("1200"), acct.reload.balance
|
||||||
|
end
|
||||||
|
|
||||||
|
test "liability overpayment with both fields positive becomes negative (credit)" do
|
||||||
|
sfin_acct = SimplefinAccount.create!(
|
||||||
|
simplefin_item: @item,
|
||||||
|
name: "BofA Visa",
|
||||||
|
account_id: "cc_bofa_2",
|
||||||
|
currency: "USD",
|
||||||
|
account_type: "credit",
|
||||||
|
current_balance: BigDecimal("75"),
|
||||||
|
available_balance: BigDecimal("5000")
|
||||||
|
)
|
||||||
|
|
||||||
|
acct = accounts(:credit_card)
|
||||||
|
acct.update!(simplefin_account: sfin_acct)
|
||||||
|
|
||||||
|
SimplefinAccount::Processor.new(sfin_acct).send(:process_account!)
|
||||||
|
|
||||||
|
assert_equal BigDecimal("-75"), acct.reload.balance
|
||||||
|
end
|
||||||
|
|
||||||
|
test "mixed signs falls back to invert observed (balance positive, avail negative => negative)" do
|
||||||
|
sfin_acct = SimplefinAccount.create!(
|
||||||
|
simplefin_item: @item,
|
||||||
|
name: "Chase Freedom",
|
||||||
|
account_id: "cc_chase_1",
|
||||||
|
currency: "USD",
|
||||||
|
account_type: "credit",
|
||||||
|
current_balance: BigDecimal("50"),
|
||||||
|
available_balance: BigDecimal("-5000")
|
||||||
|
)
|
||||||
|
|
||||||
|
acct = accounts(:credit_card)
|
||||||
|
acct.update!(simplefin_account: sfin_acct)
|
||||||
|
|
||||||
|
SimplefinAccount::Processor.new(sfin_acct).send(:process_account!)
|
||||||
|
|
||||||
|
assert_equal BigDecimal("-50"), acct.reload.balance
|
||||||
|
end
|
||||||
|
|
||||||
|
test "only available-balance present positive → negative (credit) for liability" do
|
||||||
|
sfin_acct = SimplefinAccount.create!(
|
||||||
|
simplefin_item: @item,
|
||||||
|
name: "Chase Visa",
|
||||||
|
account_id: "cc_chase_2",
|
||||||
|
currency: "USD",
|
||||||
|
account_type: "credit",
|
||||||
|
current_balance: nil,
|
||||||
|
available_balance: BigDecimal("25")
|
||||||
|
)
|
||||||
|
|
||||||
|
acct = accounts(:credit_card)
|
||||||
|
acct.update!(simplefin_account: sfin_acct)
|
||||||
|
|
||||||
|
SimplefinAccount::Processor.new(sfin_acct).send(:process_account!)
|
||||||
|
|
||||||
|
assert_equal BigDecimal("-25"), acct.reload.balance
|
||||||
|
end
|
||||||
|
|
||||||
|
test "mislinked as asset but mapper infers credit → normalize as liability" do
|
||||||
|
sfin_acct = SimplefinAccount.create!(
|
||||||
|
simplefin_item: @item,
|
||||||
|
name: "Visa Signature",
|
||||||
|
account_id: "cc_mislinked",
|
||||||
|
currency: "USD",
|
||||||
|
account_type: "credit",
|
||||||
|
current_balance: BigDecimal("100.00"),
|
||||||
|
available_balance: BigDecimal("5000.00")
|
||||||
|
)
|
||||||
|
|
||||||
|
# Link to an asset account intentionally
|
||||||
|
acct = accounts(:depository)
|
||||||
|
acct.update!(simplefin_account: sfin_acct)
|
||||||
|
|
||||||
|
SimplefinAccount::Processor.new(sfin_acct).send(:process_account!)
|
||||||
|
|
||||||
|
# Mapper should infer liability from name; final should be negative
|
||||||
|
assert_equal BigDecimal("-100.00"), acct.reload.balance
|
||||||
|
end
|
||||||
end
|
end
|
||||||
|
|||||||
Reference in New Issue
Block a user