Add pending transaction handling and duplicate reconciliation logic (#602)

* Add pending transaction handling and duplicate reconciliation logic

- Implemented logic to exclude pending transactions from budgets and analytics calculations.
- Introduced mechanisms for reconciling pending transactions with posted versions.
- Added duplicate detection with support for merging or dismissing matches.
- Updated transaction search filters to include a `status_filter` for pending/confirmed transactions.
- Introduced UI elements for reviewing and resolving duplicates.
- Enhanced `ProviderSyncSummary` with stats for reconciled and stale pending transactions.

* Refactor translation handling and enhance transaction and sync logic

- Moved hardcoded strings to locale files for improved translation support.
- Refined styling for duplicate transaction indicators and sync summaries.
- Improved logic for excluding stale pending transactions and updating timestamps on batch exclusion.
- Added unique IDs to status filters for better element targeting in UI.
- Optimized database queries to avoid N+1 issues in stale pending calculations.

* Add sync settings and enhance pending transaction handling

- Introduced a new "Sync Settings" section in hosting settings with UI to toggle inclusion of pending transactions.
- Updated handling of pending transactions with improved inference logic for `posted=0` and `transacted_at` in processors.
- Added priority order for pending transaction inclusion: explicit argument > environment variable > runtime configurable setting.
- Refactored settings and controllers to store updated sync preferences.

* Refactor sync settings and pending transaction reconciliation

- Extracted logic for pending transaction reconciliation, stale exclusion, and unmatched tracking into dedicated methods for better maintainability.
- Updated sync settings to infer defaults from multiple provider environment variables (`SIMPLEFIN_INCLUDE_PENDING`, `PLAID_INCLUDE_PENDING`).
- Refined UI and messaging to handle multi-provider configurations in sync settings.

# Conflicts:
#	app/models/simplefin_item/importer.rb

* Debounce transaction reconciliation during imports

- Added per-run reconciliation debouncing to prevent repeated scans for the same account during chunked history imports.
- Trimmed size of reconciliation stats to retain recent details only.
- Introduced error tracking for reconciliation steps to improve UI visibility of issues.

* Apply ABS() in pending transaction queries and improve error handling

- Updated pending transaction logic to use ABS() for consistent handling of negative amounts.
- Adjusted amount bounds calculations to ensure accuracy for both positive and negative values.
- Refined exception handling in `merge_duplicate` to log failures and update user alert.
- Replaced `Date.today` with `Date.current` in tests to ensure timezone consistency.
- Minor optimization to avoid COUNT queries by loading limited records directly.

* Improve error handling in duplicate suggestion and dismissal logic

- Added exception handling for `store_duplicate_suggestion` to log failures and prevent crashes during fuzzy/low-confidence matches.
- Enhanced `dismiss_duplicate` action to handle `ActiveRecord::RecordInvalid` and display appropriate user alerts.

---------

Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
This commit is contained in:
LPW
2026-01-10 14:11:00 -05:00
committed by GitHub
parent f012e38824
commit 3658e812a8
40 changed files with 2014 additions and 53 deletions

View File

@@ -785,4 +785,482 @@ class Account::ProviderImportAdapterTest < ActiveSupport::TestCase
assert_nil newer_entry.reload.external_id
end
end
# ============================================================================
# Pending→Posted Transaction Reconciliation Tests
# ============================================================================
test "reconciles pending transaction when posted version arrives with different external_id" do
# Simulate SimpleFIN giving different IDs for pending vs posted transactions
# First, import a pending transaction
pending_entry = @adapter.import_transaction(
external_id: "simplefin_pending_abc",
amount: 99.99,
currency: "USD",
date: Date.today - 2.days,
name: "Coffee Shop",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
assert pending_entry.transaction.pending?, "Entry should be marked pending"
original_id = pending_entry.id
# Now import the posted version with a DIFFERENT external_id
# This should claim the pending entry, not create a duplicate
assert_no_difference "@account.entries.count" do
posted_entry = @adapter.import_transaction(
external_id: "simplefin_posted_xyz",
amount: 99.99,
currency: "USD",
date: Date.today,
name: "Coffee Shop - Posted",
source: "simplefin",
extra: { "simplefin" => { "pending" => false } }
)
# Should be the same entry, now with updated external_id
assert_equal original_id, posted_entry.id
assert_equal "simplefin_posted_xyz", posted_entry.external_id
assert_not posted_entry.transaction.pending?, "Entry should no longer be pending"
end
end
test "does not reconcile when posted transaction has same external_id as pending" do
# When external_id matches, normal dedup should handle it
pending_entry = @adapter.import_transaction(
external_id: "simplefin_same_id",
amount: 50.00,
currency: "USD",
date: Date.today - 1.day,
name: "Gas Station",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Import posted version with SAME external_id
assert_no_difference "@account.entries.count" do
posted_entry = @adapter.import_transaction(
external_id: "simplefin_same_id",
amount: 50.00,
currency: "USD",
date: Date.today,
name: "Gas Station - Posted",
source: "simplefin",
extra: { "simplefin" => { "pending" => false } }
)
assert_equal pending_entry.id, posted_entry.id
assert_not posted_entry.transaction.pending?
end
end
test "fuzzy amount match creates suggestion instead of auto-claiming" do
# Import pending transaction (pre-tip authorization)
pending_entry = @adapter.import_transaction(
external_id: "simplefin_pending_amount_test",
amount: 100.00,
currency: "USD",
date: Date.today - 1.day,
name: "Restaurant",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Import posted with tip added - should NOT auto-claim, but should store suggestion
# Fuzzy matches now create suggestions for user review instead of auto-reconciling
assert_difference "@account.entries.count", 1 do
posted_entry = @adapter.import_transaction(
external_id: "simplefin_posted_amount_test",
amount: 105.00, # 5% tip added - within 25% tolerance
currency: "USD",
date: Date.today,
name: "Restaurant",
source: "simplefin",
extra: { "simplefin" => { "pending" => false } }
)
# Should be a NEW entry (not claimed)
assert_not_equal pending_entry.id, posted_entry.id
assert_equal "simplefin_posted_amount_test", posted_entry.external_id
# The PENDING entry should now have a potential_posted_match suggestion
pending_entry.reload
assert pending_entry.transaction.has_potential_duplicate?
assert_equal posted_entry.id, pending_entry.transaction.potential_duplicate_entry.id
end
end
test "does not reconcile pending when amount difference exceeds tolerance" do
# Import pending transaction
pending_entry = @adapter.import_transaction(
external_id: "simplefin_pending_big_diff",
amount: 50.00,
currency: "USD",
date: Date.today - 1.day,
name: "Store",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Import posted with amount >25% different - should NOT match
# $100 posted / 1.25 = $80 minimum pending, but pending is only $50
assert_difference "@account.entries.count", 1 do
posted_entry = @adapter.import_transaction(
external_id: "simplefin_posted_big_diff",
amount: 100.00, # 100% increase - way outside 25% tolerance
currency: "USD",
date: Date.today,
name: "Store",
source: "simplefin",
extra: { "simplefin" => { "pending" => false } }
)
assert_not_equal pending_entry.id, posted_entry.id
end
end
test "does not reconcile pending when date is outside window" do
# Import pending transaction
pending_entry = @adapter.import_transaction(
external_id: "simplefin_pending_date_test",
amount: 25.00,
currency: "USD",
date: Date.today - 15.days, # 15 days ago
name: "Subscription",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Import posted with date outside 7-day window - should NOT match
assert_difference "@account.entries.count", 1 do
posted_entry = @adapter.import_transaction(
external_id: "simplefin_posted_date_test",
amount: 25.00,
currency: "USD",
date: Date.today,
name: "Subscription",
source: "simplefin",
extra: { "simplefin" => { "pending" => false } }
)
assert_not_equal pending_entry.id, posted_entry.id
end
end
test "reconciles pending within 7 day window" do
# Import pending transaction
pending_entry = @adapter.import_transaction(
external_id: "simplefin_pending_window_test",
amount: 75.00,
currency: "USD",
date: Date.today - 5.days,
name: "Online Order",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Import posted within 7-day window - should match
assert_no_difference "@account.entries.count" do
posted_entry = @adapter.import_transaction(
external_id: "simplefin_posted_window_test",
amount: 75.00,
currency: "USD",
date: Date.today,
name: "Online Order - Posted",
source: "simplefin",
extra: { "simplefin" => { "pending" => false } }
)
assert_equal pending_entry.id, posted_entry.id
end
end
test "does not reconcile pending from different source" do
# Import pending from SimpleFIN
pending_entry = @adapter.import_transaction(
external_id: "simplefin_pending_source_test",
amount: 30.00,
currency: "USD",
date: Date.today - 1.day,
name: "Pharmacy",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Import from different source (plaid) - should NOT match SimpleFIN pending
assert_difference "@account.entries.count", 1 do
plaid_entry = @adapter.import_transaction(
external_id: "plaid_posted_source_test",
amount: 30.00,
currency: "USD",
date: Date.today,
name: "Pharmacy",
source: "plaid",
extra: { "plaid" => { "pending" => false } }
)
assert_not_equal pending_entry.id, plaid_entry.id
end
end
test "does not reconcile when incoming transaction is also pending" do
# Import first pending transaction
pending_entry1 = @adapter.import_transaction(
external_id: "simplefin_pending_1",
amount: 45.00,
currency: "USD",
date: Date.today - 1.day,
name: "Store",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Import another pending transaction with different ID - should NOT match
assert_difference "@account.entries.count", 1 do
pending_entry2 = @adapter.import_transaction(
external_id: "simplefin_pending_2",
amount: 45.00,
currency: "USD",
date: Date.today,
name: "Store",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
assert_not_equal pending_entry1.id, pending_entry2.id
end
end
test "reconciles most recent pending when multiple exist" do
# Create two pending transactions with same amount
older_pending = @adapter.import_transaction(
external_id: "simplefin_older_pending",
amount: 60.00,
currency: "USD",
date: Date.today - 5.days,
name: "Recurring Payment - Old",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
newer_pending = @adapter.import_transaction(
external_id: "simplefin_newer_pending",
amount: 60.00,
currency: "USD",
date: Date.today - 1.day,
name: "Recurring Payment - New",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Import posted - should match the most recent pending (by date)
assert_no_difference "@account.entries.count" do
posted_entry = @adapter.import_transaction(
external_id: "simplefin_posted_recurring",
amount: 60.00,
currency: "USD",
date: Date.today,
name: "Recurring Payment - Posted",
source: "simplefin",
extra: { "simplefin" => { "pending" => false } }
)
# Should match the newer pending entry
assert_equal newer_pending.id, posted_entry.id
# Older pending should remain untouched
assert_equal "simplefin_older_pending", older_pending.reload.external_id
end
end
test "find_pending_transaction returns nil when no pending transactions exist" do
# Create a non-pending transaction
@adapter.import_transaction(
external_id: "simplefin_not_pending",
amount: 40.00,
currency: "USD",
date: Date.today - 1.day,
name: "Regular Transaction",
source: "simplefin",
extra: { "simplefin" => { "pending" => false } }
)
result = @adapter.find_pending_transaction(
date: Date.today,
amount: 40.00,
currency: "USD",
source: "simplefin"
)
assert_nil result
end
# ============================================================================
# Critical Direction Fix Tests (CITGO Bug Prevention)
# ============================================================================
test "does not match pending transaction that is AFTER the posted date (direction fix)" do
# This is the CITGO bug scenario:
# - Posted transaction on Dec 31
# - Pending transaction on Jan 8 (AFTER the posted)
# - These should NOT match because pending MUST come BEFORE posted
# First, import a POSTED transaction on an earlier date
posted_entry = @adapter.import_transaction(
external_id: "simplefin_posted_dec31",
amount: 6.67,
currency: "USD",
date: Date.today - 8.days, # Dec 31 (earlier)
name: "CITGO Gas Station",
source: "simplefin",
extra: { "simplefin" => { "pending" => false } }
)
# Now import a PENDING transaction on a LATER date
# This should NOT be matched because the date direction is wrong
assert_difference "@account.entries.count", 1 do
pending_entry = @adapter.import_transaction(
external_id: "simplefin_pending_jan8",
amount: 6.65, # Similar but different amount
currency: "USD",
date: Date.today, # Jan 8 (later)
name: "CITGO Gas Station",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Should be a DIFFERENT entry - not matched to the earlier posted one
assert_not_equal posted_entry.id, pending_entry.id
assert pending_entry.transaction.pending?
end
end
test "find_pending_transaction only searches backward in time" do
# Create a pending transaction in the FUTURE (after the posted date we'll search from)
# This should NOT be found because pending must be ON or BEFORE posted
future_pending = @adapter.import_transaction(
external_id: "simplefin_future_pending",
amount: 50.00,
currency: "USD",
date: Date.today + 3.days, # Future date
name: "Future Transaction",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Search from today - should NOT find the future pending
result = @adapter.find_pending_transaction(
date: Date.today,
amount: 50.00,
currency: "USD",
source: "simplefin"
)
assert_nil result, "Should not find pending transactions that are in the future relative to the posted date"
end
test "find_pending_transaction finds pending transaction that is before posted date" do
# Create a pending transaction in the PAST (before the posted date)
# This SHOULD be found
past_pending = @adapter.import_transaction(
external_id: "simplefin_past_pending",
amount: 75.00,
currency: "USD",
date: Date.today - 3.days, # 3 days ago
name: "Past Transaction",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Search from today - should find the past pending
result = @adapter.find_pending_transaction(
date: Date.today,
amount: 75.00,
currency: "USD",
source: "simplefin"
)
assert_equal past_pending.id, result.id
end
# ============================================================================
# Plaid pending_transaction_id Tests
# ============================================================================
test "reconciles pending via Plaid pending_transaction_id" do
# Import a pending transaction
pending_entry = @adapter.import_transaction(
external_id: "plaid_pending_abc",
amount: 42.00,
currency: "USD",
date: Date.today - 2.days,
name: "Coffee Shop",
source: "plaid",
extra: { "plaid" => { "pending" => true } }
)
# Import posted with pending_transaction_id linking to the pending
assert_no_difference "@account.entries.count" do
posted_entry = @adapter.import_transaction(
external_id: "plaid_posted_xyz",
amount: 42.00,
currency: "USD",
date: Date.today,
name: "Coffee Shop",
source: "plaid",
pending_transaction_id: "plaid_pending_abc", # Links to pending
extra: { "plaid" => { "pending" => false, "pending_transaction_id" => "plaid_pending_abc" } }
)
# Should claim the pending entry
assert_equal pending_entry.id, posted_entry.id
assert_equal "plaid_posted_xyz", posted_entry.external_id
assert_not posted_entry.transaction.pending?
end
end
test "Plaid pending_transaction_id takes priority over amount matching" do
# Create TWO pending transactions with same amount
pending1 = @adapter.import_transaction(
external_id: "plaid_pending_1",
amount: 25.00,
currency: "USD",
date: Date.today - 1.day,
name: "Store A",
source: "plaid",
extra: { "plaid" => { "pending" => true } }
)
pending2 = @adapter.import_transaction(
external_id: "plaid_pending_2",
amount: 25.00,
currency: "USD",
date: Date.today - 1.day,
name: "Store B",
source: "plaid",
extra: { "plaid" => { "pending" => true } }
)
# Import posted that explicitly links to pending2 via pending_transaction_id
assert_no_difference "@account.entries.count" do
posted_entry = @adapter.import_transaction(
external_id: "plaid_posted_linked",
amount: 25.00,
currency: "USD",
date: Date.today,
name: "Store B",
source: "plaid",
pending_transaction_id: "plaid_pending_2", # Explicitly links to pending2
extra: { "plaid" => { "pending" => false } }
)
# Should claim pending2 specifically (not pending1)
assert_equal pending2.id, posted_entry.id
assert_equal "plaid_posted_linked", posted_entry.external_id
end
# pending1 should still exist as pending
pending1.reload
assert_equal "plaid_pending_1", pending1.external_id
end
end

View File

@@ -29,8 +29,8 @@ class SimplefinEntry::ProcessorTest < ActiveSupport::TestCase
payee: "Pizza Hut",
description: "Order #1234",
memo: "Carryout",
posted: Date.today.to_s,
transacted_at: (Date.today - 1).to_s,
posted: Date.current.to_s,
transacted_at: (Date.current - 1).to_s,
extra: { category: "restaurants", check_number: nil }
}
@@ -64,7 +64,7 @@ class SimplefinEntry::ProcessorTest < ActiveSupport::TestCase
description: "Latte",
memo: "Morning run",
posted: nil,
transacted_at: (Date.today - 3).to_s
transacted_at: (Date.current - 3).to_s
}
SimplefinEntry::Processor.new(tx, simplefin_account: @simplefin_account).process
@@ -77,8 +77,8 @@ class SimplefinEntry::ProcessorTest < ActiveSupport::TestCase
test "captures FX metadata when tx currency differs from account currency" do
# Account is USD from setup; use EUR for tx
t_date = (Date.today - 5)
p_date = Date.today
t_date = (Date.current - 5)
p_date = Date.current
tx = {
id: "tx_fx_1",
@@ -106,8 +106,8 @@ class SimplefinEntry::ProcessorTest < ActiveSupport::TestCase
payee: "Test Store",
description: "Auth",
memo: "",
posted: Date.today.to_s, # provider says pending=true should still flag
transacted_at: (Date.today - 1).to_s,
posted: Date.current.to_s, # provider says pending=true should still flag
transacted_at: (Date.current - 1).to_s,
pending: true
}
@@ -120,7 +120,7 @@ class SimplefinEntry::ProcessorTest < ActiveSupport::TestCase
test "posted==0 treated as missing, entry uses transacted_at date and flags pending" do
# Simulate provider sending epoch-like zeros for posted and an integer transacted_at
t_epoch = (Date.today - 2).to_time.to_i
t_epoch = (Date.current - 2).to_time.to_i
tx = {
id: "tx_pending_zero_posted_1",
amount: "-6.48",
@@ -141,4 +141,26 @@ class SimplefinEntry::ProcessorTest < ActiveSupport::TestCase
sf = entry.transaction.extra.fetch("simplefin")
assert_equal true, sf["pending"], "expected pending flag to be true when posted==0 and/or pending=true"
end
test "infers pending when posted is explicitly 0 and transacted_at present (no explicit pending flag)" do
# Some SimpleFIN banks indicate pending by sending posted=0 + transacted_at, without pending flag
t_epoch = (Date.current - 1).to_time.to_i
tx = {
id: "tx_inferred_pending_1",
amount: "-15.00",
currency: "USD",
payee: "Gas Station",
description: "Fuel",
memo: "",
posted: 0,
transacted_at: t_epoch
# Note: NO pending flag set
}
SimplefinEntry::Processor.new(tx, simplefin_account: @simplefin_account).process
entry = @account.entries.find_by!(external_id: "simplefin_tx_inferred_pending_1", source: "simplefin")
sf = entry.transaction.extra.fetch("simplefin")
assert_equal true, sf["pending"], "expected pending to be inferred from posted=0 + transacted_at present"
end
end