mirror of
https://github.com/we-promise/sure.git
synced 2026-04-07 14:31:25 +00:00
* Add pending transaction handling and duplicate reconciliation logic - Implemented logic to exclude pending transactions from budgets and analytics calculations. - Introduced mechanisms for reconciling pending transactions with posted versions. - Added duplicate detection with support for merging or dismissing matches. - Updated transaction search filters to include a `status_filter` for pending/confirmed transactions. - Introduced UI elements for reviewing and resolving duplicates. - Enhanced `ProviderSyncSummary` with stats for reconciled and stale pending transactions. * Refactor translation handling and enhance transaction and sync logic - Moved hardcoded strings to locale files for improved translation support. - Refined styling for duplicate transaction indicators and sync summaries. - Improved logic for excluding stale pending transactions and updating timestamps on batch exclusion. - Added unique IDs to status filters for better element targeting in UI. - Optimized database queries to avoid N+1 issues in stale pending calculations. * Add sync settings and enhance pending transaction handling - Introduced a new "Sync Settings" section in hosting settings with UI to toggle inclusion of pending transactions. - Updated handling of pending transactions with improved inference logic for `posted=0` and `transacted_at` in processors. - Added priority order for pending transaction inclusion: explicit argument > environment variable > runtime configurable setting. - Refactored settings and controllers to store updated sync preferences. * Refactor sync settings and pending transaction reconciliation - Extracted logic for pending transaction reconciliation, stale exclusion, and unmatched tracking into dedicated methods for better maintainability. - Updated sync settings to infer defaults from multiple provider environment variables (`SIMPLEFIN_INCLUDE_PENDING`, `PLAID_INCLUDE_PENDING`). - Refined UI and messaging to handle multi-provider configurations in sync settings. # Conflicts: # app/models/simplefin_item/importer.rb * Debounce transaction reconciliation during imports - Added per-run reconciliation debouncing to prevent repeated scans for the same account during chunked history imports. - Trimmed size of reconciliation stats to retain recent details only. - Introduced error tracking for reconciliation steps to improve UI visibility of issues. * Apply ABS() in pending transaction queries and improve error handling - Updated pending transaction logic to use ABS() for consistent handling of negative amounts. - Adjusted amount bounds calculations to ensure accuracy for both positive and negative values. - Refined exception handling in `merge_duplicate` to log failures and update user alert. - Replaced `Date.today` with `Date.current` in tests to ensure timezone consistency. - Minor optimization to avoid COUNT queries by loading limited records directly. * Improve error handling in duplicate suggestion and dismissal logic - Added exception handling for `store_duplicate_suggestion` to log failures and prevent crashes during fuzzy/low-confidence matches. - Enhanced `dismiss_duplicate` action to handle `ActiveRecord::RecordInvalid` and display appropriate user alerts. --------- Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
159 lines
6.3 KiB
Ruby
159 lines
6.3 KiB
Ruby
# frozen_string_literal: true
|
|
|
|
namespace :simplefin do
|
|
desc "Find and optionally remove duplicate pending transactions that have matching posted versions"
|
|
task pending_cleanup: :environment do
|
|
dry_run = ENV["DRY_RUN"] != "false"
|
|
date_window = (ENV["DATE_WINDOW"] || 8).to_i
|
|
|
|
puts "SimpleFIN Pending Transaction Cleanup"
|
|
puts "======================================"
|
|
puts "Mode: #{dry_run ? 'DRY RUN (no changes)' : 'LIVE (will delete duplicates)'}"
|
|
puts "Date window: #{date_window} days"
|
|
puts ""
|
|
|
|
# Find all pending SimpleFIN transactions
|
|
pending_entries = Entry.joins(
|
|
"INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'"
|
|
).where(source: "simplefin")
|
|
.where("transactions.extra -> 'simplefin' ->> 'pending' = ?", "true")
|
|
.includes(:account)
|
|
|
|
puts "Found #{pending_entries.count} pending SimpleFIN transactions"
|
|
puts ""
|
|
|
|
duplicates_found = 0
|
|
duplicates_to_delete = []
|
|
|
|
pending_entries.find_each do |pending_entry|
|
|
# Look for a matching posted transaction
|
|
posted_match = Entry.joins(
|
|
"INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'"
|
|
).where(account_id: pending_entry.account_id)
|
|
.where(source: "simplefin")
|
|
.where(amount: pending_entry.amount)
|
|
.where(currency: pending_entry.currency)
|
|
.where(date: pending_entry.date..(pending_entry.date + date_window.days)) # Posted must be ON or AFTER pending
|
|
.where.not(id: pending_entry.id)
|
|
.where("transactions.extra -> 'simplefin' ->> 'pending' != ? OR transactions.extra -> 'simplefin' ->> 'pending' IS NULL", "true")
|
|
.first
|
|
|
|
if posted_match
|
|
duplicates_found += 1
|
|
duplicates_to_delete << pending_entry
|
|
|
|
puts "DUPLICATE FOUND:"
|
|
puts " Pending: ID=#{pending_entry.id} | #{pending_entry.date} | #{pending_entry.name} | #{pending_entry.amount} #{pending_entry.currency}"
|
|
puts " Posted: ID=#{posted_match.id} | #{posted_match.date} | #{posted_match.name} | #{posted_match.amount} #{posted_match.currency}"
|
|
puts " Account: #{pending_entry.account.name}"
|
|
puts ""
|
|
end
|
|
end
|
|
|
|
puts "======================================"
|
|
puts "Summary: #{duplicates_found} duplicate pending transactions found"
|
|
puts ""
|
|
|
|
if duplicates_found > 0
|
|
if dry_run
|
|
puts "To delete these duplicates, run:"
|
|
puts " rails simplefin:pending_cleanup DRY_RUN=false"
|
|
puts ""
|
|
puts "To adjust the date matching window (default 8 days):"
|
|
puts " rails simplefin:pending_cleanup DATE_WINDOW=14"
|
|
else
|
|
print "Deleting #{duplicates_to_delete.count} duplicate pending entries... "
|
|
Entry.where(id: duplicates_to_delete.map(&:id)).destroy_all
|
|
puts "Done!"
|
|
end
|
|
else
|
|
puts "No duplicates found. Nothing to clean up."
|
|
end
|
|
end
|
|
|
|
desc "Un-exclude pending transactions that were wrongly matched (fixes direction bug)"
|
|
task pending_restore: :environment do
|
|
dry_run = ENV["DRY_RUN"] != "false"
|
|
date_window = (ENV["DATE_WINDOW"] || 8).to_i
|
|
|
|
puts "Restore Wrongly Excluded Pending Transactions"
|
|
puts "=============================================="
|
|
puts "Mode: #{dry_run ? 'DRY RUN (no changes)' : 'LIVE (will restore)'}"
|
|
puts "Date window: #{date_window} days (forward only)"
|
|
puts ""
|
|
|
|
# Find all EXCLUDED pending transactions (these may have been wrongly excluded)
|
|
excluded_pending = Entry.joins(
|
|
"INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'"
|
|
).where(excluded: true)
|
|
.where(<<~SQL.squish)
|
|
(transactions.extra -> 'simplefin' ->> 'pending')::boolean = true
|
|
OR (transactions.extra -> 'plaid' ->> 'pending')::boolean = true
|
|
SQL
|
|
|
|
puts "Found #{excluded_pending.count} excluded pending transactions to evaluate"
|
|
puts ""
|
|
|
|
to_restore = []
|
|
|
|
excluded_pending.includes(:account).find_each do |pending_entry|
|
|
# Check if there's a VALID posted match using CORRECT logic (forward-only dates)
|
|
# Posted date must be ON or AFTER pending date
|
|
valid_match = pending_entry.account.entries
|
|
.joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'")
|
|
.where.not(id: pending_entry.id)
|
|
.where(currency: pending_entry.currency)
|
|
.where(amount: pending_entry.amount)
|
|
.where(date: pending_entry.date..(pending_entry.date + date_window.days))
|
|
.where(<<~SQL.squish)
|
|
(transactions.extra -> 'simplefin' ->> 'pending')::boolean IS NOT TRUE
|
|
AND (transactions.extra -> 'plaid' ->> 'pending')::boolean IS NOT TRUE
|
|
SQL
|
|
.exists?
|
|
|
|
unless valid_match
|
|
to_restore << pending_entry
|
|
puts "SHOULD RESTORE (no valid match):"
|
|
puts " ID=#{pending_entry.id} | #{pending_entry.date} | #{pending_entry.name} | #{pending_entry.amount} #{pending_entry.currency}"
|
|
puts " Account: #{pending_entry.account.name}"
|
|
puts ""
|
|
end
|
|
end
|
|
|
|
puts "=============================================="
|
|
puts "Summary: #{to_restore.count} transactions should be restored"
|
|
puts ""
|
|
|
|
if to_restore.any?
|
|
if dry_run
|
|
puts "To restore these transactions, run:"
|
|
puts " rails simplefin:pending_restore DRY_RUN=false"
|
|
else
|
|
Entry.where(id: to_restore.map(&:id)).update_all(excluded: false)
|
|
puts "Restored #{to_restore.count} transactions!"
|
|
end
|
|
else
|
|
puts "No wrongly excluded transactions found."
|
|
end
|
|
end
|
|
|
|
desc "List all pending SimpleFIN transactions (for review)"
|
|
task pending_list: :environment do
|
|
pending_entries = Entry.joins(
|
|
"INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'"
|
|
).where(source: "simplefin")
|
|
.where("transactions.extra -> 'simplefin' ->> 'pending' = ?", "true")
|
|
.includes(:account)
|
|
.order(date: :desc)
|
|
|
|
puts "All Pending SimpleFIN Transactions"
|
|
puts "==================================="
|
|
puts "Total: #{pending_entries.count}"
|
|
puts ""
|
|
|
|
pending_entries.find_each do |entry|
|
|
puts "ID=#{entry.id} | #{entry.date} | #{entry.name.truncate(40)} | #{entry.amount} #{entry.currency} | Account: #{entry.account.name}"
|
|
end
|
|
end
|
|
end
|