Add pending transaction handling and duplicate reconciliation logic (#602)

* Add pending transaction handling and duplicate reconciliation logic

- Implemented logic to exclude pending transactions from budgets and analytics calculations.
- Introduced mechanisms for reconciling pending transactions with posted versions.
- Added duplicate detection with support for merging or dismissing matches.
- Updated transaction search filters to include a `status_filter` for pending/confirmed transactions.
- Introduced UI elements for reviewing and resolving duplicates.
- Enhanced `ProviderSyncSummary` with stats for reconciled and stale pending transactions.

* Refactor translation handling and enhance transaction and sync logic

- Moved hardcoded strings to locale files for improved translation support.
- Refined styling for duplicate transaction indicators and sync summaries.
- Improved logic for excluding stale pending transactions and updating timestamps on batch exclusion.
- Added unique IDs to status filters for better element targeting in UI.
- Optimized database queries to avoid N+1 issues in stale pending calculations.

* Add sync settings and enhance pending transaction handling

- Introduced a new "Sync Settings" section in hosting settings with UI to toggle inclusion of pending transactions.
- Updated handling of pending transactions with improved inference logic for `posted=0` and `transacted_at` in processors.
- Added priority order for pending transaction inclusion: explicit argument > environment variable > runtime configurable setting.
- Refactored settings and controllers to store updated sync preferences.

* Refactor sync settings and pending transaction reconciliation

- Extracted logic for pending transaction reconciliation, stale exclusion, and unmatched tracking into dedicated methods for better maintainability.
- Updated sync settings to infer defaults from multiple provider environment variables (`SIMPLEFIN_INCLUDE_PENDING`, `PLAID_INCLUDE_PENDING`).
- Refined UI and messaging to handle multi-provider configurations in sync settings.

# Conflicts:
#	app/models/simplefin_item/importer.rb

* Debounce transaction reconciliation during imports

- Added per-run reconciliation debouncing to prevent repeated scans for the same account during chunked history imports.
- Trimmed size of reconciliation stats to retain recent details only.
- Introduced error tracking for reconciliation steps to improve UI visibility of issues.

* Apply ABS() in pending transaction queries and improve error handling

- Updated pending transaction logic to use ABS() for consistent handling of negative amounts.
- Adjusted amount bounds calculations to ensure accuracy for both positive and negative values.
- Refined exception handling in `merge_duplicate` to log failures and update user alert.
- Replaced `Date.today` with `Date.current` in tests to ensure timezone consistency.
- Minor optimization to avoid COUNT queries by loading limited records directly.

* Improve error handling in duplicate suggestion and dismissal logic

- Added exception handling for `store_duplicate_suggestion` to log failures and prevent crashes during fuzzy/low-confidence matches.
- Enhanced `dismiss_duplicate` action to handle `ActiveRecord::RecordInvalid` and display appropriate user alerts.

---------

Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>

lib/tasks/simplefin_pending_cleanup.rake

@@ -0,0 +1,158 @@
+# frozen_string_literal: true
+
+namespace :simplefin do
+  desc "Find and optionally remove duplicate pending transactions that have matching posted versions"
+  task pending_cleanup: :environment do
+    dry_run = ENV["DRY_RUN"] != "false"
+    date_window = (ENV["DATE_WINDOW"] || 8).to_i
+
+    puts "SimpleFIN Pending Transaction Cleanup"
+    puts "======================================"
+    puts "Mode: #{dry_run ? 'DRY RUN (no changes)' : 'LIVE (will delete duplicates)'}"
+    puts "Date window: #{date_window} days"
+    puts ""
+
+    # Find all pending SimpleFIN transactions
+    pending_entries = Entry.joins(
+      "INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'"
+    ).where(source: "simplefin")
+     .where("transactions.extra -> 'simplefin' ->> 'pending' = ?", "true")
+     .includes(:account)
+
+    puts "Found #{pending_entries.count} pending SimpleFIN transactions"
+    puts ""
+
+    duplicates_found = 0
+    duplicates_to_delete = []
+
+    pending_entries.find_each do |pending_entry|
+      # Look for a matching posted transaction
+      posted_match = Entry.joins(
+        "INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'"
+      ).where(account_id: pending_entry.account_id)
+       .where(source: "simplefin")
+       .where(amount: pending_entry.amount)
+       .where(currency: pending_entry.currency)
+       .where(date: pending_entry.date..(pending_entry.date + date_window.days)) # Posted must be ON or AFTER pending
+       .where.not(id: pending_entry.id)
+       .where("transactions.extra -> 'simplefin' ->> 'pending' != ? OR transactions.extra -> 'simplefin' ->> 'pending' IS NULL", "true")
+       .first
+
+      if posted_match
+        duplicates_found += 1
+        duplicates_to_delete << pending_entry
+
+        puts "DUPLICATE FOUND:"
+        puts "  Pending: ID=#{pending_entry.id} | #{pending_entry.date} | #{pending_entry.name} | #{pending_entry.amount} #{pending_entry.currency}"
+        puts "  Posted:  ID=#{posted_match.id} | #{posted_match.date} | #{posted_match.name} | #{posted_match.amount} #{posted_match.currency}"
+        puts "  Account: #{pending_entry.account.name}"
+        puts ""
+      end
+    end
+
+    puts "======================================"
+    puts "Summary: #{duplicates_found} duplicate pending transactions found"
+    puts ""
+
+    if duplicates_found > 0
+      if dry_run
+        puts "To delete these duplicates, run:"
+        puts "  rails simplefin:pending_cleanup DRY_RUN=false"
+        puts ""
+        puts "To adjust the date matching window (default 8 days):"
+        puts "  rails simplefin:pending_cleanup DATE_WINDOW=14"
+      else
+        print "Deleting #{duplicates_to_delete.count} duplicate pending entries... "
+        Entry.where(id: duplicates_to_delete.map(&:id)).destroy_all
+        puts "Done!"
+      end
+    else
+      puts "No duplicates found. Nothing to clean up."
+    end
+  end
+
+  desc "Un-exclude pending transactions that were wrongly matched (fixes direction bug)"
+  task pending_restore: :environment do
+    dry_run = ENV["DRY_RUN"] != "false"
+    date_window = (ENV["DATE_WINDOW"] || 8).to_i
+
+    puts "Restore Wrongly Excluded Pending Transactions"
+    puts "=============================================="
+    puts "Mode: #{dry_run ? 'DRY RUN (no changes)' : 'LIVE (will restore)'}"
+    puts "Date window: #{date_window} days (forward only)"
+    puts ""
+
+    # Find all EXCLUDED pending transactions (these may have been wrongly excluded)
+    excluded_pending = Entry.joins(
+      "INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'"
+    ).where(excluded: true)
+     .where(<<~SQL.squish)
+       (transactions.extra -> 'simplefin' ->> 'pending')::boolean = true
+       OR (transactions.extra -> 'plaid' ->> 'pending')::boolean = true
+     SQL
+
+    puts "Found #{excluded_pending.count} excluded pending transactions to evaluate"
+    puts ""
+
+    to_restore = []
+
+    excluded_pending.includes(:account).find_each do |pending_entry|
+      # Check if there's a VALID posted match using CORRECT logic (forward-only dates)
+      # Posted date must be ON or AFTER pending date
+      valid_match = pending_entry.account.entries
+        .joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'")
+        .where.not(id: pending_entry.id)
+        .where(currency: pending_entry.currency)
+        .where(amount: pending_entry.amount)
+        .where(date: pending_entry.date..(pending_entry.date + date_window.days))
+        .where(<<~SQL.squish)
+          (transactions.extra -> 'simplefin' ->> 'pending')::boolean IS NOT TRUE
+          AND (transactions.extra -> 'plaid' ->> 'pending')::boolean IS NOT TRUE
+        SQL
+        .exists?
+
+      unless valid_match
+        to_restore << pending_entry
+        puts "SHOULD RESTORE (no valid match):"
+        puts "  ID=#{pending_entry.id} | #{pending_entry.date} | #{pending_entry.name} | #{pending_entry.amount} #{pending_entry.currency}"
+        puts "  Account: #{pending_entry.account.name}"
+        puts ""
+      end
+    end
+
+    puts "=============================================="
+    puts "Summary: #{to_restore.count} transactions should be restored"
+    puts ""
+
+    if to_restore.any?
+      if dry_run
+        puts "To restore these transactions, run:"
+        puts "  rails simplefin:pending_restore DRY_RUN=false"
+      else
+        Entry.where(id: to_restore.map(&:id)).update_all(excluded: false)
+        puts "Restored #{to_restore.count} transactions!"
+      end
+    else
+      puts "No wrongly excluded transactions found."
+    end
+  end
+
+  desc "List all pending SimpleFIN transactions (for review)"
+  task pending_list: :environment do
+    pending_entries = Entry.joins(
+      "INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'"
+    ).where(source: "simplefin")
+     .where("transactions.extra -> 'simplefin' ->> 'pending' = ?", "true")
+     .includes(:account)
+     .order(date: :desc)
+
+    puts "All Pending SimpleFIN Transactions"
+    puts "==================================="
+    puts "Total: #{pending_entries.count}"
+    puts ""
+
+    pending_entries.find_each do |entry|
+      puts "ID=#{entry.id} | #{entry.date} | #{entry.name.truncate(40)} | #{entry.amount} #{entry.currency} | Account: #{entry.account.name}"
+    end
+  end
+end