Harden SimpleFin sync: retries, safer imports, manual relinking, and data-quality reconciliation (#544)

* Add tests and enhance logic for SimpleFin account synchronization and reconciliation

- Added retry logic with exponential backoff for network errors in `Provider::Simplefin`.
- Introduced tests to verify retry functionality and error handling for rate-limit, server errors, and stale data.
- Updated `SimplefinItem` to detect stale sync status and reconciliation issues.
- Enhanced UI to display stale sync warnings and data integrity notices.
- Improved SimpleFin account matching during updates with multi-tier strategy (ID, fingerprint, fuzzy match).
- Added transaction reconciliation logic to detect data gaps, transaction count drops, and duplicate transaction IDs.

* Introduce `SimplefinConnectionUpdateJob` for asynchronous SimpleFin connection updates

- Moved SimpleFin connection update logic to `SimplefinConnectionUpdateJob` to improve response times by offloading network retries, data fetching, and reconciliation tasks.
- Enhanced SimpleFin account matching with a multi-tier strategy (ID, fingerprint, fuzzy name match).
- Added retry logic and bounded latency for token claim requests in `Provider::Simplefin`.
- Updated tests to cover the new job flow and ensure correct account reconciliation during updates.

* Remove unused SimpleFin account matching logic and improve error handling in `SimplefinConnectionUpdateJob`

- Deleted the multi-tier account matching logic from `SimplefinItemsController` as it is no longer used.
- Enhanced error handling in `SimplefinConnectionUpdateJob` to gracefully handle import failures, ensuring orphaned items can be manually resolved.
- Updated job flow to conditionally set item status based on the success of import operations.

* Fix SimpleFin sync: check both legacy FK and AccountProvider for linked accounts

* Add crypto, checking, savings, and cash account detection; refine subtype selection and linking

- Enhanced `Simplefin::AccountTypeMapper` to include detection for crypto, checking, savings, and standalone cash accounts.
- Improved subtype selection UI with validation and warning indicators for missing selections.
- Updated SimpleFin account linking to handle both legacy FK and `AccountProvider` associations consistently.
- Refined job flow and importer logic for better handling of linked accounts and subtype inference.

* Improve `SimplefinConnectionUpdateJob` and holdings processing logic

- Fixed race condition in `SimplefinConnectionUpdateJob` by moving `destroy_later` calls outside of transactions.
- Updated fuzzy name match logic to use Levenshtein distance for better accuracy.
- Enhanced synthetic ticker generation in holdings processor with hash suffix for uniqueness.

* Refine SimpleFin entry processing logic and ensure `extra` data persistence

- Simplified pending flag determination to rely solely on provider-supplied values.
- Fixed potential stale values in `extra` by ensuring deep merge overwrite with `entry.transaction.save!`.

* Replace hardcoded fallback transaction description with localized string

* Refine pending flag logic in SimpleFin processor tests

- Adjust test to prevent falsely inferring pending status from missing posted dates.
- Ensure provider explicitly sets pending flag for transactions.

* Add `has_many :holdings` association to `AccountProvider` with `dependent: :nullify`

---------

Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>

app/models/simplefin_item/syncer.rb

@@ -10,7 +10,15 @@ class SimplefinItem::Syncer
     # can review and manually link accounts first. This mirrors the historical flow
     # users expect: initial 7-day balances snapshot, then full chunked history after linking.
     begin
-      if simplefin_item.simplefin_accounts.joins(:account).count == 0
+      # Check for linked accounts via BOTH legacy FK (accounts.simplefin_account_id) AND
+      # the new AccountProvider system. An account is "linked" if either association exists.
+      linked_via_legacy = simplefin_item.simplefin_accounts.joins(:account).count
+      linked_via_provider = simplefin_item.simplefin_accounts.joins(:account_provider).count
+      total_linked = simplefin_item.simplefin_accounts.select { |sfa| sfa.current_account.present? }.count
+
+      Rails.logger.info("SimplefinItem::Syncer - linked check: legacy=#{linked_via_legacy}, provider=#{linked_via_provider}, total=#{total_linked}")
+
+      if total_linked == 0
         sync.update!(status_text: "Discovering accounts (balances only)...") if sync.respond_to?(:status_text)
         # Pre-mark the sync as balances_only for runtime only (no persistence)
         begin
@@ -52,8 +60,9 @@ class SimplefinItem::Syncer
     finalize_setup_counts(sync)
 
     # Process transactions/holdings only for linked accounts
-    linked_accounts = simplefin_item.simplefin_accounts.joins(:account)
-    if linked_accounts.any?
+    # Check both legacy FK and AccountProvider associations
+    linked_simplefin_accounts = simplefin_item.simplefin_accounts.select { |sfa| sfa.current_account.present? }
+    if linked_simplefin_accounts.any?
       sync.update!(status_text: "Processing transactions and holdings...") if sync.respond_to?(:status_text)
       simplefin_item.process_accounts
 
@@ -77,7 +86,11 @@ class SimplefinItem::Syncer
     def finalize_setup_counts(sync)
       sync.update!(status_text: "Checking account configuration...") if sync.respond_to?(:status_text)
       total_accounts = simplefin_item.simplefin_accounts.count
-      linked_accounts = simplefin_item.simplefin_accounts.joins(:account)
+
+      # Count linked accounts using both legacy FK and AccountProvider associations
+      linked_count = simplefin_item.simplefin_accounts.count { |sfa| sfa.current_account.present? }
+
+      # Unlinked = no legacy FK AND no AccountProvider
       unlinked_accounts = simplefin_item.simplefin_accounts
         .left_joins(:account, :account_provider)
         .where(accounts: { id: nil }, account_providers: { id: nil })
@@ -93,7 +106,7 @@ class SimplefinItem::Syncer
         existing = (sync.sync_stats || {})
         setup_stats = {
           "total_accounts" => total_accounts,
-          "linked_accounts" => linked_accounts.count,
+          "linked_accounts" => linked_count,
           "unlinked_accounts" => unlinked_accounts.count
         }
         sync.update!(sync_stats: existing.merge(setup_stats))
@@ -185,7 +198,8 @@ class SimplefinItem::Syncer
       window_start = sync.created_at || 30.minutes.ago
       window_end   = Time.current
 
-      account_ids = simplefin_item.simplefin_accounts.joins(:account).pluck("accounts.id")
+      # Get account IDs via BOTH legacy FK and AccountProvider to ensure we capture all linked accounts
+      account_ids = simplefin_item.simplefin_accounts.filter_map { |sfa| sfa.current_account&.id }
       return {} if account_ids.empty?
 
       tx_scope = Entry.where(account_id: account_ids, source: "simplefin", entryable_type: "Transaction")
@@ -193,14 +207,16 @@ class SimplefinItem::Syncer
       tx_updated  = tx_scope.where(updated_at: window_start..window_end).where.not(created_at: window_start..window_end).count
       tx_seen     = tx_imported + tx_updated
 
-      holdings_scope = Holding.where(account_id: account_ids)
-      holdings_processed = holdings_scope.where(created_at: window_start..window_end).count
+      # Count holdings from raw_holdings_payload (what the sync found) rather than
+      # the database. Holdings are applied asynchronously via SimplefinHoldingsApplyJob,
+      # so database counts would always be 0 at this point.
+      holdings_found = simplefin_item.simplefin_accounts.sum { |sfa| Array(sfa.raw_holdings_payload).size }
 
       {
         "tx_imported" => tx_imported,
         "tx_updated" => tx_updated,
         "tx_seen" => tx_seen,
-        "holdings_processed" => holdings_processed,
+        "holdings_found" => holdings_found,
         "window_start" => window_start,
         "window_end" => window_end
       }