Files
sure/app/views/simplefin_items/_simplefin_item.html.erb
LPW c12c585a0e Harden SimpleFin sync: retries, safer imports, manual relinking, and data-quality reconciliation (#544)
* Add tests and enhance logic for SimpleFin account synchronization and reconciliation

- Added retry logic with exponential backoff for network errors in `Provider::Simplefin`.
- Introduced tests to verify retry functionality and error handling for rate-limit, server errors, and stale data.
- Updated `SimplefinItem` to detect stale sync status and reconciliation issues.
- Enhanced UI to display stale sync warnings and data integrity notices.
- Improved SimpleFin account matching during updates with multi-tier strategy (ID, fingerprint, fuzzy match).
- Added transaction reconciliation logic to detect data gaps, transaction count drops, and duplicate transaction IDs.

* Introduce `SimplefinConnectionUpdateJob` for asynchronous SimpleFin connection updates

- Moved SimpleFin connection update logic to `SimplefinConnectionUpdateJob` to improve response times by offloading network retries, data fetching, and reconciliation tasks.
- Enhanced SimpleFin account matching with a multi-tier strategy (ID, fingerprint, fuzzy name match).
- Added retry logic and bounded latency for token claim requests in `Provider::Simplefin`.
- Updated tests to cover the new job flow and ensure correct account reconciliation during updates.

* Remove unused SimpleFin account matching logic and improve error handling in `SimplefinConnectionUpdateJob`

- Deleted the multi-tier account matching logic from `SimplefinItemsController` as it is no longer used.
- Enhanced error handling in `SimplefinConnectionUpdateJob` to gracefully handle import failures, ensuring orphaned items can be manually resolved.
- Updated job flow to conditionally set item status based on the success of import operations.

* Fix SimpleFin sync: check both legacy FK and AccountProvider for linked accounts

* Add crypto, checking, savings, and cash account detection; refine subtype selection and linking

- Enhanced `Simplefin::AccountTypeMapper` to include detection for crypto, checking, savings, and standalone cash accounts.
- Improved subtype selection UI with validation and warning indicators for missing selections.
- Updated SimpleFin account linking to handle both legacy FK and `AccountProvider` associations consistently.
- Refined job flow and importer logic for better handling of linked accounts and subtype inference.

* Improve `SimplefinConnectionUpdateJob` and holdings processing logic

- Fixed race condition in `SimplefinConnectionUpdateJob` by moving `destroy_later` calls outside of transactions.
- Updated fuzzy name match logic to use Levenshtein distance for better accuracy.
- Enhanced synthetic ticker generation in holdings processor with hash suffix for uniqueness.

* Refine SimpleFin entry processing logic and ensure `extra` data persistence

- Simplified pending flag determination to rely solely on provider-supplied values.
- Fixed potential stale values in `extra` by ensuring deep merge overwrite with `entry.transaction.save!`.

* Replace hardcoded fallback transaction description with localized string

* Refine pending flag logic in SimpleFin processor tests

- Adjust test to prevent falsely inferring pending status from missing posted dates.
- Ensure provider explicitly sets pending flag for transactions.

* Add `has_many :holdings` association to `AccountProvider` with `dependent: :nullify`

---------

Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
2026-01-05 22:11:47 +01:00

285 lines
14 KiB
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<%# locals: (simplefin_item:) %>
<%= tag.div id: dom_id(simplefin_item) do %>
<details open class="group bg-container p-4 shadow-border-xs rounded-xl">
<summary class="flex items-center justify-between gap-2">
<div class="flex items-center gap-2">
<%= icon "chevron-right", class: "group-open:transform group-open:rotate-90" %>
<div class="flex items-center justify-center h-8 w-8 bg-success/10 rounded-full">
<% if simplefin_item.logo.attached? %>
<%= image_tag simplefin_item.logo, class: "rounded-full h-full w-full", loading: "lazy" %>
<% else %>
<div class="flex items-center justify-center">
<%= tag.p simplefin_item.name.first.upcase, class: "text-success text-xs font-medium" %>
</div>
<% end %>
</div>
<%# Compute unlinked count early for badge display %>
<% header_unlinked_count = if defined?(@simplefin_unlinked_count_map) && @simplefin_unlinked_count_map
@simplefin_unlinked_count_map[simplefin_item.id] || 0
else
begin
simplefin_item.simplefin_accounts
.left_joins(:account, :account_provider)
.where(accounts: { id: nil }, account_providers: { id: nil })
.count
rescue => e
0
end
end %>
<div class="pl-1 text-sm">
<div class="flex items-center gap-2">
<%= tag.p simplefin_item.name, class: "font-medium text-primary" %>
<% if header_unlinked_count.to_i > 0 %>
<%= link_to setup_accounts_simplefin_item_path(simplefin_item),
data: { turbo_frame: :modal },
class: "inline-flex items-center gap-1 px-2 py-0.5 rounded-full text-xs font-medium bg-warning/10 text-warning hover:bg-warning/20 transition-colors" do %>
<%= icon "alert-circle", size: "xs" %>
<span><%= header_unlinked_count %> <%= header_unlinked_count == 1 ? "account" : "accounts" %> need setup</span>
<% end %>
<% end %>
<% if simplefin_item.scheduled_for_deletion? %>
<p class="text-destructive text-sm animate-pulse"><%= t(".deletion_in_progress") %></p>
<% end %>
</div>
<% if simplefin_item.accounts.any? %>
<p class="text-xs text-secondary">
<%= simplefin_item.institution_summary %>
</p>
<%# Extra inline badges from latest sync stats %>
<% stats = (@simplefin_sync_stats_map || {})[simplefin_item.id] || {} %>
<% if stats.present? %>
<div class="flex items-center gap-2 mt-1">
<% if stats["unlinked_accounts"].to_i > 0 %>
<%= render DS::Tooltip.new(text: "Accounts need setup", icon: "link-2", size: "sm") %>
<span class="text-xs text-secondary">Unlinked: <%= stats["unlinked_accounts"].to_i %></span>
<% end %>
<% if stats["accounts_skipped"].to_i > 0 %>
<%= render DS::Tooltip.new(text: "Some accounts were skipped due to errors during sync", icon: "alert-triangle", size: "sm", color: "warning") %>
<span class="text-xs text-warning">Skipped: <%= stats["accounts_skipped"].to_i %></span>
<% end %>
<% if stats["rate_limited"].present? || stats["rate_limited_at"].present? %>
<% ts = stats["rate_limited_at"] %>
<% ago = (ts.present? ? (begin; time_ago_in_words(Time.parse(ts)); rescue StandardError; nil; end) : nil) %>
<%= render DS::Tooltip.new(
text: (ago ? "Rate limited (" + ago + " ago)" : "Rate limited recently"),
icon: "clock",
size: "sm",
color: "warning"
) %>
<% end %>
<% if stats["total_errors"].to_i > 0 || (stats["errors"].is_a?(Array) && stats["errors"].any?) %>
<% tooltip_text = simplefin_error_tooltip(stats) %>
<% if tooltip_text.present? %>
<%= render DS::Tooltip.new(text: tooltip_text, icon: "alert-octagon", size: "sm", color: "warning") %>
<% end %>
<% end %>
<% if stats["total_accounts"].to_i > 0 %>
<span class="text-xs text-secondary">Total: <%= stats["total_accounts"].to_i %></span>
<% end %>
</div>
<% end %>
<% end %>
<%# Determine if all reported errors are benign duplicate-skips (suppress scary banner). Computed in controller for testability. %>
<% duplicate_only_errors = (@simplefin_duplicate_only_map || {})[simplefin_item.id] || false %>
<% if simplefin_item.syncing? %>
<div class="text-secondary flex items-center gap-1">
<%= icon "loader", size: "sm", class: "animate-spin" %>
<%= tag.span t(".syncing") %>
</div>
<% elsif simplefin_item.requires_update? %>
<div class="text-warning flex items-center gap-1">
<%= icon "alert-triangle", size: "sm", color: "warning" %>
<%= tag.span t(".requires_update") %>
</div>
<% elsif (stale_status = simplefin_item.stale_sync_status)[:stale] %>
<div class="text-warning flex items-center gap-1">
<%= icon "alert-circle", size: "sm", color: "warning" %>
<%= tag.span stale_status[:message], class: "text-sm" %>
</div>
<% elsif simplefin_item.rate_limited_message.present? %>
<div class="text-warning flex items-center gap-1">
<%= icon "clock", size: "sm", color: "warning" %>
<%= tag.span simplefin_item.rate_limited_message %>
</div>
<% elsif simplefin_item.sync_error.present? && !duplicate_only_errors %>
<div class="text-secondary flex items-center gap-1">
<%= render DS::Tooltip.new(text: simplefin_item.sync_error, icon: "alert-circle", size: "sm", color: "destructive") %>
<%= tag.span t(".error"), class: "text-destructive" %>
</div>
<% elsif duplicate_only_errors %>
<div class="text-secondary flex items-center gap-1">
<%= icon "info", size: "sm" %>
<%= tag.span "Some accounts were skipped as duplicates — use Link existing accounts to merge.", class: "text-secondary" %>
</div>
<% else %>
<p class="text-secondary">
<% if simplefin_item.last_synced_at %>
<% if simplefin_item.sync_status_summary %>
<%= t(".status_with_summary", timestamp: time_ago_in_words(simplefin_item.last_synced_at), summary: simplefin_item.sync_status_summary) %>
<% else %>
<%= t(".status", timestamp: time_ago_in_words(simplefin_item.last_synced_at)) %>
<% end %>
<% else %>
<%= t(".status_never") %>
<% end %>
</p>
<% end %>
</div>
</div>
<div class="flex items-center gap-2">
<% if simplefin_item.requires_update? %>
<%= render DS::Link.new(
text: t(".update"),
icon: "refresh-cw",
variant: "secondary",
href: edit_simplefin_item_path(simplefin_item),
frame: "modal"
) %>
<% elsif Rails.env.development? %>
<%= icon(
"refresh-cw",
as_button: true,
href: sync_simplefin_item_path(simplefin_item)
) %>
<% end %>
<%= render DS::Menu.new do |menu| %>
<% menu.with_item(
variant: "button",
text: t(".delete"),
icon: "trash-2",
href: simplefin_item_path(simplefin_item),
method: :delete,
confirm: CustomConfirm.for_resource_deletion(simplefin_item.name, high_severity: true)
) %>
<% end %>
</div>
</summary>
<% unless simplefin_item.scheduled_for_deletion? %>
<div class="space-y-4 mt-4">
<% if simplefin_item.accounts.any? %>
<%= render "accounts/index/account_groups", accounts: simplefin_item.accounts %>
<% end %>
<%# Sync summary (collapsible)
Prefer controller-provided map; fallback to latest sync stats so Turbo broadcasts
can render the summary without requiring a full page refresh. %>
<% stats = if defined?(@simplefin_sync_stats_map) && @simplefin_sync_stats_map
@simplefin_sync_stats_map[simplefin_item.id] || {}
else
# `latest_sync` is private on Syncable; access via association for broadcast renders.
simplefin_item.syncs.ordered.first&.sync_stats || {}
end %>
<% if stats.present? %>
<details class="group bg-surface rounded-lg border border-surface-inset/50">
<summary class="flex items-center justify-between gap-2 p-3 cursor-pointer">
<div class="flex items-center gap-2">
<%= icon "chevron-right", class: "group-open:transform group-open:rotate-90" %>
<span class="text-sm text-primary font-medium">Sync summary</span>
</div>
<div class="flex items-center gap-2 text-xs text-secondary">
<% if simplefin_item.last_synced_at %>
<span>Last sync: <%= time_ago_in_words(simplefin_item.last_synced_at) %> ago</span>
<% end %>
</div>
</summary>
<div class="p-3 text-sm text-secondary grid grid-cols-1 md:grid-cols-2 gap-3">
<div>
<h4 class="text-primary font-medium mb-1">Accounts</h4>
<div class="flex items-center gap-3">
<span>Total: <%= stats["total_accounts"].to_i %></span>
<span>Linked: <%= stats["linked_accounts"].to_i %></span>
<span>Unlinked: <%= stats["unlinked_accounts"].to_i %></span>
<% institutions = simplefin_item.connected_institutions %>
<span>Institutions: <%= institutions.size %></span>
</div>
</div>
<div>
<h4 class="text-primary font-medium mb-1">Transactions</h4>
<div class="flex items-center gap-3">
<span>Seen: <%= stats["tx_seen"].to_i %></span>
<span>Imported: <%= stats["tx_imported"].to_i %></span>
<span>Updated: <%= stats["tx_updated"].to_i %></span>
<span>Skipped: <%= stats["tx_skipped"].to_i %></span>
</div>
</div>
<div>
<h4 class="text-primary font-medium mb-1">Holdings</h4>
<div class="flex items-center gap-3">
<span>Found: <%= stats["holdings_found"].to_i %></span>
</div>
</div>
<div>
<h4 class="text-primary font-medium mb-1">Health</h4>
<div class="flex flex-col gap-1">
<div class="flex items-center gap-3">
<% if stats["rate_limited"].present? || stats["rate_limited_at"].present? %>
<% ts = stats["rate_limited_at"] %>
<% ago = (ts.present? ? (begin; time_ago_in_words(Time.parse(ts)); rescue StandardError; nil; end) : nil) %>
<span class="text-warning">Rate limited <%= ago ? "(#{ago} ago)" : "recently" %></span>
<% end %>
<% total_errors = stats["total_errors"].to_i %>
<% import_started = stats["import_started"].present? %>
<% if total_errors > 0 %>
<span class="text-destructive">Errors: <%= total_errors %></span>
<% elsif import_started %>
<span class="text-success">Errors: 0</span>
<% else %>
<span>Errors: 0</span>
<% end %>
</div>
</div>
</div>
</div>
</details>
<% end %>
<%# Compute unlinked SimpleFin accounts (no legacy account and no AccountProvider link)
# Prefer controller-provided map; fallback to a local query so the card stays accurate after Turbo broadcasts %>
<% unlinked_count = if defined?(@simplefin_unlinked_count_map) && @simplefin_unlinked_count_map
@simplefin_unlinked_count_map[simplefin_item.id] || 0
else
begin
simplefin_item.simplefin_accounts
.left_joins(:account, :account_provider)
.where(accounts: { id: nil }, account_providers: { id: nil })
.count
rescue => e
Rails.logger.warn("SimpleFin card: unlinked_count fallback failed: #{e.class} - #{e.message}")
0
end
end %>
<% if unlinked_count.to_i > 0 %>
<div class="p-4 flex flex-col gap-3 items-center justify-center">
<p class="text-primary font-medium text-sm"><%= t(".setup_needed") %></p>
<p class="text-secondary text-sm"><%= t(".setup_description") %></p>
<%= render DS::Link.new(
text: t(".setup_action"),
icon: "settings",
variant: "primary",
href: setup_accounts_simplefin_item_path(simplefin_item),
frame: :modal
) %>
</div>
<% elsif simplefin_item.accounts.empty? %>
<div class="p-4 flex flex-col gap-3 items-center justify-center">
<p class="text-primary font-medium text-sm"><%= t(".no_accounts_title") %></p>
<p class="text-secondary text-sm"><%= t(".no_accounts_description") %></p>
</div>
<% end %>
</div>
<% end %>
</details>
<% end %>