Add pending transaction handling and duplicate reconciliation logic (#602)

* Add pending transaction handling and duplicate reconciliation logic

- Implemented logic to exclude pending transactions from budgets and analytics calculations.
- Introduced mechanisms for reconciling pending transactions with posted versions.
- Added duplicate detection with support for merging or dismissing matches.
- Updated transaction search filters to include a `status_filter` for pending/confirmed transactions.
- Introduced UI elements for reviewing and resolving duplicates.
- Enhanced `ProviderSyncSummary` with stats for reconciled and stale pending transactions.

* Refactor translation handling and enhance transaction and sync logic

- Moved hardcoded strings to locale files for improved translation support.
- Refined styling for duplicate transaction indicators and sync summaries.
- Improved logic for excluding stale pending transactions and updating timestamps on batch exclusion.
- Added unique IDs to status filters for better element targeting in UI.
- Optimized database queries to avoid N+1 issues in stale pending calculations.

* Add sync settings and enhance pending transaction handling

- Introduced a new "Sync Settings" section in hosting settings with UI to toggle inclusion of pending transactions.
- Updated handling of pending transactions with improved inference logic for `posted=0` and `transacted_at` in processors.
- Added priority order for pending transaction inclusion: explicit argument > environment variable > runtime configurable setting.
- Refactored settings and controllers to store updated sync preferences.

* Refactor sync settings and pending transaction reconciliation

- Extracted logic for pending transaction reconciliation, stale exclusion, and unmatched tracking into dedicated methods for better maintainability.
- Updated sync settings to infer defaults from multiple provider environment variables (`SIMPLEFIN_INCLUDE_PENDING`, `PLAID_INCLUDE_PENDING`).
- Refined UI and messaging to handle multi-provider configurations in sync settings.

# Conflicts:
#	app/models/simplefin_item/importer.rb

* Debounce transaction reconciliation during imports

- Added per-run reconciliation debouncing to prevent repeated scans for the same account during chunked history imports.
- Trimmed size of reconciliation stats to retain recent details only.
- Introduced error tracking for reconciliation steps to improve UI visibility of issues.

* Apply ABS() in pending transaction queries and improve error handling

- Updated pending transaction logic to use ABS() for consistent handling of negative amounts.
- Adjusted amount bounds calculations to ensure accuracy for both positive and negative values.
- Refined exception handling in `merge_duplicate` to log failures and update user alert.
- Replaced `Date.today` with `Date.current` in tests to ensure timezone consistency.
- Minor optimization to avoid COUNT queries by loading limited records directly.

* Improve error handling in duplicate suggestion and dismissal logic

- Added exception handling for `store_duplicate_suggestion` to log failures and prevent crashes during fuzzy/low-confidence matches.
- Enhanced `dismiss_duplicate` action to handle `ActiveRecord::RecordInvalid` and display appropriate user alerts.

---------

Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
This commit is contained in:
LPW
2026-01-10 14:11:00 -05:00
committed by GitHub
parent f012e38824
commit 3658e812a8
40 changed files with 2014 additions and 53 deletions

View File

@@ -100,10 +100,14 @@ Two primary data ingestion methods:
- SimpleFIN: pending via `pending: true` or `posted` blank/0 + `transacted_at`.
- Plaid: pending via Plaid `pending: true` (stored at `extra["plaid"]["pending"]` for bank/credit transactions imported via `PlaidEntry::Processor`).
- Storage: provider data on `Transaction#extra` (e.g., `extra["simplefin"]["pending"]`; FX uses `fx_from`, `fx_date`).
- UI: Pending badge when `transaction.pending?` is true; no badge if provider omits pendings.
- Configuration (default-off)
- Centralized in `config/initializers/simplefin.rb` via `Rails.configuration.x.simplefin.*`.
- ENV-backed keys: `SIMPLEFIN_INCLUDE_PENDING=1`, `SIMPLEFIN_DEBUG_RAW=1`.
- UI: "Pending" badge when `transaction.pending?` is true; no badge if provider omits pendings.
- Configuration (default-on for pending)
- SimpleFIN: `config/initializers/simplefin.rb` via `Rails.configuration.x.simplefin.*`.
- Plaid: `config/initializers/plaid_config.rb` via `Rails.configuration.x.plaid.*`.
- Pending transactions are fetched by default and handled via reconciliation/filtering.
- Set `SIMPLEFIN_INCLUDE_PENDING=0` to disable pending fetching for SimpleFIN.
- Set `PLAID_INCLUDE_PENDING=0` to disable pending fetching for Plaid.
- Set `SIMPLEFIN_DEBUG_RAW=1` to enable raw payload debug logging.
Provider support notes:
- SimpleFIN: supports pending + FX metadata (stored under `extra["simplefin"]`).

View File

@@ -46,6 +46,42 @@
"data-auto-submit-form-target": "auto" %>
</div>
</div>
<%= render DS::Menu.new(variant: "button", no_padding: true) do |menu| %>
<% menu.with_button(
id: "activity-status-filter-button",
type: "button",
text: t("accounts.show.activity.filter"),
variant: "outline",
icon: "list-filter"
) %>
<% menu.with_custom_content do %>
<div class="p-3 space-y-3 min-w-[160px]">
<p class="text-xs font-medium text-secondary uppercase"><%= t("accounts.show.activity.status") %></p>
<div class="flex items-center gap-3">
<%= check_box_tag "q[status][]",
"confirmed",
params.dig(:q, :status)&.include?("confirmed"),
id: "q_status_confirmed",
class: "checkbox checkbox--light",
form: "entries-search",
onchange: "document.getElementById('entries-search').requestSubmit()" %>
<%= label_tag "q_status_confirmed", t("accounts.show.activity.confirmed"), class: "text-sm text-primary" %>
</div>
<div class="flex items-center gap-3">
<%= check_box_tag "q[status][]",
"pending",
params.dig(:q, :status)&.include?("pending"),
id: "q_status_pending",
class: "checkbox checkbox--light",
form: "entries-search",
onchange: "document.getElementById('entries-search').requestSubmit()" %>
<%= label_tag "q_status_pending", t("accounts.show.activity.pending"), class: "text-sm text-primary" %>
</div>
</div>
<% end %>
<% end %>
<% end %>
<%= button_tag type: "button",
id: "toggle-checkboxes-button",

View File

@@ -69,6 +69,102 @@
<% end %>
</div>
<%# Pending→posted reconciliation %>
<% if has_pending_reconciled? %>
<div class="mt-1">
<div class="flex items-center gap-1">
<%= helpers.icon "check-circle", size: "sm", color: "success" %>
<span class="text-success"><%= t("provider_sync_summary.health.pending_reconciled", count: pending_reconciled) %></span>
</div>
<% if pending_reconciled_details.any? %>
<details class="mt-1">
<summary class="text-xs cursor-pointer text-secondary hover:text-primary">
<%= t("provider_sync_summary.health.view_reconciled") %>
</summary>
<div class="mt-1 pl-2 border-l-2 border-surface-inset space-y-1">
<% pending_reconciled_details.each do |detail| %>
<p class="text-xs text-success">
<%= detail["account_name"] %>: <%= detail["pending_name"] %>
</p>
<% end %>
</div>
</details>
<% end %>
</div>
<% end %>
<%# Duplicate suggestions needing review %>
<% if has_duplicate_suggestions_created? %>
<div class="mt-1">
<div class="flex items-center gap-1">
<%= helpers.icon "alert-triangle", size: "sm", color: "warning" %>
<span class="text-warning"><%= t("provider_sync_summary.health.duplicate_suggestions", count: duplicate_suggestions_created) %></span>
</div>
<% if duplicate_suggestions_details.any? %>
<details class="mt-1">
<summary class="text-xs cursor-pointer text-secondary hover:text-primary">
<%= t("provider_sync_summary.health.view_duplicate_suggestions") %>
</summary>
<div class="mt-1 pl-2 border-l-2 border-surface-inset space-y-1">
<% duplicate_suggestions_details.each do |detail| %>
<p class="text-xs text-warning">
<%= detail["account_name"] %>: <%= detail["pending_name"] %> → <%= detail["posted_name"] %>
</p>
<% end %>
</div>
</details>
<% end %>
</div>
<% end %>
<%# Stale pending transactions (auto-excluded) %>
<% if has_stale_pending? %>
<div class="mt-1">
<div class="flex items-center gap-1">
<%= helpers.icon "clock", size: "sm", color: "warning" %>
<span class="text-warning"><%= t("provider_sync_summary.health.stale_pending", count: stale_pending_excluded) %></span>
</div>
<% if stale_pending_details.any? %>
<details class="mt-1">
<summary class="text-xs cursor-pointer text-secondary hover:text-primary">
<%= t("provider_sync_summary.health.view_stale_pending") %>
</summary>
<div class="mt-1 pl-2 border-l-2 border-surface-inset space-y-1">
<% stale_pending_details.each do |detail| %>
<p class="text-xs text-warning">
<%= detail["account_name"] %>: <%= t("provider_sync_summary.health.stale_pending_count", count: detail["count"]) %>
</p>
<% end %>
</div>
</details>
<% end %>
</div>
<% end %>
<%# Stale unmatched pending (need manual review) %>
<% if has_stale_unmatched_pending? %>
<div class="mt-1">
<div class="flex items-center gap-1">
<%= helpers.icon "help-circle", size: "sm" %>
<span class="text-secondary"><%= t("provider_sync_summary.health.stale_unmatched", count: stale_unmatched_pending) %></span>
</div>
<% if stale_unmatched_details.any? %>
<details class="mt-1">
<summary class="text-xs cursor-pointer text-secondary hover:text-primary">
<%= t("provider_sync_summary.health.view_stale_unmatched") %>
</summary>
<div class="mt-1 pl-2 border-l-2 border-surface-inset space-y-1">
<% stale_unmatched_details.each do |detail| %>
<p class="text-xs text-secondary">
<%= detail["account_name"] %>: <%= t("provider_sync_summary.health.stale_unmatched_count", count: detail["count"]) %>
</p>
<% end %>
</div>
</details>
<% end %>
</div>
<% end %>
<%# Data quality warnings %>
<% if has_data_quality_issues? %>
<div class="flex items-center gap-3 mt-1">

View File

@@ -127,6 +127,58 @@ class ProviderSyncSummary < ViewComponent::Base
total_errors > 0
end
# Stale pending transactions (auto-excluded)
def stale_pending_excluded
stats["stale_pending_excluded"].to_i
end
def has_stale_pending?
stale_pending_excluded > 0
end
def stale_pending_details
stats["stale_pending_details"] || []
end
# Stale unmatched pending (need manual review - couldn't be automatically matched)
def stale_unmatched_pending
stats["stale_unmatched_pending"].to_i
end
def has_stale_unmatched_pending?
stale_unmatched_pending > 0
end
def stale_unmatched_details
stats["stale_unmatched_details"] || []
end
# Pending→posted reconciliation stats
def pending_reconciled
stats["pending_reconciled"].to_i
end
def has_pending_reconciled?
pending_reconciled > 0
end
def pending_reconciled_details
stats["pending_reconciled_details"] || []
end
# Duplicate suggestions needing user review
def duplicate_suggestions_created
stats["duplicate_suggestions_created"].to_i
end
def has_duplicate_suggestions_created?
duplicate_suggestions_created > 0
end
def duplicate_suggestions_details
stats["duplicate_suggestions_details"] || []
end
# Data quality / warnings
def data_warnings
stats["data_warnings"].to_i

View File

@@ -35,8 +35,8 @@ class AccountsController < ApplicationController
def show
@chart_view = params[:chart_view] || "balance"
@tab = params[:tab]
@q = params.fetch(:q, {}).permit(:search)
entries = @account.entries.search(@q).reverse_chronological
@q = params.fetch(:q, {}).permit(:search, status: [])
entries = @account.entries.where(excluded: false).search(@q).reverse_chronological
@pagy, @entries = pagy(entries, limit: params[:per_page] || "10")

View File

@@ -58,6 +58,10 @@ class Settings::HostingsController < ApplicationController
Setting.securities_provider = hosting_params[:securities_provider]
end
if hosting_params.key?(:syncs_include_pending)
Setting.syncs_include_pending = hosting_params[:syncs_include_pending] == "1"
end
if hosting_params.key?(:openai_access_token)
token_param = hosting_params[:openai_access_token].to_s.strip
# Ignore blanks and redaction placeholders to prevent accidental overwrite
@@ -99,7 +103,7 @@ class Settings::HostingsController < ApplicationController
private
def hosting_params
params.require(:setting).permit(:onboarding_state, :require_email_confirmation, :brand_fetch_client_id, :twelve_data_api_key, :openai_access_token, :openai_uri_base, :openai_model, :openai_json_mode, :exchange_rate_provider, :securities_provider)
params.require(:setting).permit(:onboarding_state, :require_email_confirmation, :brand_fetch_client_id, :twelve_data_api_key, :openai_access_token, :openai_uri_base, :openai_model, :openai_json_mode, :exchange_rate_provider, :securities_provider, :syncs_include_pending)
end
def ensure_admin

View File

@@ -116,6 +116,38 @@ class TransactionsController < ApplicationController
end
end
def merge_duplicate
transaction = Current.family.transactions.includes(entry: :account).find(params[:id])
if transaction.merge_with_duplicate!
flash[:notice] = t("transactions.merge_duplicate.success")
else
flash[:alert] = t("transactions.merge_duplicate.failure")
end
redirect_to transactions_path
rescue ActiveRecord::RecordNotDestroyed, ActiveRecord::RecordInvalid => e
Rails.logger.error("Failed to merge duplicate transaction #{params[:id]}: #{e.message}")
flash[:alert] = t("transactions.merge_duplicate.failure")
redirect_to transactions_path
end
def dismiss_duplicate
transaction = Current.family.transactions.includes(entry: :account).find(params[:id])
if transaction.dismiss_duplicate_suggestion!
flash[:notice] = t("transactions.dismiss_duplicate.success")
else
flash[:alert] = t("transactions.dismiss_duplicate.failure")
end
redirect_back_or_to transactions_path
rescue ActiveRecord::RecordInvalid => e
Rails.logger.error("Failed to dismiss duplicate suggestion for transaction #{params[:id]}: #{e.message}")
flash[:alert] = t("transactions.dismiss_duplicate.failure")
redirect_back_or_to transactions_path
end
def mark_as_recurring
transaction = Current.family.transactions.includes(entry: :account).find(params[:id])
@@ -205,7 +237,7 @@ class TransactionsController < ApplicationController
:start_date, :end_date, :search, :amount,
:amount_operator, :active_accounts_only,
accounts: [], account_ids: [],
categories: [], merchants: [], types: [], tags: []
categories: [], merchants: [], types: [], tags: [], status: []
)
.to_h
.compact_blank

View File

@@ -1,13 +1,14 @@
module TransactionsHelper
def transaction_search_filters
[
{ key: "account_filter", label: "Account", icon: "layers" },
{ key: "date_filter", label: "Date", icon: "calendar" },
{ key: "type_filter", label: "Type", icon: "tag" },
{ key: "amount_filter", label: "Amount", icon: "hash" },
{ key: "category_filter", label: "Category", icon: "shapes" },
{ key: "tag_filter", label: "Tag", icon: "tags" },
{ key: "merchant_filter", label: "Merchant", icon: "store" }
{ key: "account_filter", label: t("transactions.search.filters.account"), icon: "layers" },
{ key: "date_filter", label: t("transactions.search.filters.date"), icon: "calendar" },
{ key: "type_filter", label: t("transactions.search.filters.type"), icon: "tag" },
{ key: "status_filter", label: t("transactions.search.filters.status"), icon: "clock" },
{ key: "amount_filter", label: t("transactions.search.filters.amount"), icon: "hash" },
{ key: "category_filter", label: t("transactions.search.filters.category"), icon: "shapes" },
{ key: "tag_filter", label: t("transactions.search.filters.tag"), icon: "tags" },
{ key: "merchant_filter", label: t("transactions.search.filters.merchant"), icon: "store" }
]
end

View File

@@ -16,9 +16,10 @@ class Account::ProviderImportAdapter
# @param category_id [Integer, nil] Optional category ID
# @param merchant [Merchant, nil] Optional merchant object
# @param notes [String, nil] Optional transaction notes/memo
# @param pending_transaction_id [String, nil] Plaid's linking ID for pending→posted reconciliation
# @param extra [Hash, nil] Optional provider-specific metadata to merge into transaction.extra
# @return [Entry] The created or updated entry
def import_transaction(external_id:, amount:, currency:, date:, name:, source:, category_id: nil, merchant: nil, notes: nil, extra: nil)
def import_transaction(external_id:, amount:, currency:, date:, name:, source:, category_id: nil, merchant: nil, notes: nil, pending_transaction_id: nil, extra: nil)
raise ArgumentError, "external_id is required" if external_id.blank?
raise ArgumentError, "source is required" if source.blank?
@@ -43,6 +44,42 @@ class Account::ProviderImportAdapter
end
end
# If still a new entry and this is a POSTED transaction, check for matching pending transactions
incoming_pending = extra.is_a?(Hash) && (
ActiveModel::Type::Boolean.new.cast(extra.dig("simplefin", "pending")) ||
ActiveModel::Type::Boolean.new.cast(extra.dig("plaid", "pending"))
)
if entry.new_record? && !incoming_pending
pending_match = nil
# PRIORITY 1: Use Plaid's pending_transaction_id if provided (most reliable)
# Plaid explicitly links pending→posted with this ID - no guessing required
if pending_transaction_id.present?
pending_match = account.entries.find_by(external_id: pending_transaction_id, source: source)
if pending_match
Rails.logger.info("Reconciling pending→posted via Plaid pending_transaction_id: claiming entry #{pending_match.id} (#{pending_match.name}) with new external_id #{external_id}")
end
end
# PRIORITY 2: Fallback to EXACT amount match (for SimpleFIN and providers without linking IDs)
# Only searches backward in time - pending date must be <= posted date
if pending_match.nil?
pending_match = find_pending_transaction(date: date, amount: amount, currency: currency, source: source)
if pending_match
Rails.logger.info("Reconciling pending→posted via exact amount match: claiming entry #{pending_match.id} (#{pending_match.name}) with new external_id #{external_id}")
end
end
if pending_match
entry = pending_match
entry.assign_attributes(external_id: external_id)
end
end
# Track if this is a new posted transaction (for fuzzy suggestion after save)
is_new_posted = entry.new_record? && !incoming_pending
# Validate entryable type matches to prevent external_id collisions
if entry.persisted? && !entry.entryable.is_a?(Transaction)
raise ArgumentError, "Entry with external_id '#{external_id}' already exists with different entryable type: #{entry.entryable_type}"
@@ -78,6 +115,60 @@ class Account::ProviderImportAdapter
entry.transaction.save!
end
entry.save!
# AFTER save: For NEW posted transactions, check for fuzzy matches to SUGGEST (not auto-claim)
# This handles tip adjustments where auto-matching is too risky
if is_new_posted
# PRIORITY 1: Try medium-confidence fuzzy match (≤30% amount difference)
fuzzy_suggestion = find_pending_transaction_fuzzy(
date: date,
amount: amount,
currency: currency,
source: source,
merchant_id: merchant&.id,
name: name
)
if fuzzy_suggestion
# Store suggestion on the PENDING entry for user to review
begin
store_duplicate_suggestion(
pending_entry: fuzzy_suggestion,
posted_entry: entry,
reason: "fuzzy_amount_match",
posted_amount: amount,
confidence: "medium"
)
Rails.logger.info("Suggested potential duplicate (medium confidence): pending entry #{fuzzy_suggestion.id} (#{fuzzy_suggestion.name}, #{fuzzy_suggestion.amount}) may match posted #{entry.name} (#{amount})")
rescue ActiveRecord::RecordInvalid => e
Rails.logger.warn("Failed to store duplicate suggestion for entry #{fuzzy_suggestion.id}: #{e.message}")
end
else
# PRIORITY 2: Try low-confidence match (>30% to 100% difference - big tips)
low_confidence_suggestion = find_pending_transaction_low_confidence(
date: date,
amount: amount,
currency: currency,
source: source,
merchant_id: merchant&.id,
name: name
)
if low_confidence_suggestion
begin
store_duplicate_suggestion(
pending_entry: low_confidence_suggestion,
posted_entry: entry,
reason: "low_confidence_match",
posted_amount: amount,
confidence: "low"
)
Rails.logger.info("Suggested potential duplicate (low confidence): pending entry #{low_confidence_suggestion.id} (#{low_confidence_suggestion.name}, #{low_confidence_suggestion.amount}) may match posted #{entry.name} (#{amount})")
rescue ActiveRecord::RecordInvalid => e
Rails.logger.warn("Failed to store duplicate suggestion for entry #{low_confidence_suggestion.id}: #{e.message}")
end
end
end
end
entry
end
end
@@ -444,4 +535,196 @@ class Account::ProviderImportAdapter
query.order(created_at: :asc).first
end
# Finds a pending transaction that likely matches a newly posted transaction
# Used to reconcile pending→posted when SimpleFIN gives different IDs for the same transaction
#
# @param date [Date, String] Posted transaction date
# @param amount [BigDecimal, Numeric] Transaction amount (must match exactly)
# @param currency [String] Currency code
# @param source [String] Provider name (e.g., "simplefin")
# @param date_window [Integer] Days to search around the posted date (default: 8)
# @return [Entry, nil] The pending entry or nil if not found
def find_pending_transaction(date:, amount:, currency:, source:, date_window: 8)
date = Date.parse(date.to_s) unless date.is_a?(Date)
# Look for entries that:
# 1. Same account (implicit via account.entries)
# 2. Same source (simplefin)
# 3. Same amount (exact match - this is the strongest signal)
# 4. Same currency
# 5. Date within window (pending can post days later)
# 6. Is a Transaction (not Trade or Valuation)
# 7. Has pending=true in transaction.extra["simplefin"]["pending"] or extra["plaid"]["pending"]
candidates = account.entries
.joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'")
.where(source: source)
.where(amount: amount)
.where(currency: currency)
.where(date: (date - date_window.days)..date) # Pending must be ON or BEFORE posted date
.where(<<~SQL.squish)
(transactions.extra -> 'simplefin' ->> 'pending')::boolean = true
OR (transactions.extra -> 'plaid' ->> 'pending')::boolean = true
SQL
.order(date: :desc) # Prefer most recent pending transaction
candidates.first
end
# Finds a pending transaction using fuzzy amount matching for tip adjustments
# Used when exact amount matching fails - handles restaurant tips, adjusted authorizations, etc.
#
# IMPORTANT: Only returns a match if there's exactly ONE candidate to avoid false positives
# with recurring merchant transactions (e.g., gas stations, coffee shops).
#
# @param date [Date, String] Posted transaction date
# @param amount [BigDecimal, Numeric] Posted transaction amount (typically higher due to tip)
# @param currency [String] Currency code
# @param source [String] Provider name (e.g., "simplefin")
# @param merchant_id [Integer, nil] Merchant ID for more accurate matching
# @param name [String, nil] Transaction name for fuzzy name matching
# @param date_window [Integer] Days to search backward from posted date (default: 3 for fuzzy)
# @param amount_tolerance [Float] Maximum percentage difference allowed (default: 0.30 = 30%)
# @return [Entry, nil] The pending entry or nil if not found/ambiguous
def find_pending_transaction_fuzzy(date:, amount:, currency:, source:, merchant_id: nil, name: nil, date_window: 3, amount_tolerance: 0.30)
date = Date.parse(date.to_s) unless date.is_a?(Date)
amount = BigDecimal(amount.to_s)
# Calculate amount bounds using ABS to handle both positive and negative amounts
# Posted amount should be >= pending (tips add, not subtract)
# Allow posted to be up to 30% higher than pending (covers typical tips)
abs_amount = amount.abs
min_pending_abs = abs_amount / (1 + amount_tolerance) # If posted is 100, pending could be as low as ~77
max_pending_abs = abs_amount # Pending should not be higher than posted
# Build base query for pending transactions
# CRITICAL: Pending must be ON or BEFORE the posted date (authorization happens first)
# Use tighter date window (3 days) - tips post quickly, not a week later
# Use ABS() for amount comparison to handle negative amounts correctly
candidates = account.entries
.joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'")
.where(source: source)
.where(currency: currency)
.where(date: (date - date_window.days)..date) # Pending ON or BEFORE posted
.where("ABS(entries.amount) BETWEEN ? AND ?", min_pending_abs, max_pending_abs)
.where(<<~SQL.squish)
(transactions.extra -> 'simplefin' ->> 'pending')::boolean = true
OR (transactions.extra -> 'plaid' ->> 'pending')::boolean = true
SQL
# If merchant_id is provided, prioritize matching by merchant
if merchant_id.present?
merchant_matches = candidates.where("transactions.merchant_id = ?", merchant_id).to_a
# Only match if exactly ONE candidate to avoid false positives
return merchant_matches.first if merchant_matches.size == 1
if merchant_matches.size > 1
Rails.logger.info("Skipping fuzzy pending match: #{merchant_matches.size} ambiguous merchant candidates for amount=#{amount} date=#{date}")
end
end
# If name is provided, try fuzzy name matching as fallback
if name.present?
# Extract first few significant words for comparison
name_words = name.downcase.gsub(/[^a-z0-9\s]/, "").split.first(3).join(" ")
if name_words.present?
name_matches = candidates.select do |c|
c_name_words = c.name.downcase.gsub(/[^a-z0-9\s]/, "").split.first(3).join(" ")
name_words == c_name_words
end
# Only match if exactly ONE candidate to avoid false positives
return name_matches.first if name_matches.size == 1
if name_matches.size > 1
Rails.logger.info("Skipping fuzzy pending match: #{name_matches.size} ambiguous name candidates for '#{name_words}' amount=#{amount} date=#{date}")
end
end
end
# No merchant or name match, return nil (too risky to match on amount alone)
# This prevents false positives when multiple pending transactions exist
nil
end
# Finds a pending transaction with low confidence (>30% to 100% amount difference)
# Used for large tip scenarios where normal fuzzy matching would miss
# Creates a "review recommended" suggestion rather than "possible duplicate"
#
# @param date [Date, String] Posted transaction date
# @param amount [BigDecimal, Numeric] Posted transaction amount
# @param currency [String] Currency code
# @param source [String] Provider name
# @param merchant_id [Integer, nil] Merchant ID for matching
# @param name [String, nil] Transaction name for matching
# @param date_window [Integer] Days to search backward (default: 3)
# @return [Entry, nil] The pending entry or nil if not found/ambiguous
def find_pending_transaction_low_confidence(date:, amount:, currency:, source:, merchant_id: nil, name: nil, date_window: 3)
date = Date.parse(date.to_s) unless date.is_a?(Date)
amount = BigDecimal(amount.to_s)
# Allow up to 100% difference (e.g., $50 pending → $100 posted with huge tip)
# This is low confidence - requires strong name/merchant match
# Use ABS to handle both positive and negative amounts correctly
abs_amount = amount.abs
min_pending_abs = abs_amount / 2.0 # Posted could be up to 2x pending
max_pending_abs = abs_amount * 0.77 # Pending must be at least 30% less (to not overlap with fuzzy)
# Build base query for pending transactions
# Use ABS() for amount comparison to handle negative amounts correctly
candidates = account.entries
.joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'")
.where(source: source)
.where(currency: currency)
.where(date: (date - date_window.days)..date)
.where("ABS(entries.amount) BETWEEN ? AND ?", min_pending_abs, max_pending_abs)
.where(<<~SQL.squish)
(transactions.extra -> 'simplefin' ->> 'pending')::boolean = true
OR (transactions.extra -> 'plaid' ->> 'pending')::boolean = true
SQL
# For low confidence, require BOTH merchant AND name match (stronger signal needed)
if merchant_id.present? && name.present?
name_words = name.downcase.gsub(/[^a-z0-9\s]/, "").split.first(3).join(" ")
return nil if name_words.blank?
merchant_matches = candidates.where("transactions.merchant_id = ?", merchant_id).to_a
name_matches = merchant_matches.select do |c|
c_name_words = c.name.downcase.gsub(/[^a-z0-9\s]/, "").split.first(3).join(" ")
name_words == c_name_words
end
# Only match if exactly ONE candidate
return name_matches.first if name_matches.size == 1
end
nil
end
# Stores a duplicate suggestion on a pending entry for user review
# The suggestion is stored in the pending transaction's extra field
#
# @param pending_entry [Entry] The pending entry that may be a duplicate
# @param posted_entry [Entry] The posted entry it may match
# @param reason [String] Why this was flagged (e.g., "fuzzy_amount_match", "low_confidence_match")
# @param posted_amount [BigDecimal] The posted transaction amount
# @param confidence [String] Confidence level: "medium" (≤30% diff) or "low" (>30% diff)
def store_duplicate_suggestion(pending_entry:, posted_entry:, reason:, posted_amount:, confidence: "medium")
return unless pending_entry&.entryable.is_a?(Transaction)
pending_transaction = pending_entry.entryable
existing_extra = pending_transaction.extra || {}
# Don't overwrite if already has a suggestion (keep first one found)
return if existing_extra["potential_posted_match"].present?
pending_transaction.update!(
extra: existing_extra.merge(
"potential_posted_match" => {
"entry_id" => posted_entry.id,
"reason" => reason,
"posted_amount" => posted_amount.to_s,
"confidence" => confidence,
"detected_at" => Date.current.to_s
}
)
)
end
end

View File

@@ -35,6 +35,188 @@ class Entry < ApplicationRecord
)
}
# Pending transaction scopes - check Transaction.extra for provider pending flags
# Works with any provider that stores pending status in extra["provider_name"]["pending"]
scope :pending, -> {
joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'")
.where(<<~SQL.squish)
(transactions.extra -> 'simplefin' ->> 'pending')::boolean = true
OR (transactions.extra -> 'plaid' ->> 'pending')::boolean = true
SQL
}
scope :excluding_pending, -> {
# For non-Transaction entries (Trade, Valuation), always include
# For Transaction entries, exclude if pending flag is true
where(<<~SQL.squish)
entries.entryable_type != 'Transaction'
OR NOT EXISTS (
SELECT 1 FROM transactions t
WHERE t.id = entries.entryable_id
AND (
(t.extra -> 'simplefin' ->> 'pending')::boolean = true
OR (t.extra -> 'plaid' ->> 'pending')::boolean = true
)
)
SQL
}
# Find stale pending transactions (pending for more than X days with no matching posted version)
scope :stale_pending, ->(days: 8) {
pending.where("entries.date < ?", days.days.ago.to_date)
}
# Auto-exclude stale pending transactions for an account
# Called during sync to clean up pending transactions that never posted
# @param account [Account] The account to clean up
# @param days [Integer] Number of days after which pending is considered stale (default: 8)
# @return [Integer] Number of entries excluded
def self.auto_exclude_stale_pending(account:, days: 8)
stale_entries = account.entries.stale_pending(days: days).where(excluded: false)
count = stale_entries.count
if count > 0
stale_entries.update_all(excluded: true, updated_at: Time.current)
Rails.logger.info("Auto-excluded #{count} stale pending transaction(s) for account #{account.id} (#{account.name})")
end
count
end
# Retroactively reconcile pending transactions that have a matching posted version
# This handles duplicates created before reconciliation code was deployed
#
# @param account [Account, nil] Specific account to clean up, or nil for all accounts
# @param dry_run [Boolean] If true, only report what would be done without making changes
# @param date_window [Integer] Days to search forward for posted matches (default: 8)
# @param amount_tolerance [Float] Percentage difference allowed for fuzzy matching (default: 0.25)
# @return [Hash] Stats about what was reconciled
def self.reconcile_pending_duplicates(account: nil, dry_run: false, date_window: 8, amount_tolerance: 0.25)
stats = { checked: 0, reconciled: 0, details: [] }
# Get pending entries to check
scope = Entry.pending.where(excluded: false)
scope = scope.where(account: account) if account
scope.includes(:account, :entryable).find_each do |pending_entry|
stats[:checked] += 1
acct = pending_entry.account
# PRIORITY 1: Look for posted transaction with EXACT amount match
# CRITICAL: Only search forward in time - posted date must be >= pending date
exact_candidates = acct.entries
.joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'")
.where.not(id: pending_entry.id)
.where(currency: pending_entry.currency)
.where(amount: pending_entry.amount)
.where(date: pending_entry.date..(pending_entry.date + date_window.days)) # Posted must be ON or AFTER pending date
.where(<<~SQL.squish)
(transactions.extra -> 'simplefin' ->> 'pending')::boolean IS NOT TRUE
AND (transactions.extra -> 'plaid' ->> 'pending')::boolean IS NOT TRUE
SQL
.limit(2) # Only need to know if 0, 1, or 2+ candidates
.to_a # Load limited records to avoid COUNT(*) on .size
# Handle exact match - auto-exclude only if exactly ONE candidate (high confidence)
# Multiple candidates = ambiguous = skip to avoid excluding wrong entry
if exact_candidates.size == 1
posted_match = exact_candidates.first
detail = {
pending_id: pending_entry.id,
pending_name: pending_entry.name,
pending_amount: pending_entry.amount.to_f,
pending_date: pending_entry.date,
posted_id: posted_match.id,
posted_name: posted_match.name,
posted_amount: posted_match.amount.to_f,
posted_date: posted_match.date,
account: acct.name,
match_type: "exact"
}
stats[:details] << detail
stats[:reconciled] += 1
unless dry_run
pending_entry.update!(excluded: true)
Rails.logger.info("Reconciled pending→posted duplicate: excluded entry #{pending_entry.id} (#{pending_entry.name}) matched to #{posted_match.id}")
end
next
end
# PRIORITY 2: If no exact match, try fuzzy amount match for tip adjustments
# Store as SUGGESTION instead of auto-excluding (medium confidence)
pending_amount = pending_entry.amount.abs
min_amount = pending_amount
max_amount = pending_amount * (1 + amount_tolerance)
fuzzy_date_window = 3
candidates = acct.entries
.joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'")
.where.not(id: pending_entry.id)
.where(currency: pending_entry.currency)
.where(date: pending_entry.date..(pending_entry.date + fuzzy_date_window.days)) # Posted ON or AFTER pending
.where("ABS(entries.amount) BETWEEN ? AND ?", min_amount, max_amount)
.where(<<~SQL.squish)
(transactions.extra -> 'simplefin' ->> 'pending')::boolean IS NOT TRUE
AND (transactions.extra -> 'plaid' ->> 'pending')::boolean IS NOT TRUE
SQL
# Match by name similarity (first 3 words)
name_words = pending_entry.name.downcase.gsub(/[^a-z0-9\s]/, "").split.first(3).join(" ")
if name_words.present?
matching_candidates = candidates.select do |c|
c_words = c.name.downcase.gsub(/[^a-z0-9\s]/, "").split.first(3).join(" ")
name_words == c_words
end
# Only suggest if there's exactly ONE matching candidate
# Multiple matches = ambiguous (e.g., recurring gas station visits) = skip
if matching_candidates.size == 1
fuzzy_match = matching_candidates.first
detail = {
pending_id: pending_entry.id,
pending_name: pending_entry.name,
pending_amount: pending_entry.amount.to_f,
pending_date: pending_entry.date,
posted_id: fuzzy_match.id,
posted_name: fuzzy_match.name,
posted_amount: fuzzy_match.amount.to_f,
posted_date: fuzzy_match.date,
account: acct.name,
match_type: "fuzzy_suggestion"
}
stats[:details] << detail
unless dry_run
# Store suggestion on the pending entry instead of auto-excluding
pending_transaction = pending_entry.entryable
if pending_transaction.is_a?(Transaction)
existing_extra = pending_transaction.extra || {}
unless existing_extra["potential_posted_match"].present?
pending_transaction.update!(
extra: existing_extra.merge(
"potential_posted_match" => {
"entry_id" => fuzzy_match.id,
"reason" => "fuzzy_amount_match",
"posted_amount" => fuzzy_match.amount.to_s,
"detected_at" => Date.current.to_s
}
)
)
Rails.logger.info("Stored duplicate suggestion for entry #{pending_entry.id} (#{pending_entry.name}) → #{fuzzy_match.id}")
end
end
end
elsif matching_candidates.size > 1
Rails.logger.info("Skipping fuzzy reconciliation for #{pending_entry.id} (#{pending_entry.name}): #{matching_candidates.size} ambiguous candidates")
end
end
end
stats
end
def classification
amount.negative? ? "income" : "expense"
end

View File

@@ -6,6 +6,7 @@ class EntrySearch
attribute :amount, :string
attribute :amount_operator, :string
attribute :types, :string
attribute :status, array: true
attribute :accounts, array: true
attribute :account_ids, array: true
attribute :start_date, :string
@@ -56,6 +57,44 @@ class EntrySearch
query = query.where(accounts: { id: account_ids }) if account_ids.present?
query
end
def apply_status_filter(scope, statuses)
return scope unless statuses.present?
return scope if statuses.uniq.sort == %w[confirmed pending] # Both selected = no filter
pending_condition = <<~SQL.squish
entries.entryable_type = 'Transaction'
AND EXISTS (
SELECT 1 FROM transactions t
WHERE t.id = entries.entryable_id
AND (
(t.extra -> 'simplefin' ->> 'pending')::boolean = true
OR (t.extra -> 'plaid' ->> 'pending')::boolean = true
)
)
SQL
confirmed_condition = <<~SQL.squish
entries.entryable_type != 'Transaction'
OR NOT EXISTS (
SELECT 1 FROM transactions t
WHERE t.id = entries.entryable_id
AND (
(t.extra -> 'simplefin' ->> 'pending')::boolean = true
OR (t.extra -> 'plaid' ->> 'pending')::boolean = true
)
)
SQL
case statuses.sort
when [ "pending" ]
scope.where(pending_condition)
when [ "confirmed" ]
scope.where(confirmed_condition)
else
scope
end
end
end
def build_query(scope)
@@ -64,6 +103,7 @@ class EntrySearch
query = self.class.apply_date_filters(query, start_date, end_date)
query = self.class.apply_amount_filter(query, amount, amount_operator)
query = self.class.apply_accounts_filter(query, accounts, account_ids)
query = self.class.apply_status_filter(query, status)
query
end
end

View File

@@ -12,7 +12,9 @@ class IncomeStatement
end
def totals(transactions_scope: nil, date_range:)
transactions_scope ||= family.transactions.visible
# Default to excluding pending transactions from budget/analytics calculations
# Pending transactions shouldn't affect budget totals until they post
transactions_scope ||= family.transactions.visible.excluding_pending
result = totals_query(transactions_scope: transactions_scope, date_range: date_range)
@@ -64,7 +66,8 @@ class IncomeStatement
end
def build_period_total(classification:, period:)
totals = totals_query(transactions_scope: family.transactions.visible.in_period(period), date_range: period.date_range).select { |t| t.classification == classification }
# Exclude pending transactions from budget calculations
totals = totals_query(transactions_scope: family.transactions.visible.excluding_pending.in_period(period), date_range: period.date_range).select { |t| t.classification == classification }
classification_total = totals.sum(&:total)
uncategorized_category = family.categories.uncategorized

View File

@@ -49,6 +49,8 @@ class IncomeStatement::CategoryStats
WHERE a.family_id = :family_id
AND t.kind NOT IN ('funds_movement', 'one_time', 'cc_payment')
AND ae.excluded = false
AND (t.extra -> 'simplefin' ->> 'pending')::boolean IS DISTINCT FROM true
AND (t.extra -> 'plaid' ->> 'pending')::boolean IS DISTINCT FROM true
GROUP BY c.id, period, CASE WHEN ae.amount < 0 THEN 'income' ELSE 'expense' END
)
SELECT

View File

@@ -46,6 +46,8 @@ class IncomeStatement::FamilyStats
WHERE a.family_id = :family_id
AND t.kind NOT IN ('funds_movement', 'one_time', 'cc_payment')
AND ae.excluded = false
AND (t.extra -> 'simplefin' ->> 'pending')::boolean IS DISTINCT FROM true
AND (t.extra -> 'plaid' ->> 'pending')::boolean IS DISTINCT FROM true
GROUP BY period, CASE WHEN ae.amount < 0 THEN 'income' ELSE 'expense' END
)
SELECT

View File

@@ -51,7 +51,20 @@ class PlaidAccount::Transactions::Processor
modified = plaid_account.raw_transactions_payload["modified"] || []
added = plaid_account.raw_transactions_payload["added"] || []
modified + added
transactions = modified + added
# Filter out pending transactions based on env var or Setting
# Priority: env var > Setting (allows runtime changes via UI)
include_pending = if ENV["PLAID_INCLUDE_PENDING"].present?
Rails.configuration.x.plaid.include_pending
else
Setting.syncs_include_pending
end
unless include_pending
transactions = transactions.reject { |t| t["pending"] == true }
end
transactions
end
def removed_transactions

View File

@@ -16,9 +16,11 @@ class PlaidEntry::Processor
source: "plaid",
category_id: matched_category&.id,
merchant: merchant,
pending_transaction_id: pending_transaction_id, # Plaid's linking ID for pending→posted
extra: {
plaid: {
pending: plaid_transaction["pending"]
pending: plaid_transaction["pending"],
pending_transaction_id: pending_transaction_id # Also store for reference
}
}
)
@@ -55,6 +57,12 @@ class PlaidEntry::Processor
plaid_transaction["date"]
end
# Plaid provides this linking ID when a posted transaction matches a pending one
# This is the most reliable way to reconcile pending→posted
def pending_transaction_id
plaid_transaction["pending_transaction_id"]
end
def detailed_category
plaid_transaction.dig("personal_finance_category", "detailed")
end

View File

@@ -16,6 +16,15 @@ class Setting < RailsSettings::Base
field :exchange_rate_provider, type: :string, default: ENV.fetch("EXCHANGE_RATE_PROVIDER", "twelve_data")
field :securities_provider, type: :string, default: ENV.fetch("SECURITIES_PROVIDER", "twelve_data")
# Sync settings - check both provider env vars for default
# Only defaults to true if neither provider explicitly disables pending
SYNCS_INCLUDE_PENDING_DEFAULT = begin
simplefin = ENV.fetch("SIMPLEFIN_INCLUDE_PENDING", "1") == "1"
plaid = ENV.fetch("PLAID_INCLUDE_PENDING", "1") == "1"
simplefin && plaid
end
field :syncs_include_pending, type: :boolean, default: SYNCS_INCLUDE_PENDING_DEFAULT
# Dynamic fields are now stored as individual entries with "dynamic:" prefix
# This prevents race conditions and ensures each field is independently managed

View File

@@ -34,9 +34,27 @@ class SimplefinEntry::Processor
# Include provider-supplied extra hash if present
sf["extra"] = data[:extra] if data[:extra].is_a?(Hash)
# Pending detection: only use explicit provider flag
# Pending detection: explicit flag OR inferred from posted=0 + transacted_at
# SimpleFIN indicates pending via:
# 1. pending: true (explicit flag)
# 2. posted=0 (epoch zero) + transacted_at present (implicit - some banks use this pattern)
#
# Note: We only infer from posted=0, NOT from posted=nil/blank, because some providers
# don't supply posted dates even for settled transactions (would cause false positives).
# We always set the key (true or false) to ensure deep_merge overwrites any stale value
if ActiveModel::Type::Boolean.new.cast(data[:pending])
is_pending = if ActiveModel::Type::Boolean.new.cast(data[:pending])
true
else
# Infer pending ONLY when posted is explicitly 0 (epoch) AND transacted_at is present
# posted=nil/blank is NOT treated as pending (some providers omit posted for settled txns)
posted_val = data[:posted]
transacted_val = data[:transacted_at]
posted_is_epoch_zero = posted_val.present? && posted_val.to_i.zero?
transacted_present = transacted_val.present? && transacted_val.to_i > 0
posted_is_epoch_zero && transacted_present
end
if is_pending
sf["pending"] = true
Rails.logger.debug("SimpleFIN: flagged pending transaction #{external_id}")
else

View File

@@ -271,8 +271,8 @@ class SimplefinItem < ApplicationRecord
return nil unless latest
# If sync has statistics, use them
if latest.sync_stats.present?
stats = latest.sync_stats
stats = parse_sync_stats(latest.sync_stats)
if stats.present?
total = stats["total_accounts"] || 0
linked = stats["linked_accounts"] || 0
unlinked = stats["unlinked_accounts"] || 0
@@ -399,7 +399,68 @@ class SimplefinItem < ApplicationRecord
issues
end
# Get reconciled duplicates count from the last sync
# Returns { count: N, message: "..." } or { count: 0 } if none
def last_sync_reconciled_status
latest_sync = syncs.ordered.first
return { count: 0 } unless latest_sync
stats = parse_sync_stats(latest_sync.sync_stats)
count = stats&.dig("pending_reconciled").to_i
if count > 0
{
count: count,
message: I18n.t("simplefin_items.reconciled_status.message", count: count)
}
else
{ count: 0 }
end
end
# Count stale pending transactions (>8 days old) across all linked accounts
# Returns { count: N, accounts: [names] } or { count: 0 } if none
def stale_pending_status(days: 8)
# Get all accounts linked to this SimpleFIN item
# Eager-load both association paths to avoid N+1 on current_account method
linked_accounts = simplefin_accounts.includes(:account, :linked_account).filter_map(&:current_account)
return { count: 0 } if linked_accounts.empty?
# Batch query to avoid N+1
account_ids = linked_accounts.map(&:id)
counts_by_account = Entry.stale_pending(days: days)
.where(excluded: false)
.where(account_id: account_ids)
.group(:account_id)
.count
account_counts = linked_accounts
.map { |account| { account: account, count: counts_by_account[account.id].to_i } }
.select { |ac| ac[:count] > 0 }
total = account_counts.sum { |ac| ac[:count] }
if total > 0
{
count: total,
accounts: account_counts.map { |ac| ac[:account].name },
message: I18n.t("simplefin_items.stale_pending_status.message", count: total, days: days)
}
else
{ count: 0 }
end
end
private
# Parse sync_stats, handling cases where it might be a raw JSON string
# (e.g., from console testing or bypassed serialization)
def parse_sync_stats(sync_stats)
return nil if sync_stats.blank?
return sync_stats if sync_stats.is_a?(Hash)
if sync_stats.is_a?(String)
JSON.parse(sync_stats) rescue nil
end
end
def remove_simplefin_item
# SimpleFin doesn't require server-side cleanup like Plaid
# The access URL just becomes inactive

View File

@@ -9,6 +9,7 @@ class SimplefinItem::Importer
@simplefin_provider = simplefin_provider
@sync = sync
@enqueued_holdings_job_ids = Set.new
@reconciled_account_ids = Set.new # Debounce pending reconciliation per run
end
def import
@@ -201,8 +202,8 @@ class SimplefinItem::Importer
end
adapter.update_balance(
balance: normalized,
cash_balance: cash,
balance: account_data[:balance],
cash_balance: account_data[:"available-balance"],
source: "simplefin"
)
end
@@ -428,9 +429,12 @@ class SimplefinItem::Importer
perform_account_discovery
# Step 2: Fetch transactions/holdings using the regular window.
# Note: Don't pass explicit `pending:` here - let fetch_accounts_data use the
# SIMPLEFIN_INCLUDE_PENDING config. This allows users to disable pending transactions
# if their bank's SimpleFIN integration produces duplicates when pending→posted.
start_date = determine_sync_start_date
Rails.logger.info "SimplefinItem::Importer - import_regular_sync: last_synced_at=#{simplefin_item.last_synced_at&.strftime('%Y-%m-%d %H:%M')} => start_date=#{start_date&.strftime('%Y-%m-%d')}"
accounts_data = fetch_accounts_data(start_date: start_date, pending: true)
accounts_data = fetch_accounts_data(start_date: start_date)
return if accounts_data.nil? # Error already handled
# Store raw payload
@@ -554,9 +558,15 @@ class SimplefinItem::Importer
# Returns a Hash payload with keys like :accounts, or nil when an error is
# handled internally via `handle_errors`.
def fetch_accounts_data(start_date:, end_date: nil, pending: nil)
# Determine whether to include pending based on explicit arg or global config.
# `Rails.configuration.x.simplefin.include_pending` is ENV-backed.
effective_pending = pending.nil? ? Rails.configuration.x.simplefin.include_pending : pending
# Determine whether to include pending based on explicit arg, env var, or Setting.
# Priority: explicit arg > env var > Setting (allows runtime changes via UI)
effective_pending = if !pending.nil?
pending
elsif ENV["SIMPLEFIN_INCLUDE_PENDING"].present?
Rails.configuration.x.simplefin.include_pending
else
Setting.syncs_include_pending
end
# Debug logging to track exactly what's being sent to SimpleFin API
start_str = start_date.respond_to?(:strftime) ? start_date.strftime("%Y-%m-%d") : "none"
@@ -806,6 +816,15 @@ class SimplefinItem::Importer
# Post-save side effects
acct = simplefin_account.current_account
if acct
# Handle pending transaction reconciliation (debounced per run to avoid
# repeated scans during chunked history imports)
unless @reconciled_account_ids.include?(acct.id)
@reconciled_account_ids << acct.id
reconcile_and_track_pending_duplicates(acct)
exclude_and_track_stale_pending(acct)
track_stale_unmatched_pending(acct)
end
# Refresh credit attributes when available-balance present
if acct.accountable_type == "CreditCard" && account_data[:"available-balance"].present?
begin
@@ -1146,19 +1165,103 @@ class SimplefinItem::Importer
ids.group_by(&:itself).select { |_, v| v.size > 1 }.keys
end
# --- Simple helpers for numeric handling in normalization ---
def to_decimal(value)
return BigDecimal("0") if value.nil?
case value
when BigDecimal then value
when String then BigDecimal(value) rescue BigDecimal("0")
when Numeric then BigDecimal(value.to_s)
else
BigDecimal("0")
# Reconcile pending transactions that have a matching posted version
# Handles duplicates where pending and posted both exist (tip adjustments, etc.)
def reconcile_and_track_pending_duplicates(account)
reconcile_stats = Entry.reconcile_pending_duplicates(account: account, dry_run: false)
exact_matches = reconcile_stats[:details].select { |d| d[:match_type] == "exact" }
fuzzy_suggestions = reconcile_stats[:details].select { |d| d[:match_type] == "fuzzy_suggestion" }
if exact_matches.any?
stats["pending_reconciled"] = stats.fetch("pending_reconciled", 0) + exact_matches.size
stats["pending_reconciled_details"] ||= []
exact_matches.each do |detail|
stats["pending_reconciled_details"] << {
"account_name" => detail[:account],
"pending_name" => detail[:pending_name],
"posted_name" => detail[:posted_name]
}
end
stats["pending_reconciled_details"] = stats["pending_reconciled_details"].last(50)
end
if fuzzy_suggestions.any?
stats["duplicate_suggestions_created"] = stats.fetch("duplicate_suggestions_created", 0) + fuzzy_suggestions.size
stats["duplicate_suggestions_details"] ||= []
fuzzy_suggestions.each do |detail|
stats["duplicate_suggestions_details"] << {
"account_name" => detail[:account],
"pending_name" => detail[:pending_name],
"posted_name" => detail[:posted_name]
}
end
stats["duplicate_suggestions_details"] = stats["duplicate_suggestions_details"].last(50)
end
rescue => e
Rails.logger.warn("SimpleFin: pending reconciliation failed for account #{account.id}: #{e.class} - #{e.message}")
record_reconciliation_error("pending_reconciliation", account, e)
end
def same_sign?(a, b)
(a.positive? && b.positive?) || (a.negative? && b.negative?)
# Auto-exclude stale pending transactions (>8 days old with no matching posted version)
# Prevents orphaned pending transactions from affecting budgets indefinitely
def exclude_and_track_stale_pending(account)
excluded_count = Entry.auto_exclude_stale_pending(account: account)
return unless excluded_count > 0
stats["stale_pending_excluded"] = stats.fetch("stale_pending_excluded", 0) + excluded_count
stats["stale_pending_details"] ||= []
stats["stale_pending_details"] << {
"account_name" => account.name,
"account_id" => account.id,
"count" => excluded_count
}
stats["stale_pending_details"] = stats["stale_pending_details"].last(50)
rescue => e
Rails.logger.warn("SimpleFin: stale pending cleanup failed for account #{account.id}: #{e.class} - #{e.message}")
record_reconciliation_error("stale_pending_cleanup", account, e)
end
# Track stale pending transactions that couldn't be matched (for user awareness)
# These are >8 days old, still pending, and have no duplicate suggestion
def track_stale_unmatched_pending(account)
stale_unmatched = account.entries
.joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'")
.where(excluded: false)
.where("entries.date < ?", 8.days.ago.to_date)
.where(<<~SQL.squish)
(transactions.extra -> 'simplefin' ->> 'pending')::boolean = true
OR (transactions.extra -> 'plaid' ->> 'pending')::boolean = true
SQL
.where(<<~SQL.squish)
transactions.extra -> 'potential_posted_match' IS NULL
SQL
.count
return unless stale_unmatched > 0
stats["stale_unmatched_pending"] = stats.fetch("stale_unmatched_pending", 0) + stale_unmatched
stats["stale_unmatched_details"] ||= []
stats["stale_unmatched_details"] << {
"account_name" => account.name,
"account_id" => account.id,
"count" => stale_unmatched
}
stats["stale_unmatched_details"] = stats["stale_unmatched_details"].last(50)
rescue => e
Rails.logger.warn("SimpleFin: stale unmatched tracking failed for account #{account.id}: #{e.class} - #{e.message}")
record_reconciliation_error("stale_unmatched_tracking", account, e)
end
# Record reconciliation errors to sync_stats for UI visibility
def record_reconciliation_error(context, account, error)
stats["reconciliation_errors"] ||= []
stats["reconciliation_errors"] << {
"context" => context,
"account_id" => account.id,
"account_name" => account.name,
"error" => "#{error.class}: #{error.message}"
}
stats["reconciliation_errors"] = stats["reconciliation_errors"].last(20)
end
end

View File

@@ -19,6 +19,22 @@ class Transaction < ApplicationRecord
one_time: "one_time" # A one-time expense/income, excluded from budget analytics
}
# Pending transaction scopes - filter based on provider pending flags in extra JSONB
# Works with any provider that stores pending status in extra["provider_name"]["pending"]
scope :pending, -> {
where(<<~SQL.squish)
(transactions.extra -> 'simplefin' ->> 'pending')::boolean = true
OR (transactions.extra -> 'plaid' ->> 'pending')::boolean = true
SQL
}
scope :excluding_pending, -> {
where(<<~SQL.squish)
(transactions.extra -> 'simplefin' ->> 'pending')::boolean IS DISTINCT FROM true
AND (transactions.extra -> 'plaid' ->> 'pending')::boolean IS DISTINCT FROM true
SQL
}
# Overarching grouping method for all transfer-type transactions
def transfer?
funds_movement? || cc_payment? || loan_payment?
@@ -42,7 +58,85 @@ class Transaction < ApplicationRecord
false
end
# Potential duplicate matching methods
# These help users review and resolve fuzzy-matched pending/posted pairs
def has_potential_duplicate?
potential_posted_match_data.present? && !potential_duplicate_dismissed?
end
def potential_duplicate_entry
return nil unless has_potential_duplicate?
Entry.find_by(id: potential_posted_match_data["entry_id"])
end
def potential_duplicate_reason
potential_posted_match_data&.dig("reason")
end
def potential_duplicate_confidence
potential_posted_match_data&.dig("confidence") || "medium"
end
def low_confidence_duplicate?
potential_duplicate_confidence == "low"
end
def potential_duplicate_posted_amount
potential_posted_match_data&.dig("posted_amount")&.to_d
end
def potential_duplicate_dismissed?
potential_posted_match_data&.dig("dismissed") == true
end
# Merge this pending transaction with its suggested posted match
# This DELETES the pending entry since the posted version is canonical
def merge_with_duplicate!
return false unless has_potential_duplicate?
posted_entry = potential_duplicate_entry
return false unless posted_entry
pending_entry_id = entry.id
pending_entry_name = entry.name
# Delete this pending entry completely (no need to keep it around)
entry.destroy!
Rails.logger.info("User merged pending entry #{pending_entry_id} (#{pending_entry_name}) with posted entry #{posted_entry.id}")
true
end
# Dismiss the duplicate suggestion - user says these are NOT the same transaction
def dismiss_duplicate_suggestion!
return false unless potential_posted_match_data.present?
updated_extra = (extra || {}).deep_dup
updated_extra["potential_posted_match"]["dismissed"] = true
update!(extra: updated_extra)
Rails.logger.info("User dismissed duplicate suggestion for entry #{entry.id}")
true
end
# Clear the duplicate suggestion entirely
def clear_duplicate_suggestion!
return false unless potential_posted_match_data.present?
updated_extra = (extra || {}).deep_dup
updated_extra.delete("potential_posted_match")
update!(extra: updated_extra)
true
end
private
def potential_posted_match_data
return nil unless extra.is_a?(Hash)
extra["potential_posted_match"]
end
def clear_merchant_unlinked_association
return unless merchant_id.present? && merchant.is_a?(ProviderMerchant)

View File

@@ -6,6 +6,7 @@ class Transaction::Search
attribute :amount, :string
attribute :amount_operator, :string
attribute :types, array: true
attribute :status, array: true
attribute :accounts, array: true
attribute :account_ids, array: true
attribute :start_date, :string
@@ -30,6 +31,7 @@ class Transaction::Search
query = apply_active_accounts_filter(query, active_accounts_only)
query = apply_category_filter(query, categories)
query = apply_type_filter(query, types)
query = apply_status_filter(query, status)
query = apply_merchant_filter(query, merchants)
query = apply_tag_filter(query, tags)
query = EntrySearch.apply_search_filter(query, search)
@@ -153,4 +155,28 @@ class Transaction::Search
return query unless tags.present?
query.joins(:tags).where(tags: { name: tags })
end
def apply_status_filter(query, statuses)
return query unless statuses.present?
return query if statuses.uniq.sort == [ "confirmed", "pending" ] # Both selected = no filter
pending_condition = <<~SQL.squish
(transactions.extra -> 'simplefin' ->> 'pending')::boolean = true
OR (transactions.extra -> 'plaid' ->> 'pending')::boolean = true
SQL
confirmed_condition = <<~SQL.squish
(transactions.extra -> 'simplefin' ->> 'pending')::boolean IS DISTINCT FROM true
AND (transactions.extra -> 'plaid' ->> 'pending')::boolean IS DISTINCT FROM true
SQL
case statuses.sort
when [ "pending" ]
query.where(pending_condition)
when [ "confirmed" ]
query.where(confirmed_condition)
else
query
end
end
end

View File

@@ -0,0 +1,30 @@
<% env_configured = ENV["SIMPLEFIN_INCLUDE_PENDING"].present? || ENV["PLAID_INCLUDE_PENDING"].present? %>
<div class="space-y-4">
<div class="flex items-center justify-between">
<div class="space-y-1">
<p class="text-sm"><%= t(".include_pending_label") %></p>
<p class="text-secondary text-sm"><%= t(".include_pending_description") %></p>
</div>
<%= styled_form_with model: Setting.new,
url: settings_hosting_path,
method: :patch,
data: { controller: "auto-submit-form", auto_submit_form_trigger_event_value: "change" } do |form| %>
<%= form.toggle :syncs_include_pending,
checked: Setting.syncs_include_pending,
disabled: env_configured,
data: { auto_submit_form_target: "auto" } %>
<% end %>
</div>
<% if env_configured %>
<div class="bg-warning-50 border border-warning-200 rounded-lg p-3">
<div class="flex items-start gap-2">
<%= icon("alert-circle", class: "w-5 h-5 text-warning-600 mt-0.5 shrink-0") %>
<p class="text-sm text-warning-800">
<%= t(".env_configured_message") %>
</p>
</div>
</div>
<% end %>
</div>

View File

@@ -16,6 +16,9 @@
<% end %>
</div>
<% end %>
<%= settings_section title: t(".sync_settings") do %>
<%= render "settings/hostings/sync_settings" %>
<% end %>
<%= settings_section title: t(".invites") do %>
<%= render "settings/hostings/invite_code_settings" %>
<% end %>

View File

@@ -104,6 +104,25 @@
<%= icon "alert-circle", size: "sm", color: "warning" %>
<%= tag.span stale_status[:message], class: "text-sm" %>
</div>
<% elsif (pending_status = simplefin_item.stale_pending_status)[:count] > 0 %>
<div class="text-secondary">
<div class="flex items-center gap-1">
<%= icon "clock", size: "sm", color: "secondary" %>
<%= tag.span pending_status[:message], class: "text-sm" %>
<span class="text-xs text-tertiary"><%= t(".stale_pending_note") %></span>
</div>
<% if pending_status[:accounts]&.any? %>
<div class="text-xs text-tertiary ml-5">
<%= t(".stale_pending_accounts", accounts: pending_status[:accounts].join(", ")) %>
</div>
<% end %>
</div>
<% elsif (reconciled_status = simplefin_item.last_sync_reconciled_status)[:count] > 0 %>
<div class="text-success flex items-center gap-1">
<%= icon "check-circle", size: "sm", color: "success" %>
<%= tag.span reconciled_status[:message], class: "text-sm" %>
<span class="text-xs text-tertiary"><%= t(".reconciled_details_note") %></span>
</div>
<% elsif simplefin_item.rate_limited_message.present? %>
<div class="text-warning flex items-center gap-1">
<%= icon "clock", size: "sm", color: "warning" %>
@@ -117,7 +136,7 @@
<% elsif duplicate_only_errors %>
<div class="text-secondary flex items-center gap-1">
<%= icon "info", size: "sm" %>
<%= tag.span "Some accounts were skipped as duplicates — use Link existing accounts to merge.", class: "text-secondary" %>
<%= tag.span t(".duplicate_accounts_skipped"), class: "text-secondary" %>
</div>
<% else %>
<p class="text-secondary">

View File

@@ -80,12 +80,27 @@
<%# Pending indicator %>
<% if transaction.pending? %>
<span class="inline-flex items-center gap-1 text-xs font-medium rounded-full px-1.5 py-0.5 border border-secondary text-secondary" title="Pending — may change when posted">
<span class="inline-flex items-center gap-1 text-xs font-medium rounded-full px-1.5 py-0.5 border border-secondary text-secondary" title="<%= t('transactions.transaction.pending_tooltip') %>">
<%= icon "clock", size: "sm", color: "current" %>
Pending
<%= t("transactions.transaction.pending") %>
</span>
<% end %>
<%# Potential duplicate indicator - different styling for low vs medium confidence %>
<% if transaction.has_potential_duplicate? %>
<% if transaction.low_confidence_duplicate? %>
<span class="inline-flex items-center gap-1 text-xs font-medium rounded-full px-1.5 py-0.5 border border-secondary bg-surface-inset text-secondary" title="<%= t('transactions.transaction.review_recommended_tooltip') %>">
<%= icon "help-circle", size: "sm", color: "current" %>
<%= t("transactions.transaction.review_recommended") %>
</span>
<% else %>
<span class="inline-flex items-center gap-1 text-xs font-medium rounded-full px-1.5 py-0.5 border border-warning bg-warning/10 text-warning" title="<%= t('transactions.transaction.potential_duplicate_tooltip') %>">
<%= icon "alert-triangle", size: "sm", color: "current" %>
<%= t("transactions.transaction.possible_duplicate") %>
</span>
<% end %>
<% end %>
<% if transaction.transfer.present? %>
<%= render "transactions/transfer_match", transaction: transaction %>
<% end %>

View File

@@ -35,6 +35,11 @@
end %>"></div>
<p><%= t(".#{param_value.downcase}") %></p>
</div>
<% elsif param_key == "status" %>
<div class="flex items-center gap-2 px-1">
<%= icon(param_value.downcase == "pending" ? "clock" : "check", size: "sm") %>
<p><%= t(".#{param_value.downcase}") %></p>
</div>
<% else %>
<div class="flex items-center gap-2">
<p><%= param_value %></p>

View File

@@ -0,0 +1,26 @@
<%# locals: (form:) %>
<div class="p-2 space-y-3">
<div class="flex items-center gap-3" data-filter-name="confirmed">
<%= form.check_box :status,
{
multiple: true,
checked: @q[:status]&.include?("confirmed"),
class: "checkbox checkbox--light"
},
"confirmed",
nil %>
<%= form.label :status, t(".confirmed"), value: "confirmed", class: "text-sm text-primary" %>
</div>
<div class="flex items-center gap-3" data-filter-name="pending">
<%= form.check_box :status,
{
multiple: true,
checked: @q[:status]&.include?("pending"),
class: "checkbox checkbox--light"
},
"pending",
nil %>
<%= form.label :status, t(".pending"), value: "pending", class: "text-sm text-primary" %>
</div>
</div>

View File

@@ -4,6 +4,47 @@
<% end %>
<% dialog.with_body do %>
<%# Potential duplicate alert %>
<% if @entry.transaction.has_potential_duplicate? %>
<% potential_match = @entry.transaction.potential_duplicate_entry %>
<% if potential_match %>
<div class="mx-4 my-3 p-4 rounded-lg border border-warning bg-warning/5">
<div class="flex items-start gap-3">
<%= icon "alert-triangle", size: "md", color: "warning" %>
<div class="flex-1 space-y-2">
<h4 class="text-sm font-medium text-primary"><%= t("transactions.show.potential_duplicate_title") %></h4>
<p class="text-sm text-secondary"><%= t("transactions.show.potential_duplicate_description") %></p>
<div class="mt-3 p-3 rounded bg-container border border-primary">
<div class="flex items-center justify-between">
<div>
<p class="text-sm font-medium text-primary"><%= potential_match.name %></p>
<p class="text-xs text-secondary"><%= potential_match.date.strftime("%b %d, %Y") %> • <%= potential_match.account.name %></p>
</div>
<p class="text-sm font-medium <%= potential_match.amount.negative? ? 'text-green-600' : 'text-primary' %>">
<%= format_money(-potential_match.amount_money) %>
</p>
</div>
</div>
<div class="flex items-center gap-2 mt-3">
<%= button_to t("transactions.show.merge_duplicate"),
merge_duplicate_transaction_path(@entry.transaction),
method: :post,
class: "btn btn--primary btn--sm",
data: { turbo_frame: "_top" } %>
<%= button_to t("transactions.show.keep_both"),
dismiss_duplicate_transaction_path(@entry.transaction),
method: :post,
class: "btn btn--outline btn--sm",
data: { turbo_frame: "_top" } %>
</div>
</div>
</div>
</div>
<% end %>
<% end %>
<% dialog.with_section(title: t(".overview"), open: true) do %>
<div class="pb-4">
<%= styled_form_with model: @entry,

View File

@@ -4,4 +4,13 @@
Rails.application.configure do
config.plaid = nil
config.plaid_eu = nil
# Plaid pending transaction settings (mirrors SimpleFIN config pattern)
falsy = %w[0 false no off]
config.x.plaid ||= ActiveSupport::OrderedOptions.new
# Default to true - fetch pending transactions for display with "Pending" badge
# and reconciliation when posted versions arrive (Plaid provides pending_transaction_id for reliable linking)
# Set PLAID_INCLUDE_PENDING=0 to disable if user prefers not to see pending transactions
pending_env = ENV["PLAID_INCLUDE_PENDING"].to_s.strip.downcase
config.x.plaid.include_pending = pending_env.blank? ? true : !falsy.include?(pending_env)
end

View File

@@ -1,7 +1,15 @@
Rails.application.configure do
truthy = %w[1 true yes on]
falsy = %w[0 false no off]
config.x.simplefin ||= ActiveSupport::OrderedOptions.new
config.x.simplefin.include_pending = truthy.include?(ENV["SIMPLEFIN_INCLUDE_PENDING"].to_s.strip.downcase)
# Default to true - always fetch pending transactions so they can be:
# - Displayed with a "Pending" badge
# - Excluded from budgets (but included in net worth)
# - Reconciled when posted versions arrive (avoiding duplicates)
# - Auto-excluded after 8 days if they remain stale
# Set SIMPLEFIN_INCLUDE_PENDING=0 to disable if a bank's integration causes issues
pending_env = ENV["SIMPLEFIN_INCLUDE_PENDING"].to_s.strip.downcase
config.x.simplefin.include_pending = pending_env.blank? ? true : !falsy.include?(pending_env)
config.x.simplefin.debug_raw = truthy.include?(ENV["SIMPLEFIN_DEBUG_RAW"].to_s.strip.downcase)
end

View File

@@ -50,15 +50,19 @@ en:
activity:
amount: Amount
balance: Balance
confirmed: Confirmed
date: Date
entries: entries
entry: entry
filter: Filter
new: New
new_balance: New balance
new_transaction: New transaction
no_entries: No entries found
pending: Pending
search:
placeholder: Search entries by name
status: Status
title: Activity
chart:
balance: Balance

View File

@@ -24,6 +24,28 @@ en:
rate_limited: "Rate limited %{time_ago}"
recently: recently
errors: "Errors: %{count}"
pending_reconciled:
one: "%{count} duplicate pending transaction reconciled"
other: "%{count} duplicate pending transactions reconciled"
view_reconciled: View reconciled transactions
duplicate_suggestions:
one: "%{count} possible duplicate needs review"
other: "%{count} possible duplicates need review"
view_duplicate_suggestions: View suggested duplicates
stale_pending:
one: "%{count} stale pending transaction (excluded from budgets)"
other: "%{count} stale pending transactions (excluded from budgets)"
view_stale_pending: View affected accounts
stale_pending_count:
one: "%{count} transaction"
other: "%{count} transactions"
stale_unmatched:
one: "%{count} pending transaction needs manual review"
other: "%{count} pending transactions need manual review"
view_stale_unmatched: View transactions needing review
stale_unmatched_count:
one: "%{count} transaction"
other: "%{count} transactions"
data_warnings: "Data warnings: %{count}"
notices: "Notices: %{count}"
view_data_quality: View data quality details

View File

@@ -17,6 +17,7 @@ en:
show:
general: General Settings
financial_data_providers: Financial Data Providers
sync_settings: Sync Settings
invites: Invite Codes
title: Self-Hosting
danger_zone: Danger Zone
@@ -77,3 +78,7 @@ en:
clear_cache:
cache_cleared: Data cache has been cleared. This may take a few moments to complete.
not_authorized: You are not authorized to perform this action
sync_settings:
include_pending_label: Include pending transactions
include_pending_description: When enabled, pending (uncleared) transactions will be imported and automatically reconciled when they post. Disable if your bank provides unreliable pending data.
env_configured_message: This setting is disabled because a provider environment variable (SIMPLEFIN_INCLUDE_PENDING or PLAID_INCLUDE_PENDING) is set. Remove it to enable this setting.

View File

@@ -70,6 +70,10 @@ en:
status_with_summary: "Last synced %{timestamp} ago • %{summary}"
syncing: Syncing...
update: Update
stale_pending_note: "(excluded from budgets)"
stale_pending_accounts: "in: %{accounts}"
reconciled_details_note: "(see sync summary for details)"
duplicate_accounts_skipped: "Some accounts were skipped as duplicates — use 'Link existing accounts' to merge."
select_existing_account:
title: "Link %{account_name} to SimpleFIN"
description: Select a SimpleFIN account to link to your existing account
@@ -86,3 +90,11 @@ en:
errors:
only_manual: Only manual accounts can be linked
invalid_simplefin_account: Invalid SimpleFIN account selected
reconciled_status:
message:
one: "%{count} duplicate pending transaction reconciled"
other: "%{count} duplicate pending transactions reconciled"
stale_pending_status:
message:
one: "%{count} pending transaction older than %{days} days"
other: "%{count} pending transactions older than %{days} days"

View File

@@ -44,6 +44,23 @@ en:
settings: Settings
tags_label: Tags
uncategorized: "(uncategorized)"
potential_duplicate_title: Possible duplicate detected
potential_duplicate_description: This pending transaction may be the same as the posted transaction below. If so, merge them to avoid double-counting.
merge_duplicate: Yes, merge them
keep_both: No, keep both
transaction:
pending: Pending
pending_tooltip: Pending transaction — may change when posted
possible_duplicate: Duplicate?
potential_duplicate_tooltip: This may be a duplicate of another transaction
review_recommended: Review
review_recommended_tooltip: Large amount difference — review recommended to check if this is a duplicate
merge_duplicate:
success: Transactions merged successfully
failure: Could not merge transactions
dismiss_duplicate:
success: Kept as separate transactions
failure: Could not dismiss duplicate suggestion
header:
edit_categories: Edit categories
edit_imports: Edit imports
@@ -55,6 +72,16 @@ en:
transactions: transactions
import: Import
toggle_recurring_section: Toggle upcoming recurring transactions
search:
filters:
account: Account
date: Date
type: Type
status: Status
amount: Amount
category: Category
tag: Tag
merchant: Merchant
searches:
filters:
amount_filter:
@@ -68,10 +95,15 @@ en:
on_or_after: on or after %{date}
on_or_before: on or before %{date}
transfer: Transfer
confirmed: Confirmed
pending: Pending
type_filter:
expense: Expense
income: Income
transfer: Transfer
status_filter:
confirmed: Confirmed
pending: Pending
menu:
account_filter: Account
amount_filter: Amount
@@ -81,6 +113,7 @@ en:
clear_filters: Clear filters
date_filter: Date
merchant_filter: Merchant
status_filter: Status
tag_filter: Tag
type_filter: Type
search:

View File

@@ -191,6 +191,8 @@ Rails.application.routes.draw do
member do
post :mark_as_recurring
post :merge_duplicate
post :dismiss_duplicate
end
end

View File

@@ -0,0 +1,158 @@
# frozen_string_literal: true
namespace :simplefin do
desc "Find and optionally remove duplicate pending transactions that have matching posted versions"
task pending_cleanup: :environment do
dry_run = ENV["DRY_RUN"] != "false"
date_window = (ENV["DATE_WINDOW"] || 8).to_i
puts "SimpleFIN Pending Transaction Cleanup"
puts "======================================"
puts "Mode: #{dry_run ? 'DRY RUN (no changes)' : 'LIVE (will delete duplicates)'}"
puts "Date window: #{date_window} days"
puts ""
# Find all pending SimpleFIN transactions
pending_entries = Entry.joins(
"INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'"
).where(source: "simplefin")
.where("transactions.extra -> 'simplefin' ->> 'pending' = ?", "true")
.includes(:account)
puts "Found #{pending_entries.count} pending SimpleFIN transactions"
puts ""
duplicates_found = 0
duplicates_to_delete = []
pending_entries.find_each do |pending_entry|
# Look for a matching posted transaction
posted_match = Entry.joins(
"INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'"
).where(account_id: pending_entry.account_id)
.where(source: "simplefin")
.where(amount: pending_entry.amount)
.where(currency: pending_entry.currency)
.where(date: pending_entry.date..(pending_entry.date + date_window.days)) # Posted must be ON or AFTER pending
.where.not(id: pending_entry.id)
.where("transactions.extra -> 'simplefin' ->> 'pending' != ? OR transactions.extra -> 'simplefin' ->> 'pending' IS NULL", "true")
.first
if posted_match
duplicates_found += 1
duplicates_to_delete << pending_entry
puts "DUPLICATE FOUND:"
puts " Pending: ID=#{pending_entry.id} | #{pending_entry.date} | #{pending_entry.name} | #{pending_entry.amount} #{pending_entry.currency}"
puts " Posted: ID=#{posted_match.id} | #{posted_match.date} | #{posted_match.name} | #{posted_match.amount} #{posted_match.currency}"
puts " Account: #{pending_entry.account.name}"
puts ""
end
end
puts "======================================"
puts "Summary: #{duplicates_found} duplicate pending transactions found"
puts ""
if duplicates_found > 0
if dry_run
puts "To delete these duplicates, run:"
puts " rails simplefin:pending_cleanup DRY_RUN=false"
puts ""
puts "To adjust the date matching window (default 8 days):"
puts " rails simplefin:pending_cleanup DATE_WINDOW=14"
else
print "Deleting #{duplicates_to_delete.count} duplicate pending entries... "
Entry.where(id: duplicates_to_delete.map(&:id)).destroy_all
puts "Done!"
end
else
puts "No duplicates found. Nothing to clean up."
end
end
desc "Un-exclude pending transactions that were wrongly matched (fixes direction bug)"
task pending_restore: :environment do
dry_run = ENV["DRY_RUN"] != "false"
date_window = (ENV["DATE_WINDOW"] || 8).to_i
puts "Restore Wrongly Excluded Pending Transactions"
puts "=============================================="
puts "Mode: #{dry_run ? 'DRY RUN (no changes)' : 'LIVE (will restore)'}"
puts "Date window: #{date_window} days (forward only)"
puts ""
# Find all EXCLUDED pending transactions (these may have been wrongly excluded)
excluded_pending = Entry.joins(
"INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'"
).where(excluded: true)
.where(<<~SQL.squish)
(transactions.extra -> 'simplefin' ->> 'pending')::boolean = true
OR (transactions.extra -> 'plaid' ->> 'pending')::boolean = true
SQL
puts "Found #{excluded_pending.count} excluded pending transactions to evaluate"
puts ""
to_restore = []
excluded_pending.includes(:account).find_each do |pending_entry|
# Check if there's a VALID posted match using CORRECT logic (forward-only dates)
# Posted date must be ON or AFTER pending date
valid_match = pending_entry.account.entries
.joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'")
.where.not(id: pending_entry.id)
.where(currency: pending_entry.currency)
.where(amount: pending_entry.amount)
.where(date: pending_entry.date..(pending_entry.date + date_window.days))
.where(<<~SQL.squish)
(transactions.extra -> 'simplefin' ->> 'pending')::boolean IS NOT TRUE
AND (transactions.extra -> 'plaid' ->> 'pending')::boolean IS NOT TRUE
SQL
.exists?
unless valid_match
to_restore << pending_entry
puts "SHOULD RESTORE (no valid match):"
puts " ID=#{pending_entry.id} | #{pending_entry.date} | #{pending_entry.name} | #{pending_entry.amount} #{pending_entry.currency}"
puts " Account: #{pending_entry.account.name}"
puts ""
end
end
puts "=============================================="
puts "Summary: #{to_restore.count} transactions should be restored"
puts ""
if to_restore.any?
if dry_run
puts "To restore these transactions, run:"
puts " rails simplefin:pending_restore DRY_RUN=false"
else
Entry.where(id: to_restore.map(&:id)).update_all(excluded: false)
puts "Restored #{to_restore.count} transactions!"
end
else
puts "No wrongly excluded transactions found."
end
end
desc "List all pending SimpleFIN transactions (for review)"
task pending_list: :environment do
pending_entries = Entry.joins(
"INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'"
).where(source: "simplefin")
.where("transactions.extra -> 'simplefin' ->> 'pending' = ?", "true")
.includes(:account)
.order(date: :desc)
puts "All Pending SimpleFIN Transactions"
puts "==================================="
puts "Total: #{pending_entries.count}"
puts ""
pending_entries.find_each do |entry|
puts "ID=#{entry.id} | #{entry.date} | #{entry.name.truncate(40)} | #{entry.amount} #{entry.currency} | Account: #{entry.account.name}"
end
end
end

View File

@@ -785,4 +785,482 @@ class Account::ProviderImportAdapterTest < ActiveSupport::TestCase
assert_nil newer_entry.reload.external_id
end
end
# ============================================================================
# Pending→Posted Transaction Reconciliation Tests
# ============================================================================
test "reconciles pending transaction when posted version arrives with different external_id" do
# Simulate SimpleFIN giving different IDs for pending vs posted transactions
# First, import a pending transaction
pending_entry = @adapter.import_transaction(
external_id: "simplefin_pending_abc",
amount: 99.99,
currency: "USD",
date: Date.today - 2.days,
name: "Coffee Shop",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
assert pending_entry.transaction.pending?, "Entry should be marked pending"
original_id = pending_entry.id
# Now import the posted version with a DIFFERENT external_id
# This should claim the pending entry, not create a duplicate
assert_no_difference "@account.entries.count" do
posted_entry = @adapter.import_transaction(
external_id: "simplefin_posted_xyz",
amount: 99.99,
currency: "USD",
date: Date.today,
name: "Coffee Shop - Posted",
source: "simplefin",
extra: { "simplefin" => { "pending" => false } }
)
# Should be the same entry, now with updated external_id
assert_equal original_id, posted_entry.id
assert_equal "simplefin_posted_xyz", posted_entry.external_id
assert_not posted_entry.transaction.pending?, "Entry should no longer be pending"
end
end
test "does not reconcile when posted transaction has same external_id as pending" do
# When external_id matches, normal dedup should handle it
pending_entry = @adapter.import_transaction(
external_id: "simplefin_same_id",
amount: 50.00,
currency: "USD",
date: Date.today - 1.day,
name: "Gas Station",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Import posted version with SAME external_id
assert_no_difference "@account.entries.count" do
posted_entry = @adapter.import_transaction(
external_id: "simplefin_same_id",
amount: 50.00,
currency: "USD",
date: Date.today,
name: "Gas Station - Posted",
source: "simplefin",
extra: { "simplefin" => { "pending" => false } }
)
assert_equal pending_entry.id, posted_entry.id
assert_not posted_entry.transaction.pending?
end
end
test "fuzzy amount match creates suggestion instead of auto-claiming" do
# Import pending transaction (pre-tip authorization)
pending_entry = @adapter.import_transaction(
external_id: "simplefin_pending_amount_test",
amount: 100.00,
currency: "USD",
date: Date.today - 1.day,
name: "Restaurant",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Import posted with tip added - should NOT auto-claim, but should store suggestion
# Fuzzy matches now create suggestions for user review instead of auto-reconciling
assert_difference "@account.entries.count", 1 do
posted_entry = @adapter.import_transaction(
external_id: "simplefin_posted_amount_test",
amount: 105.00, # 5% tip added - within 25% tolerance
currency: "USD",
date: Date.today,
name: "Restaurant",
source: "simplefin",
extra: { "simplefin" => { "pending" => false } }
)
# Should be a NEW entry (not claimed)
assert_not_equal pending_entry.id, posted_entry.id
assert_equal "simplefin_posted_amount_test", posted_entry.external_id
# The PENDING entry should now have a potential_posted_match suggestion
pending_entry.reload
assert pending_entry.transaction.has_potential_duplicate?
assert_equal posted_entry.id, pending_entry.transaction.potential_duplicate_entry.id
end
end
test "does not reconcile pending when amount difference exceeds tolerance" do
# Import pending transaction
pending_entry = @adapter.import_transaction(
external_id: "simplefin_pending_big_diff",
amount: 50.00,
currency: "USD",
date: Date.today - 1.day,
name: "Store",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Import posted with amount >25% different - should NOT match
# $100 posted / 1.25 = $80 minimum pending, but pending is only $50
assert_difference "@account.entries.count", 1 do
posted_entry = @adapter.import_transaction(
external_id: "simplefin_posted_big_diff",
amount: 100.00, # 100% increase - way outside 25% tolerance
currency: "USD",
date: Date.today,
name: "Store",
source: "simplefin",
extra: { "simplefin" => { "pending" => false } }
)
assert_not_equal pending_entry.id, posted_entry.id
end
end
test "does not reconcile pending when date is outside window" do
# Import pending transaction
pending_entry = @adapter.import_transaction(
external_id: "simplefin_pending_date_test",
amount: 25.00,
currency: "USD",
date: Date.today - 15.days, # 15 days ago
name: "Subscription",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Import posted with date outside 7-day window - should NOT match
assert_difference "@account.entries.count", 1 do
posted_entry = @adapter.import_transaction(
external_id: "simplefin_posted_date_test",
amount: 25.00,
currency: "USD",
date: Date.today,
name: "Subscription",
source: "simplefin",
extra: { "simplefin" => { "pending" => false } }
)
assert_not_equal pending_entry.id, posted_entry.id
end
end
test "reconciles pending within 7 day window" do
# Import pending transaction
pending_entry = @adapter.import_transaction(
external_id: "simplefin_pending_window_test",
amount: 75.00,
currency: "USD",
date: Date.today - 5.days,
name: "Online Order",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Import posted within 7-day window - should match
assert_no_difference "@account.entries.count" do
posted_entry = @adapter.import_transaction(
external_id: "simplefin_posted_window_test",
amount: 75.00,
currency: "USD",
date: Date.today,
name: "Online Order - Posted",
source: "simplefin",
extra: { "simplefin" => { "pending" => false } }
)
assert_equal pending_entry.id, posted_entry.id
end
end
test "does not reconcile pending from different source" do
# Import pending from SimpleFIN
pending_entry = @adapter.import_transaction(
external_id: "simplefin_pending_source_test",
amount: 30.00,
currency: "USD",
date: Date.today - 1.day,
name: "Pharmacy",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Import from different source (plaid) - should NOT match SimpleFIN pending
assert_difference "@account.entries.count", 1 do
plaid_entry = @adapter.import_transaction(
external_id: "plaid_posted_source_test",
amount: 30.00,
currency: "USD",
date: Date.today,
name: "Pharmacy",
source: "plaid",
extra: { "plaid" => { "pending" => false } }
)
assert_not_equal pending_entry.id, plaid_entry.id
end
end
test "does not reconcile when incoming transaction is also pending" do
# Import first pending transaction
pending_entry1 = @adapter.import_transaction(
external_id: "simplefin_pending_1",
amount: 45.00,
currency: "USD",
date: Date.today - 1.day,
name: "Store",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Import another pending transaction with different ID - should NOT match
assert_difference "@account.entries.count", 1 do
pending_entry2 = @adapter.import_transaction(
external_id: "simplefin_pending_2",
amount: 45.00,
currency: "USD",
date: Date.today,
name: "Store",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
assert_not_equal pending_entry1.id, pending_entry2.id
end
end
test "reconciles most recent pending when multiple exist" do
# Create two pending transactions with same amount
older_pending = @adapter.import_transaction(
external_id: "simplefin_older_pending",
amount: 60.00,
currency: "USD",
date: Date.today - 5.days,
name: "Recurring Payment - Old",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
newer_pending = @adapter.import_transaction(
external_id: "simplefin_newer_pending",
amount: 60.00,
currency: "USD",
date: Date.today - 1.day,
name: "Recurring Payment - New",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Import posted - should match the most recent pending (by date)
assert_no_difference "@account.entries.count" do
posted_entry = @adapter.import_transaction(
external_id: "simplefin_posted_recurring",
amount: 60.00,
currency: "USD",
date: Date.today,
name: "Recurring Payment - Posted",
source: "simplefin",
extra: { "simplefin" => { "pending" => false } }
)
# Should match the newer pending entry
assert_equal newer_pending.id, posted_entry.id
# Older pending should remain untouched
assert_equal "simplefin_older_pending", older_pending.reload.external_id
end
end
test "find_pending_transaction returns nil when no pending transactions exist" do
# Create a non-pending transaction
@adapter.import_transaction(
external_id: "simplefin_not_pending",
amount: 40.00,
currency: "USD",
date: Date.today - 1.day,
name: "Regular Transaction",
source: "simplefin",
extra: { "simplefin" => { "pending" => false } }
)
result = @adapter.find_pending_transaction(
date: Date.today,
amount: 40.00,
currency: "USD",
source: "simplefin"
)
assert_nil result
end
# ============================================================================
# Critical Direction Fix Tests (CITGO Bug Prevention)
# ============================================================================
test "does not match pending transaction that is AFTER the posted date (direction fix)" do
# This is the CITGO bug scenario:
# - Posted transaction on Dec 31
# - Pending transaction on Jan 8 (AFTER the posted)
# - These should NOT match because pending MUST come BEFORE posted
# First, import a POSTED transaction on an earlier date
posted_entry = @adapter.import_transaction(
external_id: "simplefin_posted_dec31",
amount: 6.67,
currency: "USD",
date: Date.today - 8.days, # Dec 31 (earlier)
name: "CITGO Gas Station",
source: "simplefin",
extra: { "simplefin" => { "pending" => false } }
)
# Now import a PENDING transaction on a LATER date
# This should NOT be matched because the date direction is wrong
assert_difference "@account.entries.count", 1 do
pending_entry = @adapter.import_transaction(
external_id: "simplefin_pending_jan8",
amount: 6.65, # Similar but different amount
currency: "USD",
date: Date.today, # Jan 8 (later)
name: "CITGO Gas Station",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Should be a DIFFERENT entry - not matched to the earlier posted one
assert_not_equal posted_entry.id, pending_entry.id
assert pending_entry.transaction.pending?
end
end
test "find_pending_transaction only searches backward in time" do
# Create a pending transaction in the FUTURE (after the posted date we'll search from)
# This should NOT be found because pending must be ON or BEFORE posted
future_pending = @adapter.import_transaction(
external_id: "simplefin_future_pending",
amount: 50.00,
currency: "USD",
date: Date.today + 3.days, # Future date
name: "Future Transaction",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Search from today - should NOT find the future pending
result = @adapter.find_pending_transaction(
date: Date.today,
amount: 50.00,
currency: "USD",
source: "simplefin"
)
assert_nil result, "Should not find pending transactions that are in the future relative to the posted date"
end
test "find_pending_transaction finds pending transaction that is before posted date" do
# Create a pending transaction in the PAST (before the posted date)
# This SHOULD be found
past_pending = @adapter.import_transaction(
external_id: "simplefin_past_pending",
amount: 75.00,
currency: "USD",
date: Date.today - 3.days, # 3 days ago
name: "Past Transaction",
source: "simplefin",
extra: { "simplefin" => { "pending" => true } }
)
# Search from today - should find the past pending
result = @adapter.find_pending_transaction(
date: Date.today,
amount: 75.00,
currency: "USD",
source: "simplefin"
)
assert_equal past_pending.id, result.id
end
# ============================================================================
# Plaid pending_transaction_id Tests
# ============================================================================
test "reconciles pending via Plaid pending_transaction_id" do
# Import a pending transaction
pending_entry = @adapter.import_transaction(
external_id: "plaid_pending_abc",
amount: 42.00,
currency: "USD",
date: Date.today - 2.days,
name: "Coffee Shop",
source: "plaid",
extra: { "plaid" => { "pending" => true } }
)
# Import posted with pending_transaction_id linking to the pending
assert_no_difference "@account.entries.count" do
posted_entry = @adapter.import_transaction(
external_id: "plaid_posted_xyz",
amount: 42.00,
currency: "USD",
date: Date.today,
name: "Coffee Shop",
source: "plaid",
pending_transaction_id: "plaid_pending_abc", # Links to pending
extra: { "plaid" => { "pending" => false, "pending_transaction_id" => "plaid_pending_abc" } }
)
# Should claim the pending entry
assert_equal pending_entry.id, posted_entry.id
assert_equal "plaid_posted_xyz", posted_entry.external_id
assert_not posted_entry.transaction.pending?
end
end
test "Plaid pending_transaction_id takes priority over amount matching" do
# Create TWO pending transactions with same amount
pending1 = @adapter.import_transaction(
external_id: "plaid_pending_1",
amount: 25.00,
currency: "USD",
date: Date.today - 1.day,
name: "Store A",
source: "plaid",
extra: { "plaid" => { "pending" => true } }
)
pending2 = @adapter.import_transaction(
external_id: "plaid_pending_2",
amount: 25.00,
currency: "USD",
date: Date.today - 1.day,
name: "Store B",
source: "plaid",
extra: { "plaid" => { "pending" => true } }
)
# Import posted that explicitly links to pending2 via pending_transaction_id
assert_no_difference "@account.entries.count" do
posted_entry = @adapter.import_transaction(
external_id: "plaid_posted_linked",
amount: 25.00,
currency: "USD",
date: Date.today,
name: "Store B",
source: "plaid",
pending_transaction_id: "plaid_pending_2", # Explicitly links to pending2
extra: { "plaid" => { "pending" => false } }
)
# Should claim pending2 specifically (not pending1)
assert_equal pending2.id, posted_entry.id
assert_equal "plaid_posted_linked", posted_entry.external_id
end
# pending1 should still exist as pending
pending1.reload
assert_equal "plaid_pending_1", pending1.external_id
end
end

View File

@@ -29,8 +29,8 @@ class SimplefinEntry::ProcessorTest < ActiveSupport::TestCase
payee: "Pizza Hut",
description: "Order #1234",
memo: "Carryout",
posted: Date.today.to_s,
transacted_at: (Date.today - 1).to_s,
posted: Date.current.to_s,
transacted_at: (Date.current - 1).to_s,
extra: { category: "restaurants", check_number: nil }
}
@@ -64,7 +64,7 @@ class SimplefinEntry::ProcessorTest < ActiveSupport::TestCase
description: "Latte",
memo: "Morning run",
posted: nil,
transacted_at: (Date.today - 3).to_s
transacted_at: (Date.current - 3).to_s
}
SimplefinEntry::Processor.new(tx, simplefin_account: @simplefin_account).process
@@ -77,8 +77,8 @@ class SimplefinEntry::ProcessorTest < ActiveSupport::TestCase
test "captures FX metadata when tx currency differs from account currency" do
# Account is USD from setup; use EUR for tx
t_date = (Date.today - 5)
p_date = Date.today
t_date = (Date.current - 5)
p_date = Date.current
tx = {
id: "tx_fx_1",
@@ -106,8 +106,8 @@ class SimplefinEntry::ProcessorTest < ActiveSupport::TestCase
payee: "Test Store",
description: "Auth",
memo: "",
posted: Date.today.to_s, # provider says pending=true should still flag
transacted_at: (Date.today - 1).to_s,
posted: Date.current.to_s, # provider says pending=true should still flag
transacted_at: (Date.current - 1).to_s,
pending: true
}
@@ -120,7 +120,7 @@ class SimplefinEntry::ProcessorTest < ActiveSupport::TestCase
test "posted==0 treated as missing, entry uses transacted_at date and flags pending" do
# Simulate provider sending epoch-like zeros for posted and an integer transacted_at
t_epoch = (Date.today - 2).to_time.to_i
t_epoch = (Date.current - 2).to_time.to_i
tx = {
id: "tx_pending_zero_posted_1",
amount: "-6.48",
@@ -141,4 +141,26 @@ class SimplefinEntry::ProcessorTest < ActiveSupport::TestCase
sf = entry.transaction.extra.fetch("simplefin")
assert_equal true, sf["pending"], "expected pending flag to be true when posted==0 and/or pending=true"
end
test "infers pending when posted is explicitly 0 and transacted_at present (no explicit pending flag)" do
# Some SimpleFIN banks indicate pending by sending posted=0 + transacted_at, without pending flag
t_epoch = (Date.current - 1).to_time.to_i
tx = {
id: "tx_inferred_pending_1",
amount: "-15.00",
currency: "USD",
payee: "Gas Station",
description: "Fuel",
memo: "",
posted: 0,
transacted_at: t_epoch
# Note: NO pending flag set
}
SimplefinEntry::Processor.new(tx, simplefin_account: @simplefin_account).process
entry = @account.entries.find_by!(external_id: "simplefin_tx_inferred_pending_1", source: "simplefin")
sf = entry.transaction.extra.fetch("simplefin")
assert_equal true, sf["pending"], "expected pending to be inferred from posted=0 + transacted_at present"
end
end