mirror of
https://github.com/we-promise/sure.git
synced 2026-06-06 19:29:03 +00:00
Add pending transaction handling and duplicate reconciliation logic (#602)
* Add pending transaction handling and duplicate reconciliation logic - Implemented logic to exclude pending transactions from budgets and analytics calculations. - Introduced mechanisms for reconciling pending transactions with posted versions. - Added duplicate detection with support for merging or dismissing matches. - Updated transaction search filters to include a `status_filter` for pending/confirmed transactions. - Introduced UI elements for reviewing and resolving duplicates. - Enhanced `ProviderSyncSummary` with stats for reconciled and stale pending transactions. * Refactor translation handling and enhance transaction and sync logic - Moved hardcoded strings to locale files for improved translation support. - Refined styling for duplicate transaction indicators and sync summaries. - Improved logic for excluding stale pending transactions and updating timestamps on batch exclusion. - Added unique IDs to status filters for better element targeting in UI. - Optimized database queries to avoid N+1 issues in stale pending calculations. * Add sync settings and enhance pending transaction handling - Introduced a new "Sync Settings" section in hosting settings with UI to toggle inclusion of pending transactions. - Updated handling of pending transactions with improved inference logic for `posted=0` and `transacted_at` in processors. - Added priority order for pending transaction inclusion: explicit argument > environment variable > runtime configurable setting. - Refactored settings and controllers to store updated sync preferences. * Refactor sync settings and pending transaction reconciliation - Extracted logic for pending transaction reconciliation, stale exclusion, and unmatched tracking into dedicated methods for better maintainability. - Updated sync settings to infer defaults from multiple provider environment variables (`SIMPLEFIN_INCLUDE_PENDING`, `PLAID_INCLUDE_PENDING`). - Refined UI and messaging to handle multi-provider configurations in sync settings. # Conflicts: # app/models/simplefin_item/importer.rb * Debounce transaction reconciliation during imports - Added per-run reconciliation debouncing to prevent repeated scans for the same account during chunked history imports. - Trimmed size of reconciliation stats to retain recent details only. - Introduced error tracking for reconciliation steps to improve UI visibility of issues. * Apply ABS() in pending transaction queries and improve error handling - Updated pending transaction logic to use ABS() for consistent handling of negative amounts. - Adjusted amount bounds calculations to ensure accuracy for both positive and negative values. - Refined exception handling in `merge_duplicate` to log failures and update user alert. - Replaced `Date.today` with `Date.current` in tests to ensure timezone consistency. - Minor optimization to avoid COUNT queries by loading limited records directly. * Improve error handling in duplicate suggestion and dismissal logic - Added exception handling for `store_duplicate_suggestion` to log failures and prevent crashes during fuzzy/low-confidence matches. - Enhanced `dismiss_duplicate` action to handle `ActiveRecord::RecordInvalid` and display appropriate user alerts. --------- Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
This commit is contained in:
12
CLAUDE.md
12
CLAUDE.md
@@ -100,10 +100,14 @@ Two primary data ingestion methods:
|
||||
- SimpleFIN: pending via `pending: true` or `posted` blank/0 + `transacted_at`.
|
||||
- Plaid: pending via Plaid `pending: true` (stored at `extra["plaid"]["pending"]` for bank/credit transactions imported via `PlaidEntry::Processor`).
|
||||
- Storage: provider data on `Transaction#extra` (e.g., `extra["simplefin"]["pending"]`; FX uses `fx_from`, `fx_date`).
|
||||
- UI: “Pending” badge when `transaction.pending?` is true; no badge if provider omits pendings.
|
||||
- Configuration (default-off)
|
||||
- Centralized in `config/initializers/simplefin.rb` via `Rails.configuration.x.simplefin.*`.
|
||||
- ENV-backed keys: `SIMPLEFIN_INCLUDE_PENDING=1`, `SIMPLEFIN_DEBUG_RAW=1`.
|
||||
- UI: "Pending" badge when `transaction.pending?` is true; no badge if provider omits pendings.
|
||||
- Configuration (default-on for pending)
|
||||
- SimpleFIN: `config/initializers/simplefin.rb` via `Rails.configuration.x.simplefin.*`.
|
||||
- Plaid: `config/initializers/plaid_config.rb` via `Rails.configuration.x.plaid.*`.
|
||||
- Pending transactions are fetched by default and handled via reconciliation/filtering.
|
||||
- Set `SIMPLEFIN_INCLUDE_PENDING=0` to disable pending fetching for SimpleFIN.
|
||||
- Set `PLAID_INCLUDE_PENDING=0` to disable pending fetching for Plaid.
|
||||
- Set `SIMPLEFIN_DEBUG_RAW=1` to enable raw payload debug logging.
|
||||
|
||||
Provider support notes:
|
||||
- SimpleFIN: supports pending + FX metadata (stored under `extra["simplefin"]`).
|
||||
|
||||
@@ -46,6 +46,42 @@
|
||||
"data-auto-submit-form-target": "auto" %>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<%= render DS::Menu.new(variant: "button", no_padding: true) do |menu| %>
|
||||
<% menu.with_button(
|
||||
id: "activity-status-filter-button",
|
||||
type: "button",
|
||||
text: t("accounts.show.activity.filter"),
|
||||
variant: "outline",
|
||||
icon: "list-filter"
|
||||
) %>
|
||||
|
||||
<% menu.with_custom_content do %>
|
||||
<div class="p-3 space-y-3 min-w-[160px]">
|
||||
<p class="text-xs font-medium text-secondary uppercase"><%= t("accounts.show.activity.status") %></p>
|
||||
<div class="flex items-center gap-3">
|
||||
<%= check_box_tag "q[status][]",
|
||||
"confirmed",
|
||||
params.dig(:q, :status)&.include?("confirmed"),
|
||||
id: "q_status_confirmed",
|
||||
class: "checkbox checkbox--light",
|
||||
form: "entries-search",
|
||||
onchange: "document.getElementById('entries-search').requestSubmit()" %>
|
||||
<%= label_tag "q_status_confirmed", t("accounts.show.activity.confirmed"), class: "text-sm text-primary" %>
|
||||
</div>
|
||||
<div class="flex items-center gap-3">
|
||||
<%= check_box_tag "q[status][]",
|
||||
"pending",
|
||||
params.dig(:q, :status)&.include?("pending"),
|
||||
id: "q_status_pending",
|
||||
class: "checkbox checkbox--light",
|
||||
form: "entries-search",
|
||||
onchange: "document.getElementById('entries-search').requestSubmit()" %>
|
||||
<%= label_tag "q_status_pending", t("accounts.show.activity.pending"), class: "text-sm text-primary" %>
|
||||
</div>
|
||||
</div>
|
||||
<% end %>
|
||||
<% end %>
|
||||
<% end %>
|
||||
<%= button_tag type: "button",
|
||||
id: "toggle-checkboxes-button",
|
||||
|
||||
@@ -69,6 +69,102 @@
|
||||
<% end %>
|
||||
</div>
|
||||
|
||||
<%# Pending→posted reconciliation %>
|
||||
<% if has_pending_reconciled? %>
|
||||
<div class="mt-1">
|
||||
<div class="flex items-center gap-1">
|
||||
<%= helpers.icon "check-circle", size: "sm", color: "success" %>
|
||||
<span class="text-success"><%= t("provider_sync_summary.health.pending_reconciled", count: pending_reconciled) %></span>
|
||||
</div>
|
||||
<% if pending_reconciled_details.any? %>
|
||||
<details class="mt-1">
|
||||
<summary class="text-xs cursor-pointer text-secondary hover:text-primary">
|
||||
<%= t("provider_sync_summary.health.view_reconciled") %>
|
||||
</summary>
|
||||
<div class="mt-1 pl-2 border-l-2 border-surface-inset space-y-1">
|
||||
<% pending_reconciled_details.each do |detail| %>
|
||||
<p class="text-xs text-success">
|
||||
<%= detail["account_name"] %>: <%= detail["pending_name"] %>
|
||||
</p>
|
||||
<% end %>
|
||||
</div>
|
||||
</details>
|
||||
<% end %>
|
||||
</div>
|
||||
<% end %>
|
||||
|
||||
<%# Duplicate suggestions needing review %>
|
||||
<% if has_duplicate_suggestions_created? %>
|
||||
<div class="mt-1">
|
||||
<div class="flex items-center gap-1">
|
||||
<%= helpers.icon "alert-triangle", size: "sm", color: "warning" %>
|
||||
<span class="text-warning"><%= t("provider_sync_summary.health.duplicate_suggestions", count: duplicate_suggestions_created) %></span>
|
||||
</div>
|
||||
<% if duplicate_suggestions_details.any? %>
|
||||
<details class="mt-1">
|
||||
<summary class="text-xs cursor-pointer text-secondary hover:text-primary">
|
||||
<%= t("provider_sync_summary.health.view_duplicate_suggestions") %>
|
||||
</summary>
|
||||
<div class="mt-1 pl-2 border-l-2 border-surface-inset space-y-1">
|
||||
<% duplicate_suggestions_details.each do |detail| %>
|
||||
<p class="text-xs text-warning">
|
||||
<%= detail["account_name"] %>: <%= detail["pending_name"] %> → <%= detail["posted_name"] %>
|
||||
</p>
|
||||
<% end %>
|
||||
</div>
|
||||
</details>
|
||||
<% end %>
|
||||
</div>
|
||||
<% end %>
|
||||
|
||||
<%# Stale pending transactions (auto-excluded) %>
|
||||
<% if has_stale_pending? %>
|
||||
<div class="mt-1">
|
||||
<div class="flex items-center gap-1">
|
||||
<%= helpers.icon "clock", size: "sm", color: "warning" %>
|
||||
<span class="text-warning"><%= t("provider_sync_summary.health.stale_pending", count: stale_pending_excluded) %></span>
|
||||
</div>
|
||||
<% if stale_pending_details.any? %>
|
||||
<details class="mt-1">
|
||||
<summary class="text-xs cursor-pointer text-secondary hover:text-primary">
|
||||
<%= t("provider_sync_summary.health.view_stale_pending") %>
|
||||
</summary>
|
||||
<div class="mt-1 pl-2 border-l-2 border-surface-inset space-y-1">
|
||||
<% stale_pending_details.each do |detail| %>
|
||||
<p class="text-xs text-warning">
|
||||
<%= detail["account_name"] %>: <%= t("provider_sync_summary.health.stale_pending_count", count: detail["count"]) %>
|
||||
</p>
|
||||
<% end %>
|
||||
</div>
|
||||
</details>
|
||||
<% end %>
|
||||
</div>
|
||||
<% end %>
|
||||
|
||||
<%# Stale unmatched pending (need manual review) %>
|
||||
<% if has_stale_unmatched_pending? %>
|
||||
<div class="mt-1">
|
||||
<div class="flex items-center gap-1">
|
||||
<%= helpers.icon "help-circle", size: "sm" %>
|
||||
<span class="text-secondary"><%= t("provider_sync_summary.health.stale_unmatched", count: stale_unmatched_pending) %></span>
|
||||
</div>
|
||||
<% if stale_unmatched_details.any? %>
|
||||
<details class="mt-1">
|
||||
<summary class="text-xs cursor-pointer text-secondary hover:text-primary">
|
||||
<%= t("provider_sync_summary.health.view_stale_unmatched") %>
|
||||
</summary>
|
||||
<div class="mt-1 pl-2 border-l-2 border-surface-inset space-y-1">
|
||||
<% stale_unmatched_details.each do |detail| %>
|
||||
<p class="text-xs text-secondary">
|
||||
<%= detail["account_name"] %>: <%= t("provider_sync_summary.health.stale_unmatched_count", count: detail["count"]) %>
|
||||
</p>
|
||||
<% end %>
|
||||
</div>
|
||||
</details>
|
||||
<% end %>
|
||||
</div>
|
||||
<% end %>
|
||||
|
||||
<%# Data quality warnings %>
|
||||
<% if has_data_quality_issues? %>
|
||||
<div class="flex items-center gap-3 mt-1">
|
||||
|
||||
@@ -127,6 +127,58 @@ class ProviderSyncSummary < ViewComponent::Base
|
||||
total_errors > 0
|
||||
end
|
||||
|
||||
# Stale pending transactions (auto-excluded)
|
||||
def stale_pending_excluded
|
||||
stats["stale_pending_excluded"].to_i
|
||||
end
|
||||
|
||||
def has_stale_pending?
|
||||
stale_pending_excluded > 0
|
||||
end
|
||||
|
||||
def stale_pending_details
|
||||
stats["stale_pending_details"] || []
|
||||
end
|
||||
|
||||
# Stale unmatched pending (need manual review - couldn't be automatically matched)
|
||||
def stale_unmatched_pending
|
||||
stats["stale_unmatched_pending"].to_i
|
||||
end
|
||||
|
||||
def has_stale_unmatched_pending?
|
||||
stale_unmatched_pending > 0
|
||||
end
|
||||
|
||||
def stale_unmatched_details
|
||||
stats["stale_unmatched_details"] || []
|
||||
end
|
||||
|
||||
# Pending→posted reconciliation stats
|
||||
def pending_reconciled
|
||||
stats["pending_reconciled"].to_i
|
||||
end
|
||||
|
||||
def has_pending_reconciled?
|
||||
pending_reconciled > 0
|
||||
end
|
||||
|
||||
def pending_reconciled_details
|
||||
stats["pending_reconciled_details"] || []
|
||||
end
|
||||
|
||||
# Duplicate suggestions needing user review
|
||||
def duplicate_suggestions_created
|
||||
stats["duplicate_suggestions_created"].to_i
|
||||
end
|
||||
|
||||
def has_duplicate_suggestions_created?
|
||||
duplicate_suggestions_created > 0
|
||||
end
|
||||
|
||||
def duplicate_suggestions_details
|
||||
stats["duplicate_suggestions_details"] || []
|
||||
end
|
||||
|
||||
# Data quality / warnings
|
||||
def data_warnings
|
||||
stats["data_warnings"].to_i
|
||||
|
||||
@@ -35,8 +35,8 @@ class AccountsController < ApplicationController
|
||||
def show
|
||||
@chart_view = params[:chart_view] || "balance"
|
||||
@tab = params[:tab]
|
||||
@q = params.fetch(:q, {}).permit(:search)
|
||||
entries = @account.entries.search(@q).reverse_chronological
|
||||
@q = params.fetch(:q, {}).permit(:search, status: [])
|
||||
entries = @account.entries.where(excluded: false).search(@q).reverse_chronological
|
||||
|
||||
@pagy, @entries = pagy(entries, limit: params[:per_page] || "10")
|
||||
|
||||
|
||||
@@ -58,6 +58,10 @@ class Settings::HostingsController < ApplicationController
|
||||
Setting.securities_provider = hosting_params[:securities_provider]
|
||||
end
|
||||
|
||||
if hosting_params.key?(:syncs_include_pending)
|
||||
Setting.syncs_include_pending = hosting_params[:syncs_include_pending] == "1"
|
||||
end
|
||||
|
||||
if hosting_params.key?(:openai_access_token)
|
||||
token_param = hosting_params[:openai_access_token].to_s.strip
|
||||
# Ignore blanks and redaction placeholders to prevent accidental overwrite
|
||||
@@ -99,7 +103,7 @@ class Settings::HostingsController < ApplicationController
|
||||
|
||||
private
|
||||
def hosting_params
|
||||
params.require(:setting).permit(:onboarding_state, :require_email_confirmation, :brand_fetch_client_id, :twelve_data_api_key, :openai_access_token, :openai_uri_base, :openai_model, :openai_json_mode, :exchange_rate_provider, :securities_provider)
|
||||
params.require(:setting).permit(:onboarding_state, :require_email_confirmation, :brand_fetch_client_id, :twelve_data_api_key, :openai_access_token, :openai_uri_base, :openai_model, :openai_json_mode, :exchange_rate_provider, :securities_provider, :syncs_include_pending)
|
||||
end
|
||||
|
||||
def ensure_admin
|
||||
|
||||
@@ -116,6 +116,38 @@ class TransactionsController < ApplicationController
|
||||
end
|
||||
end
|
||||
|
||||
def merge_duplicate
|
||||
transaction = Current.family.transactions.includes(entry: :account).find(params[:id])
|
||||
|
||||
if transaction.merge_with_duplicate!
|
||||
flash[:notice] = t("transactions.merge_duplicate.success")
|
||||
else
|
||||
flash[:alert] = t("transactions.merge_duplicate.failure")
|
||||
end
|
||||
|
||||
redirect_to transactions_path
|
||||
rescue ActiveRecord::RecordNotDestroyed, ActiveRecord::RecordInvalid => e
|
||||
Rails.logger.error("Failed to merge duplicate transaction #{params[:id]}: #{e.message}")
|
||||
flash[:alert] = t("transactions.merge_duplicate.failure")
|
||||
redirect_to transactions_path
|
||||
end
|
||||
|
||||
def dismiss_duplicate
|
||||
transaction = Current.family.transactions.includes(entry: :account).find(params[:id])
|
||||
|
||||
if transaction.dismiss_duplicate_suggestion!
|
||||
flash[:notice] = t("transactions.dismiss_duplicate.success")
|
||||
else
|
||||
flash[:alert] = t("transactions.dismiss_duplicate.failure")
|
||||
end
|
||||
|
||||
redirect_back_or_to transactions_path
|
||||
rescue ActiveRecord::RecordInvalid => e
|
||||
Rails.logger.error("Failed to dismiss duplicate suggestion for transaction #{params[:id]}: #{e.message}")
|
||||
flash[:alert] = t("transactions.dismiss_duplicate.failure")
|
||||
redirect_back_or_to transactions_path
|
||||
end
|
||||
|
||||
def mark_as_recurring
|
||||
transaction = Current.family.transactions.includes(entry: :account).find(params[:id])
|
||||
|
||||
@@ -205,7 +237,7 @@ class TransactionsController < ApplicationController
|
||||
:start_date, :end_date, :search, :amount,
|
||||
:amount_operator, :active_accounts_only,
|
||||
accounts: [], account_ids: [],
|
||||
categories: [], merchants: [], types: [], tags: []
|
||||
categories: [], merchants: [], types: [], tags: [], status: []
|
||||
)
|
||||
.to_h
|
||||
.compact_blank
|
||||
|
||||
@@ -1,13 +1,14 @@
|
||||
module TransactionsHelper
|
||||
def transaction_search_filters
|
||||
[
|
||||
{ key: "account_filter", label: "Account", icon: "layers" },
|
||||
{ key: "date_filter", label: "Date", icon: "calendar" },
|
||||
{ key: "type_filter", label: "Type", icon: "tag" },
|
||||
{ key: "amount_filter", label: "Amount", icon: "hash" },
|
||||
{ key: "category_filter", label: "Category", icon: "shapes" },
|
||||
{ key: "tag_filter", label: "Tag", icon: "tags" },
|
||||
{ key: "merchant_filter", label: "Merchant", icon: "store" }
|
||||
{ key: "account_filter", label: t("transactions.search.filters.account"), icon: "layers" },
|
||||
{ key: "date_filter", label: t("transactions.search.filters.date"), icon: "calendar" },
|
||||
{ key: "type_filter", label: t("transactions.search.filters.type"), icon: "tag" },
|
||||
{ key: "status_filter", label: t("transactions.search.filters.status"), icon: "clock" },
|
||||
{ key: "amount_filter", label: t("transactions.search.filters.amount"), icon: "hash" },
|
||||
{ key: "category_filter", label: t("transactions.search.filters.category"), icon: "shapes" },
|
||||
{ key: "tag_filter", label: t("transactions.search.filters.tag"), icon: "tags" },
|
||||
{ key: "merchant_filter", label: t("transactions.search.filters.merchant"), icon: "store" }
|
||||
]
|
||||
end
|
||||
|
||||
|
||||
@@ -16,9 +16,10 @@ class Account::ProviderImportAdapter
|
||||
# @param category_id [Integer, nil] Optional category ID
|
||||
# @param merchant [Merchant, nil] Optional merchant object
|
||||
# @param notes [String, nil] Optional transaction notes/memo
|
||||
# @param pending_transaction_id [String, nil] Plaid's linking ID for pending→posted reconciliation
|
||||
# @param extra [Hash, nil] Optional provider-specific metadata to merge into transaction.extra
|
||||
# @return [Entry] The created or updated entry
|
||||
def import_transaction(external_id:, amount:, currency:, date:, name:, source:, category_id: nil, merchant: nil, notes: nil, extra: nil)
|
||||
def import_transaction(external_id:, amount:, currency:, date:, name:, source:, category_id: nil, merchant: nil, notes: nil, pending_transaction_id: nil, extra: nil)
|
||||
raise ArgumentError, "external_id is required" if external_id.blank?
|
||||
raise ArgumentError, "source is required" if source.blank?
|
||||
|
||||
@@ -43,6 +44,42 @@ class Account::ProviderImportAdapter
|
||||
end
|
||||
end
|
||||
|
||||
# If still a new entry and this is a POSTED transaction, check for matching pending transactions
|
||||
incoming_pending = extra.is_a?(Hash) && (
|
||||
ActiveModel::Type::Boolean.new.cast(extra.dig("simplefin", "pending")) ||
|
||||
ActiveModel::Type::Boolean.new.cast(extra.dig("plaid", "pending"))
|
||||
)
|
||||
|
||||
if entry.new_record? && !incoming_pending
|
||||
pending_match = nil
|
||||
|
||||
# PRIORITY 1: Use Plaid's pending_transaction_id if provided (most reliable)
|
||||
# Plaid explicitly links pending→posted with this ID - no guessing required
|
||||
if pending_transaction_id.present?
|
||||
pending_match = account.entries.find_by(external_id: pending_transaction_id, source: source)
|
||||
if pending_match
|
||||
Rails.logger.info("Reconciling pending→posted via Plaid pending_transaction_id: claiming entry #{pending_match.id} (#{pending_match.name}) with new external_id #{external_id}")
|
||||
end
|
||||
end
|
||||
|
||||
# PRIORITY 2: Fallback to EXACT amount match (for SimpleFIN and providers without linking IDs)
|
||||
# Only searches backward in time - pending date must be <= posted date
|
||||
if pending_match.nil?
|
||||
pending_match = find_pending_transaction(date: date, amount: amount, currency: currency, source: source)
|
||||
if pending_match
|
||||
Rails.logger.info("Reconciling pending→posted via exact amount match: claiming entry #{pending_match.id} (#{pending_match.name}) with new external_id #{external_id}")
|
||||
end
|
||||
end
|
||||
|
||||
if pending_match
|
||||
entry = pending_match
|
||||
entry.assign_attributes(external_id: external_id)
|
||||
end
|
||||
end
|
||||
|
||||
# Track if this is a new posted transaction (for fuzzy suggestion after save)
|
||||
is_new_posted = entry.new_record? && !incoming_pending
|
||||
|
||||
# Validate entryable type matches to prevent external_id collisions
|
||||
if entry.persisted? && !entry.entryable.is_a?(Transaction)
|
||||
raise ArgumentError, "Entry with external_id '#{external_id}' already exists with different entryable type: #{entry.entryable_type}"
|
||||
@@ -78,6 +115,60 @@ class Account::ProviderImportAdapter
|
||||
entry.transaction.save!
|
||||
end
|
||||
entry.save!
|
||||
|
||||
# AFTER save: For NEW posted transactions, check for fuzzy matches to SUGGEST (not auto-claim)
|
||||
# This handles tip adjustments where auto-matching is too risky
|
||||
if is_new_posted
|
||||
# PRIORITY 1: Try medium-confidence fuzzy match (≤30% amount difference)
|
||||
fuzzy_suggestion = find_pending_transaction_fuzzy(
|
||||
date: date,
|
||||
amount: amount,
|
||||
currency: currency,
|
||||
source: source,
|
||||
merchant_id: merchant&.id,
|
||||
name: name
|
||||
)
|
||||
if fuzzy_suggestion
|
||||
# Store suggestion on the PENDING entry for user to review
|
||||
begin
|
||||
store_duplicate_suggestion(
|
||||
pending_entry: fuzzy_suggestion,
|
||||
posted_entry: entry,
|
||||
reason: "fuzzy_amount_match",
|
||||
posted_amount: amount,
|
||||
confidence: "medium"
|
||||
)
|
||||
Rails.logger.info("Suggested potential duplicate (medium confidence): pending entry #{fuzzy_suggestion.id} (#{fuzzy_suggestion.name}, #{fuzzy_suggestion.amount}) may match posted #{entry.name} (#{amount})")
|
||||
rescue ActiveRecord::RecordInvalid => e
|
||||
Rails.logger.warn("Failed to store duplicate suggestion for entry #{fuzzy_suggestion.id}: #{e.message}")
|
||||
end
|
||||
else
|
||||
# PRIORITY 2: Try low-confidence match (>30% to 100% difference - big tips)
|
||||
low_confidence_suggestion = find_pending_transaction_low_confidence(
|
||||
date: date,
|
||||
amount: amount,
|
||||
currency: currency,
|
||||
source: source,
|
||||
merchant_id: merchant&.id,
|
||||
name: name
|
||||
)
|
||||
if low_confidence_suggestion
|
||||
begin
|
||||
store_duplicate_suggestion(
|
||||
pending_entry: low_confidence_suggestion,
|
||||
posted_entry: entry,
|
||||
reason: "low_confidence_match",
|
||||
posted_amount: amount,
|
||||
confidence: "low"
|
||||
)
|
||||
Rails.logger.info("Suggested potential duplicate (low confidence): pending entry #{low_confidence_suggestion.id} (#{low_confidence_suggestion.name}, #{low_confidence_suggestion.amount}) may match posted #{entry.name} (#{amount})")
|
||||
rescue ActiveRecord::RecordInvalid => e
|
||||
Rails.logger.warn("Failed to store duplicate suggestion for entry #{low_confidence_suggestion.id}: #{e.message}")
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
entry
|
||||
end
|
||||
end
|
||||
@@ -444,4 +535,196 @@ class Account::ProviderImportAdapter
|
||||
|
||||
query.order(created_at: :asc).first
|
||||
end
|
||||
|
||||
# Finds a pending transaction that likely matches a newly posted transaction
|
||||
# Used to reconcile pending→posted when SimpleFIN gives different IDs for the same transaction
|
||||
#
|
||||
# @param date [Date, String] Posted transaction date
|
||||
# @param amount [BigDecimal, Numeric] Transaction amount (must match exactly)
|
||||
# @param currency [String] Currency code
|
||||
# @param source [String] Provider name (e.g., "simplefin")
|
||||
# @param date_window [Integer] Days to search around the posted date (default: 8)
|
||||
# @return [Entry, nil] The pending entry or nil if not found
|
||||
def find_pending_transaction(date:, amount:, currency:, source:, date_window: 8)
|
||||
date = Date.parse(date.to_s) unless date.is_a?(Date)
|
||||
|
||||
# Look for entries that:
|
||||
# 1. Same account (implicit via account.entries)
|
||||
# 2. Same source (simplefin)
|
||||
# 3. Same amount (exact match - this is the strongest signal)
|
||||
# 4. Same currency
|
||||
# 5. Date within window (pending can post days later)
|
||||
# 6. Is a Transaction (not Trade or Valuation)
|
||||
# 7. Has pending=true in transaction.extra["simplefin"]["pending"] or extra["plaid"]["pending"]
|
||||
candidates = account.entries
|
||||
.joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'")
|
||||
.where(source: source)
|
||||
.where(amount: amount)
|
||||
.where(currency: currency)
|
||||
.where(date: (date - date_window.days)..date) # Pending must be ON or BEFORE posted date
|
||||
.where(<<~SQL.squish)
|
||||
(transactions.extra -> 'simplefin' ->> 'pending')::boolean = true
|
||||
OR (transactions.extra -> 'plaid' ->> 'pending')::boolean = true
|
||||
SQL
|
||||
.order(date: :desc) # Prefer most recent pending transaction
|
||||
|
||||
candidates.first
|
||||
end
|
||||
|
||||
# Finds a pending transaction using fuzzy amount matching for tip adjustments
|
||||
# Used when exact amount matching fails - handles restaurant tips, adjusted authorizations, etc.
|
||||
#
|
||||
# IMPORTANT: Only returns a match if there's exactly ONE candidate to avoid false positives
|
||||
# with recurring merchant transactions (e.g., gas stations, coffee shops).
|
||||
#
|
||||
# @param date [Date, String] Posted transaction date
|
||||
# @param amount [BigDecimal, Numeric] Posted transaction amount (typically higher due to tip)
|
||||
# @param currency [String] Currency code
|
||||
# @param source [String] Provider name (e.g., "simplefin")
|
||||
# @param merchant_id [Integer, nil] Merchant ID for more accurate matching
|
||||
# @param name [String, nil] Transaction name for fuzzy name matching
|
||||
# @param date_window [Integer] Days to search backward from posted date (default: 3 for fuzzy)
|
||||
# @param amount_tolerance [Float] Maximum percentage difference allowed (default: 0.30 = 30%)
|
||||
# @return [Entry, nil] The pending entry or nil if not found/ambiguous
|
||||
def find_pending_transaction_fuzzy(date:, amount:, currency:, source:, merchant_id: nil, name: nil, date_window: 3, amount_tolerance: 0.30)
|
||||
date = Date.parse(date.to_s) unless date.is_a?(Date)
|
||||
amount = BigDecimal(amount.to_s)
|
||||
|
||||
# Calculate amount bounds using ABS to handle both positive and negative amounts
|
||||
# Posted amount should be >= pending (tips add, not subtract)
|
||||
# Allow posted to be up to 30% higher than pending (covers typical tips)
|
||||
abs_amount = amount.abs
|
||||
min_pending_abs = abs_amount / (1 + amount_tolerance) # If posted is 100, pending could be as low as ~77
|
||||
max_pending_abs = abs_amount # Pending should not be higher than posted
|
||||
|
||||
# Build base query for pending transactions
|
||||
# CRITICAL: Pending must be ON or BEFORE the posted date (authorization happens first)
|
||||
# Use tighter date window (3 days) - tips post quickly, not a week later
|
||||
# Use ABS() for amount comparison to handle negative amounts correctly
|
||||
candidates = account.entries
|
||||
.joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'")
|
||||
.where(source: source)
|
||||
.where(currency: currency)
|
||||
.where(date: (date - date_window.days)..date) # Pending ON or BEFORE posted
|
||||
.where("ABS(entries.amount) BETWEEN ? AND ?", min_pending_abs, max_pending_abs)
|
||||
.where(<<~SQL.squish)
|
||||
(transactions.extra -> 'simplefin' ->> 'pending')::boolean = true
|
||||
OR (transactions.extra -> 'plaid' ->> 'pending')::boolean = true
|
||||
SQL
|
||||
|
||||
# If merchant_id is provided, prioritize matching by merchant
|
||||
if merchant_id.present?
|
||||
merchant_matches = candidates.where("transactions.merchant_id = ?", merchant_id).to_a
|
||||
# Only match if exactly ONE candidate to avoid false positives
|
||||
return merchant_matches.first if merchant_matches.size == 1
|
||||
if merchant_matches.size > 1
|
||||
Rails.logger.info("Skipping fuzzy pending match: #{merchant_matches.size} ambiguous merchant candidates for amount=#{amount} date=#{date}")
|
||||
end
|
||||
end
|
||||
|
||||
# If name is provided, try fuzzy name matching as fallback
|
||||
if name.present?
|
||||
# Extract first few significant words for comparison
|
||||
name_words = name.downcase.gsub(/[^a-z0-9\s]/, "").split.first(3).join(" ")
|
||||
if name_words.present?
|
||||
name_matches = candidates.select do |c|
|
||||
c_name_words = c.name.downcase.gsub(/[^a-z0-9\s]/, "").split.first(3).join(" ")
|
||||
name_words == c_name_words
|
||||
end
|
||||
# Only match if exactly ONE candidate to avoid false positives
|
||||
return name_matches.first if name_matches.size == 1
|
||||
if name_matches.size > 1
|
||||
Rails.logger.info("Skipping fuzzy pending match: #{name_matches.size} ambiguous name candidates for '#{name_words}' amount=#{amount} date=#{date}")
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
# No merchant or name match, return nil (too risky to match on amount alone)
|
||||
# This prevents false positives when multiple pending transactions exist
|
||||
nil
|
||||
end
|
||||
|
||||
# Finds a pending transaction with low confidence (>30% to 100% amount difference)
|
||||
# Used for large tip scenarios where normal fuzzy matching would miss
|
||||
# Creates a "review recommended" suggestion rather than "possible duplicate"
|
||||
#
|
||||
# @param date [Date, String] Posted transaction date
|
||||
# @param amount [BigDecimal, Numeric] Posted transaction amount
|
||||
# @param currency [String] Currency code
|
||||
# @param source [String] Provider name
|
||||
# @param merchant_id [Integer, nil] Merchant ID for matching
|
||||
# @param name [String, nil] Transaction name for matching
|
||||
# @param date_window [Integer] Days to search backward (default: 3)
|
||||
# @return [Entry, nil] The pending entry or nil if not found/ambiguous
|
||||
def find_pending_transaction_low_confidence(date:, amount:, currency:, source:, merchant_id: nil, name: nil, date_window: 3)
|
||||
date = Date.parse(date.to_s) unless date.is_a?(Date)
|
||||
amount = BigDecimal(amount.to_s)
|
||||
|
||||
# Allow up to 100% difference (e.g., $50 pending → $100 posted with huge tip)
|
||||
# This is low confidence - requires strong name/merchant match
|
||||
# Use ABS to handle both positive and negative amounts correctly
|
||||
abs_amount = amount.abs
|
||||
min_pending_abs = abs_amount / 2.0 # Posted could be up to 2x pending
|
||||
max_pending_abs = abs_amount * 0.77 # Pending must be at least 30% less (to not overlap with fuzzy)
|
||||
|
||||
# Build base query for pending transactions
|
||||
# Use ABS() for amount comparison to handle negative amounts correctly
|
||||
candidates = account.entries
|
||||
.joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'")
|
||||
.where(source: source)
|
||||
.where(currency: currency)
|
||||
.where(date: (date - date_window.days)..date)
|
||||
.where("ABS(entries.amount) BETWEEN ? AND ?", min_pending_abs, max_pending_abs)
|
||||
.where(<<~SQL.squish)
|
||||
(transactions.extra -> 'simplefin' ->> 'pending')::boolean = true
|
||||
OR (transactions.extra -> 'plaid' ->> 'pending')::boolean = true
|
||||
SQL
|
||||
|
||||
# For low confidence, require BOTH merchant AND name match (stronger signal needed)
|
||||
if merchant_id.present? && name.present?
|
||||
name_words = name.downcase.gsub(/[^a-z0-9\s]/, "").split.first(3).join(" ")
|
||||
return nil if name_words.blank?
|
||||
|
||||
merchant_matches = candidates.where("transactions.merchant_id = ?", merchant_id).to_a
|
||||
name_matches = merchant_matches.select do |c|
|
||||
c_name_words = c.name.downcase.gsub(/[^a-z0-9\s]/, "").split.first(3).join(" ")
|
||||
name_words == c_name_words
|
||||
end
|
||||
|
||||
# Only match if exactly ONE candidate
|
||||
return name_matches.first if name_matches.size == 1
|
||||
end
|
||||
|
||||
nil
|
||||
end
|
||||
|
||||
# Stores a duplicate suggestion on a pending entry for user review
|
||||
# The suggestion is stored in the pending transaction's extra field
|
||||
#
|
||||
# @param pending_entry [Entry] The pending entry that may be a duplicate
|
||||
# @param posted_entry [Entry] The posted entry it may match
|
||||
# @param reason [String] Why this was flagged (e.g., "fuzzy_amount_match", "low_confidence_match")
|
||||
# @param posted_amount [BigDecimal] The posted transaction amount
|
||||
# @param confidence [String] Confidence level: "medium" (≤30% diff) or "low" (>30% diff)
|
||||
def store_duplicate_suggestion(pending_entry:, posted_entry:, reason:, posted_amount:, confidence: "medium")
|
||||
return unless pending_entry&.entryable.is_a?(Transaction)
|
||||
|
||||
pending_transaction = pending_entry.entryable
|
||||
existing_extra = pending_transaction.extra || {}
|
||||
|
||||
# Don't overwrite if already has a suggestion (keep first one found)
|
||||
return if existing_extra["potential_posted_match"].present?
|
||||
|
||||
pending_transaction.update!(
|
||||
extra: existing_extra.merge(
|
||||
"potential_posted_match" => {
|
||||
"entry_id" => posted_entry.id,
|
||||
"reason" => reason,
|
||||
"posted_amount" => posted_amount.to_s,
|
||||
"confidence" => confidence,
|
||||
"detected_at" => Date.current.to_s
|
||||
}
|
||||
)
|
||||
)
|
||||
end
|
||||
end
|
||||
|
||||
@@ -35,6 +35,188 @@ class Entry < ApplicationRecord
|
||||
)
|
||||
}
|
||||
|
||||
# Pending transaction scopes - check Transaction.extra for provider pending flags
|
||||
# Works with any provider that stores pending status in extra["provider_name"]["pending"]
|
||||
scope :pending, -> {
|
||||
joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'")
|
||||
.where(<<~SQL.squish)
|
||||
(transactions.extra -> 'simplefin' ->> 'pending')::boolean = true
|
||||
OR (transactions.extra -> 'plaid' ->> 'pending')::boolean = true
|
||||
SQL
|
||||
}
|
||||
|
||||
scope :excluding_pending, -> {
|
||||
# For non-Transaction entries (Trade, Valuation), always include
|
||||
# For Transaction entries, exclude if pending flag is true
|
||||
where(<<~SQL.squish)
|
||||
entries.entryable_type != 'Transaction'
|
||||
OR NOT EXISTS (
|
||||
SELECT 1 FROM transactions t
|
||||
WHERE t.id = entries.entryable_id
|
||||
AND (
|
||||
(t.extra -> 'simplefin' ->> 'pending')::boolean = true
|
||||
OR (t.extra -> 'plaid' ->> 'pending')::boolean = true
|
||||
)
|
||||
)
|
||||
SQL
|
||||
}
|
||||
|
||||
# Find stale pending transactions (pending for more than X days with no matching posted version)
|
||||
scope :stale_pending, ->(days: 8) {
|
||||
pending.where("entries.date < ?", days.days.ago.to_date)
|
||||
}
|
||||
|
||||
# Auto-exclude stale pending transactions for an account
|
||||
# Called during sync to clean up pending transactions that never posted
|
||||
# @param account [Account] The account to clean up
|
||||
# @param days [Integer] Number of days after which pending is considered stale (default: 8)
|
||||
# @return [Integer] Number of entries excluded
|
||||
def self.auto_exclude_stale_pending(account:, days: 8)
|
||||
stale_entries = account.entries.stale_pending(days: days).where(excluded: false)
|
||||
count = stale_entries.count
|
||||
|
||||
if count > 0
|
||||
stale_entries.update_all(excluded: true, updated_at: Time.current)
|
||||
Rails.logger.info("Auto-excluded #{count} stale pending transaction(s) for account #{account.id} (#{account.name})")
|
||||
end
|
||||
|
||||
count
|
||||
end
|
||||
|
||||
# Retroactively reconcile pending transactions that have a matching posted version
|
||||
# This handles duplicates created before reconciliation code was deployed
|
||||
#
|
||||
# @param account [Account, nil] Specific account to clean up, or nil for all accounts
|
||||
# @param dry_run [Boolean] If true, only report what would be done without making changes
|
||||
# @param date_window [Integer] Days to search forward for posted matches (default: 8)
|
||||
# @param amount_tolerance [Float] Percentage difference allowed for fuzzy matching (default: 0.25)
|
||||
# @return [Hash] Stats about what was reconciled
|
||||
def self.reconcile_pending_duplicates(account: nil, dry_run: false, date_window: 8, amount_tolerance: 0.25)
|
||||
stats = { checked: 0, reconciled: 0, details: [] }
|
||||
|
||||
# Get pending entries to check
|
||||
scope = Entry.pending.where(excluded: false)
|
||||
scope = scope.where(account: account) if account
|
||||
|
||||
scope.includes(:account, :entryable).find_each do |pending_entry|
|
||||
stats[:checked] += 1
|
||||
acct = pending_entry.account
|
||||
|
||||
# PRIORITY 1: Look for posted transaction with EXACT amount match
|
||||
# CRITICAL: Only search forward in time - posted date must be >= pending date
|
||||
exact_candidates = acct.entries
|
||||
.joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'")
|
||||
.where.not(id: pending_entry.id)
|
||||
.where(currency: pending_entry.currency)
|
||||
.where(amount: pending_entry.amount)
|
||||
.where(date: pending_entry.date..(pending_entry.date + date_window.days)) # Posted must be ON or AFTER pending date
|
||||
.where(<<~SQL.squish)
|
||||
(transactions.extra -> 'simplefin' ->> 'pending')::boolean IS NOT TRUE
|
||||
AND (transactions.extra -> 'plaid' ->> 'pending')::boolean IS NOT TRUE
|
||||
SQL
|
||||
.limit(2) # Only need to know if 0, 1, or 2+ candidates
|
||||
.to_a # Load limited records to avoid COUNT(*) on .size
|
||||
|
||||
# Handle exact match - auto-exclude only if exactly ONE candidate (high confidence)
|
||||
# Multiple candidates = ambiguous = skip to avoid excluding wrong entry
|
||||
if exact_candidates.size == 1
|
||||
posted_match = exact_candidates.first
|
||||
detail = {
|
||||
pending_id: pending_entry.id,
|
||||
pending_name: pending_entry.name,
|
||||
pending_amount: pending_entry.amount.to_f,
|
||||
pending_date: pending_entry.date,
|
||||
posted_id: posted_match.id,
|
||||
posted_name: posted_match.name,
|
||||
posted_amount: posted_match.amount.to_f,
|
||||
posted_date: posted_match.date,
|
||||
account: acct.name,
|
||||
match_type: "exact"
|
||||
}
|
||||
stats[:details] << detail
|
||||
stats[:reconciled] += 1
|
||||
|
||||
unless dry_run
|
||||
pending_entry.update!(excluded: true)
|
||||
Rails.logger.info("Reconciled pending→posted duplicate: excluded entry #{pending_entry.id} (#{pending_entry.name}) matched to #{posted_match.id}")
|
||||
end
|
||||
next
|
||||
end
|
||||
|
||||
# PRIORITY 2: If no exact match, try fuzzy amount match for tip adjustments
|
||||
# Store as SUGGESTION instead of auto-excluding (medium confidence)
|
||||
pending_amount = pending_entry.amount.abs
|
||||
min_amount = pending_amount
|
||||
max_amount = pending_amount * (1 + amount_tolerance)
|
||||
|
||||
fuzzy_date_window = 3
|
||||
candidates = acct.entries
|
||||
.joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'")
|
||||
.where.not(id: pending_entry.id)
|
||||
.where(currency: pending_entry.currency)
|
||||
.where(date: pending_entry.date..(pending_entry.date + fuzzy_date_window.days)) # Posted ON or AFTER pending
|
||||
.where("ABS(entries.amount) BETWEEN ? AND ?", min_amount, max_amount)
|
||||
.where(<<~SQL.squish)
|
||||
(transactions.extra -> 'simplefin' ->> 'pending')::boolean IS NOT TRUE
|
||||
AND (transactions.extra -> 'plaid' ->> 'pending')::boolean IS NOT TRUE
|
||||
SQL
|
||||
|
||||
# Match by name similarity (first 3 words)
|
||||
name_words = pending_entry.name.downcase.gsub(/[^a-z0-9\s]/, "").split.first(3).join(" ")
|
||||
if name_words.present?
|
||||
matching_candidates = candidates.select do |c|
|
||||
c_words = c.name.downcase.gsub(/[^a-z0-9\s]/, "").split.first(3).join(" ")
|
||||
name_words == c_words
|
||||
end
|
||||
|
||||
# Only suggest if there's exactly ONE matching candidate
|
||||
# Multiple matches = ambiguous (e.g., recurring gas station visits) = skip
|
||||
if matching_candidates.size == 1
|
||||
fuzzy_match = matching_candidates.first
|
||||
|
||||
detail = {
|
||||
pending_id: pending_entry.id,
|
||||
pending_name: pending_entry.name,
|
||||
pending_amount: pending_entry.amount.to_f,
|
||||
pending_date: pending_entry.date,
|
||||
posted_id: fuzzy_match.id,
|
||||
posted_name: fuzzy_match.name,
|
||||
posted_amount: fuzzy_match.amount.to_f,
|
||||
posted_date: fuzzy_match.date,
|
||||
account: acct.name,
|
||||
match_type: "fuzzy_suggestion"
|
||||
}
|
||||
stats[:details] << detail
|
||||
|
||||
unless dry_run
|
||||
# Store suggestion on the pending entry instead of auto-excluding
|
||||
pending_transaction = pending_entry.entryable
|
||||
if pending_transaction.is_a?(Transaction)
|
||||
existing_extra = pending_transaction.extra || {}
|
||||
unless existing_extra["potential_posted_match"].present?
|
||||
pending_transaction.update!(
|
||||
extra: existing_extra.merge(
|
||||
"potential_posted_match" => {
|
||||
"entry_id" => fuzzy_match.id,
|
||||
"reason" => "fuzzy_amount_match",
|
||||
"posted_amount" => fuzzy_match.amount.to_s,
|
||||
"detected_at" => Date.current.to_s
|
||||
}
|
||||
)
|
||||
)
|
||||
Rails.logger.info("Stored duplicate suggestion for entry #{pending_entry.id} (#{pending_entry.name}) → #{fuzzy_match.id}")
|
||||
end
|
||||
end
|
||||
end
|
||||
elsif matching_candidates.size > 1
|
||||
Rails.logger.info("Skipping fuzzy reconciliation for #{pending_entry.id} (#{pending_entry.name}): #{matching_candidates.size} ambiguous candidates")
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
stats
|
||||
end
|
||||
|
||||
def classification
|
||||
amount.negative? ? "income" : "expense"
|
||||
end
|
||||
|
||||
@@ -6,6 +6,7 @@ class EntrySearch
|
||||
attribute :amount, :string
|
||||
attribute :amount_operator, :string
|
||||
attribute :types, :string
|
||||
attribute :status, array: true
|
||||
attribute :accounts, array: true
|
||||
attribute :account_ids, array: true
|
||||
attribute :start_date, :string
|
||||
@@ -56,6 +57,44 @@ class EntrySearch
|
||||
query = query.where(accounts: { id: account_ids }) if account_ids.present?
|
||||
query
|
||||
end
|
||||
|
||||
def apply_status_filter(scope, statuses)
|
||||
return scope unless statuses.present?
|
||||
return scope if statuses.uniq.sort == %w[confirmed pending] # Both selected = no filter
|
||||
|
||||
pending_condition = <<~SQL.squish
|
||||
entries.entryable_type = 'Transaction'
|
||||
AND EXISTS (
|
||||
SELECT 1 FROM transactions t
|
||||
WHERE t.id = entries.entryable_id
|
||||
AND (
|
||||
(t.extra -> 'simplefin' ->> 'pending')::boolean = true
|
||||
OR (t.extra -> 'plaid' ->> 'pending')::boolean = true
|
||||
)
|
||||
)
|
||||
SQL
|
||||
|
||||
confirmed_condition = <<~SQL.squish
|
||||
entries.entryable_type != 'Transaction'
|
||||
OR NOT EXISTS (
|
||||
SELECT 1 FROM transactions t
|
||||
WHERE t.id = entries.entryable_id
|
||||
AND (
|
||||
(t.extra -> 'simplefin' ->> 'pending')::boolean = true
|
||||
OR (t.extra -> 'plaid' ->> 'pending')::boolean = true
|
||||
)
|
||||
)
|
||||
SQL
|
||||
|
||||
case statuses.sort
|
||||
when [ "pending" ]
|
||||
scope.where(pending_condition)
|
||||
when [ "confirmed" ]
|
||||
scope.where(confirmed_condition)
|
||||
else
|
||||
scope
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def build_query(scope)
|
||||
@@ -64,6 +103,7 @@ class EntrySearch
|
||||
query = self.class.apply_date_filters(query, start_date, end_date)
|
||||
query = self.class.apply_amount_filter(query, amount, amount_operator)
|
||||
query = self.class.apply_accounts_filter(query, accounts, account_ids)
|
||||
query = self.class.apply_status_filter(query, status)
|
||||
query
|
||||
end
|
||||
end
|
||||
|
||||
@@ -12,7 +12,9 @@ class IncomeStatement
|
||||
end
|
||||
|
||||
def totals(transactions_scope: nil, date_range:)
|
||||
transactions_scope ||= family.transactions.visible
|
||||
# Default to excluding pending transactions from budget/analytics calculations
|
||||
# Pending transactions shouldn't affect budget totals until they post
|
||||
transactions_scope ||= family.transactions.visible.excluding_pending
|
||||
|
||||
result = totals_query(transactions_scope: transactions_scope, date_range: date_range)
|
||||
|
||||
@@ -64,7 +66,8 @@ class IncomeStatement
|
||||
end
|
||||
|
||||
def build_period_total(classification:, period:)
|
||||
totals = totals_query(transactions_scope: family.transactions.visible.in_period(period), date_range: period.date_range).select { |t| t.classification == classification }
|
||||
# Exclude pending transactions from budget calculations
|
||||
totals = totals_query(transactions_scope: family.transactions.visible.excluding_pending.in_period(period), date_range: period.date_range).select { |t| t.classification == classification }
|
||||
classification_total = totals.sum(&:total)
|
||||
|
||||
uncategorized_category = family.categories.uncategorized
|
||||
|
||||
@@ -49,6 +49,8 @@ class IncomeStatement::CategoryStats
|
||||
WHERE a.family_id = :family_id
|
||||
AND t.kind NOT IN ('funds_movement', 'one_time', 'cc_payment')
|
||||
AND ae.excluded = false
|
||||
AND (t.extra -> 'simplefin' ->> 'pending')::boolean IS DISTINCT FROM true
|
||||
AND (t.extra -> 'plaid' ->> 'pending')::boolean IS DISTINCT FROM true
|
||||
GROUP BY c.id, period, CASE WHEN ae.amount < 0 THEN 'income' ELSE 'expense' END
|
||||
)
|
||||
SELECT
|
||||
|
||||
@@ -46,6 +46,8 @@ class IncomeStatement::FamilyStats
|
||||
WHERE a.family_id = :family_id
|
||||
AND t.kind NOT IN ('funds_movement', 'one_time', 'cc_payment')
|
||||
AND ae.excluded = false
|
||||
AND (t.extra -> 'simplefin' ->> 'pending')::boolean IS DISTINCT FROM true
|
||||
AND (t.extra -> 'plaid' ->> 'pending')::boolean IS DISTINCT FROM true
|
||||
GROUP BY period, CASE WHEN ae.amount < 0 THEN 'income' ELSE 'expense' END
|
||||
)
|
||||
SELECT
|
||||
|
||||
@@ -51,7 +51,20 @@ class PlaidAccount::Transactions::Processor
|
||||
modified = plaid_account.raw_transactions_payload["modified"] || []
|
||||
added = plaid_account.raw_transactions_payload["added"] || []
|
||||
|
||||
modified + added
|
||||
transactions = modified + added
|
||||
|
||||
# Filter out pending transactions based on env var or Setting
|
||||
# Priority: env var > Setting (allows runtime changes via UI)
|
||||
include_pending = if ENV["PLAID_INCLUDE_PENDING"].present?
|
||||
Rails.configuration.x.plaid.include_pending
|
||||
else
|
||||
Setting.syncs_include_pending
|
||||
end
|
||||
unless include_pending
|
||||
transactions = transactions.reject { |t| t["pending"] == true }
|
||||
end
|
||||
|
||||
transactions
|
||||
end
|
||||
|
||||
def removed_transactions
|
||||
|
||||
@@ -16,9 +16,11 @@ class PlaidEntry::Processor
|
||||
source: "plaid",
|
||||
category_id: matched_category&.id,
|
||||
merchant: merchant,
|
||||
pending_transaction_id: pending_transaction_id, # Plaid's linking ID for pending→posted
|
||||
extra: {
|
||||
plaid: {
|
||||
pending: plaid_transaction["pending"]
|
||||
pending: plaid_transaction["pending"],
|
||||
pending_transaction_id: pending_transaction_id # Also store for reference
|
||||
}
|
||||
}
|
||||
)
|
||||
@@ -55,6 +57,12 @@ class PlaidEntry::Processor
|
||||
plaid_transaction["date"]
|
||||
end
|
||||
|
||||
# Plaid provides this linking ID when a posted transaction matches a pending one
|
||||
# This is the most reliable way to reconcile pending→posted
|
||||
def pending_transaction_id
|
||||
plaid_transaction["pending_transaction_id"]
|
||||
end
|
||||
|
||||
def detailed_category
|
||||
plaid_transaction.dig("personal_finance_category", "detailed")
|
||||
end
|
||||
|
||||
@@ -16,6 +16,15 @@ class Setting < RailsSettings::Base
|
||||
field :exchange_rate_provider, type: :string, default: ENV.fetch("EXCHANGE_RATE_PROVIDER", "twelve_data")
|
||||
field :securities_provider, type: :string, default: ENV.fetch("SECURITIES_PROVIDER", "twelve_data")
|
||||
|
||||
# Sync settings - check both provider env vars for default
|
||||
# Only defaults to true if neither provider explicitly disables pending
|
||||
SYNCS_INCLUDE_PENDING_DEFAULT = begin
|
||||
simplefin = ENV.fetch("SIMPLEFIN_INCLUDE_PENDING", "1") == "1"
|
||||
plaid = ENV.fetch("PLAID_INCLUDE_PENDING", "1") == "1"
|
||||
simplefin && plaid
|
||||
end
|
||||
field :syncs_include_pending, type: :boolean, default: SYNCS_INCLUDE_PENDING_DEFAULT
|
||||
|
||||
# Dynamic fields are now stored as individual entries with "dynamic:" prefix
|
||||
# This prevents race conditions and ensures each field is independently managed
|
||||
|
||||
|
||||
@@ -34,9 +34,27 @@ class SimplefinEntry::Processor
|
||||
# Include provider-supplied extra hash if present
|
||||
sf["extra"] = data[:extra] if data[:extra].is_a?(Hash)
|
||||
|
||||
# Pending detection: only use explicit provider flag
|
||||
# Pending detection: explicit flag OR inferred from posted=0 + transacted_at
|
||||
# SimpleFIN indicates pending via:
|
||||
# 1. pending: true (explicit flag)
|
||||
# 2. posted=0 (epoch zero) + transacted_at present (implicit - some banks use this pattern)
|
||||
#
|
||||
# Note: We only infer from posted=0, NOT from posted=nil/blank, because some providers
|
||||
# don't supply posted dates even for settled transactions (would cause false positives).
|
||||
# We always set the key (true or false) to ensure deep_merge overwrites any stale value
|
||||
if ActiveModel::Type::Boolean.new.cast(data[:pending])
|
||||
is_pending = if ActiveModel::Type::Boolean.new.cast(data[:pending])
|
||||
true
|
||||
else
|
||||
# Infer pending ONLY when posted is explicitly 0 (epoch) AND transacted_at is present
|
||||
# posted=nil/blank is NOT treated as pending (some providers omit posted for settled txns)
|
||||
posted_val = data[:posted]
|
||||
transacted_val = data[:transacted_at]
|
||||
posted_is_epoch_zero = posted_val.present? && posted_val.to_i.zero?
|
||||
transacted_present = transacted_val.present? && transacted_val.to_i > 0
|
||||
posted_is_epoch_zero && transacted_present
|
||||
end
|
||||
|
||||
if is_pending
|
||||
sf["pending"] = true
|
||||
Rails.logger.debug("SimpleFIN: flagged pending transaction #{external_id}")
|
||||
else
|
||||
|
||||
@@ -271,8 +271,8 @@ class SimplefinItem < ApplicationRecord
|
||||
return nil unless latest
|
||||
|
||||
# If sync has statistics, use them
|
||||
if latest.sync_stats.present?
|
||||
stats = latest.sync_stats
|
||||
stats = parse_sync_stats(latest.sync_stats)
|
||||
if stats.present?
|
||||
total = stats["total_accounts"] || 0
|
||||
linked = stats["linked_accounts"] || 0
|
||||
unlinked = stats["unlinked_accounts"] || 0
|
||||
@@ -399,7 +399,68 @@ class SimplefinItem < ApplicationRecord
|
||||
issues
|
||||
end
|
||||
|
||||
# Get reconciled duplicates count from the last sync
|
||||
# Returns { count: N, message: "..." } or { count: 0 } if none
|
||||
def last_sync_reconciled_status
|
||||
latest_sync = syncs.ordered.first
|
||||
return { count: 0 } unless latest_sync
|
||||
|
||||
stats = parse_sync_stats(latest_sync.sync_stats)
|
||||
count = stats&.dig("pending_reconciled").to_i
|
||||
if count > 0
|
||||
{
|
||||
count: count,
|
||||
message: I18n.t("simplefin_items.reconciled_status.message", count: count)
|
||||
}
|
||||
else
|
||||
{ count: 0 }
|
||||
end
|
||||
end
|
||||
|
||||
# Count stale pending transactions (>8 days old) across all linked accounts
|
||||
# Returns { count: N, accounts: [names] } or { count: 0 } if none
|
||||
def stale_pending_status(days: 8)
|
||||
# Get all accounts linked to this SimpleFIN item
|
||||
# Eager-load both association paths to avoid N+1 on current_account method
|
||||
linked_accounts = simplefin_accounts.includes(:account, :linked_account).filter_map(&:current_account)
|
||||
return { count: 0 } if linked_accounts.empty?
|
||||
|
||||
# Batch query to avoid N+1
|
||||
account_ids = linked_accounts.map(&:id)
|
||||
counts_by_account = Entry.stale_pending(days: days)
|
||||
.where(excluded: false)
|
||||
.where(account_id: account_ids)
|
||||
.group(:account_id)
|
||||
.count
|
||||
|
||||
account_counts = linked_accounts
|
||||
.map { |account| { account: account, count: counts_by_account[account.id].to_i } }
|
||||
.select { |ac| ac[:count] > 0 }
|
||||
|
||||
total = account_counts.sum { |ac| ac[:count] }
|
||||
if total > 0
|
||||
{
|
||||
count: total,
|
||||
accounts: account_counts.map { |ac| ac[:account].name },
|
||||
message: I18n.t("simplefin_items.stale_pending_status.message", count: total, days: days)
|
||||
}
|
||||
else
|
||||
{ count: 0 }
|
||||
end
|
||||
end
|
||||
|
||||
private
|
||||
# Parse sync_stats, handling cases where it might be a raw JSON string
|
||||
# (e.g., from console testing or bypassed serialization)
|
||||
def parse_sync_stats(sync_stats)
|
||||
return nil if sync_stats.blank?
|
||||
return sync_stats if sync_stats.is_a?(Hash)
|
||||
|
||||
if sync_stats.is_a?(String)
|
||||
JSON.parse(sync_stats) rescue nil
|
||||
end
|
||||
end
|
||||
|
||||
def remove_simplefin_item
|
||||
# SimpleFin doesn't require server-side cleanup like Plaid
|
||||
# The access URL just becomes inactive
|
||||
|
||||
@@ -9,6 +9,7 @@ class SimplefinItem::Importer
|
||||
@simplefin_provider = simplefin_provider
|
||||
@sync = sync
|
||||
@enqueued_holdings_job_ids = Set.new
|
||||
@reconciled_account_ids = Set.new # Debounce pending reconciliation per run
|
||||
end
|
||||
|
||||
def import
|
||||
@@ -201,8 +202,8 @@ class SimplefinItem::Importer
|
||||
end
|
||||
|
||||
adapter.update_balance(
|
||||
balance: normalized,
|
||||
cash_balance: cash,
|
||||
balance: account_data[:balance],
|
||||
cash_balance: account_data[:"available-balance"],
|
||||
source: "simplefin"
|
||||
)
|
||||
end
|
||||
@@ -428,9 +429,12 @@ class SimplefinItem::Importer
|
||||
perform_account_discovery
|
||||
|
||||
# Step 2: Fetch transactions/holdings using the regular window.
|
||||
# Note: Don't pass explicit `pending:` here - let fetch_accounts_data use the
|
||||
# SIMPLEFIN_INCLUDE_PENDING config. This allows users to disable pending transactions
|
||||
# if their bank's SimpleFIN integration produces duplicates when pending→posted.
|
||||
start_date = determine_sync_start_date
|
||||
Rails.logger.info "SimplefinItem::Importer - import_regular_sync: last_synced_at=#{simplefin_item.last_synced_at&.strftime('%Y-%m-%d %H:%M')} => start_date=#{start_date&.strftime('%Y-%m-%d')}"
|
||||
accounts_data = fetch_accounts_data(start_date: start_date, pending: true)
|
||||
accounts_data = fetch_accounts_data(start_date: start_date)
|
||||
return if accounts_data.nil? # Error already handled
|
||||
|
||||
# Store raw payload
|
||||
@@ -554,9 +558,15 @@ class SimplefinItem::Importer
|
||||
# Returns a Hash payload with keys like :accounts, or nil when an error is
|
||||
# handled internally via `handle_errors`.
|
||||
def fetch_accounts_data(start_date:, end_date: nil, pending: nil)
|
||||
# Determine whether to include pending based on explicit arg or global config.
|
||||
# `Rails.configuration.x.simplefin.include_pending` is ENV-backed.
|
||||
effective_pending = pending.nil? ? Rails.configuration.x.simplefin.include_pending : pending
|
||||
# Determine whether to include pending based on explicit arg, env var, or Setting.
|
||||
# Priority: explicit arg > env var > Setting (allows runtime changes via UI)
|
||||
effective_pending = if !pending.nil?
|
||||
pending
|
||||
elsif ENV["SIMPLEFIN_INCLUDE_PENDING"].present?
|
||||
Rails.configuration.x.simplefin.include_pending
|
||||
else
|
||||
Setting.syncs_include_pending
|
||||
end
|
||||
|
||||
# Debug logging to track exactly what's being sent to SimpleFin API
|
||||
start_str = start_date.respond_to?(:strftime) ? start_date.strftime("%Y-%m-%d") : "none"
|
||||
@@ -806,6 +816,15 @@ class SimplefinItem::Importer
|
||||
# Post-save side effects
|
||||
acct = simplefin_account.current_account
|
||||
if acct
|
||||
# Handle pending transaction reconciliation (debounced per run to avoid
|
||||
# repeated scans during chunked history imports)
|
||||
unless @reconciled_account_ids.include?(acct.id)
|
||||
@reconciled_account_ids << acct.id
|
||||
reconcile_and_track_pending_duplicates(acct)
|
||||
exclude_and_track_stale_pending(acct)
|
||||
track_stale_unmatched_pending(acct)
|
||||
end
|
||||
|
||||
# Refresh credit attributes when available-balance present
|
||||
if acct.accountable_type == "CreditCard" && account_data[:"available-balance"].present?
|
||||
begin
|
||||
@@ -1146,19 +1165,103 @@ class SimplefinItem::Importer
|
||||
ids.group_by(&:itself).select { |_, v| v.size > 1 }.keys
|
||||
end
|
||||
|
||||
# --- Simple helpers for numeric handling in normalization ---
|
||||
def to_decimal(value)
|
||||
return BigDecimal("0") if value.nil?
|
||||
case value
|
||||
when BigDecimal then value
|
||||
when String then BigDecimal(value) rescue BigDecimal("0")
|
||||
when Numeric then BigDecimal(value.to_s)
|
||||
else
|
||||
BigDecimal("0")
|
||||
# Reconcile pending transactions that have a matching posted version
|
||||
# Handles duplicates where pending and posted both exist (tip adjustments, etc.)
|
||||
def reconcile_and_track_pending_duplicates(account)
|
||||
reconcile_stats = Entry.reconcile_pending_duplicates(account: account, dry_run: false)
|
||||
|
||||
exact_matches = reconcile_stats[:details].select { |d| d[:match_type] == "exact" }
|
||||
fuzzy_suggestions = reconcile_stats[:details].select { |d| d[:match_type] == "fuzzy_suggestion" }
|
||||
|
||||
if exact_matches.any?
|
||||
stats["pending_reconciled"] = stats.fetch("pending_reconciled", 0) + exact_matches.size
|
||||
stats["pending_reconciled_details"] ||= []
|
||||
exact_matches.each do |detail|
|
||||
stats["pending_reconciled_details"] << {
|
||||
"account_name" => detail[:account],
|
||||
"pending_name" => detail[:pending_name],
|
||||
"posted_name" => detail[:posted_name]
|
||||
}
|
||||
end
|
||||
stats["pending_reconciled_details"] = stats["pending_reconciled_details"].last(50)
|
||||
end
|
||||
|
||||
if fuzzy_suggestions.any?
|
||||
stats["duplicate_suggestions_created"] = stats.fetch("duplicate_suggestions_created", 0) + fuzzy_suggestions.size
|
||||
stats["duplicate_suggestions_details"] ||= []
|
||||
fuzzy_suggestions.each do |detail|
|
||||
stats["duplicate_suggestions_details"] << {
|
||||
"account_name" => detail[:account],
|
||||
"pending_name" => detail[:pending_name],
|
||||
"posted_name" => detail[:posted_name]
|
||||
}
|
||||
end
|
||||
stats["duplicate_suggestions_details"] = stats["duplicate_suggestions_details"].last(50)
|
||||
end
|
||||
rescue => e
|
||||
Rails.logger.warn("SimpleFin: pending reconciliation failed for account #{account.id}: #{e.class} - #{e.message}")
|
||||
record_reconciliation_error("pending_reconciliation", account, e)
|
||||
end
|
||||
|
||||
def same_sign?(a, b)
|
||||
(a.positive? && b.positive?) || (a.negative? && b.negative?)
|
||||
# Auto-exclude stale pending transactions (>8 days old with no matching posted version)
|
||||
# Prevents orphaned pending transactions from affecting budgets indefinitely
|
||||
def exclude_and_track_stale_pending(account)
|
||||
excluded_count = Entry.auto_exclude_stale_pending(account: account)
|
||||
return unless excluded_count > 0
|
||||
|
||||
stats["stale_pending_excluded"] = stats.fetch("stale_pending_excluded", 0) + excluded_count
|
||||
stats["stale_pending_details"] ||= []
|
||||
stats["stale_pending_details"] << {
|
||||
"account_name" => account.name,
|
||||
"account_id" => account.id,
|
||||
"count" => excluded_count
|
||||
}
|
||||
stats["stale_pending_details"] = stats["stale_pending_details"].last(50)
|
||||
rescue => e
|
||||
Rails.logger.warn("SimpleFin: stale pending cleanup failed for account #{account.id}: #{e.class} - #{e.message}")
|
||||
record_reconciliation_error("stale_pending_cleanup", account, e)
|
||||
end
|
||||
|
||||
# Track stale pending transactions that couldn't be matched (for user awareness)
|
||||
# These are >8 days old, still pending, and have no duplicate suggestion
|
||||
def track_stale_unmatched_pending(account)
|
||||
stale_unmatched = account.entries
|
||||
.joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'")
|
||||
.where(excluded: false)
|
||||
.where("entries.date < ?", 8.days.ago.to_date)
|
||||
.where(<<~SQL.squish)
|
||||
(transactions.extra -> 'simplefin' ->> 'pending')::boolean = true
|
||||
OR (transactions.extra -> 'plaid' ->> 'pending')::boolean = true
|
||||
SQL
|
||||
.where(<<~SQL.squish)
|
||||
transactions.extra -> 'potential_posted_match' IS NULL
|
||||
SQL
|
||||
.count
|
||||
|
||||
return unless stale_unmatched > 0
|
||||
|
||||
stats["stale_unmatched_pending"] = stats.fetch("stale_unmatched_pending", 0) + stale_unmatched
|
||||
stats["stale_unmatched_details"] ||= []
|
||||
stats["stale_unmatched_details"] << {
|
||||
"account_name" => account.name,
|
||||
"account_id" => account.id,
|
||||
"count" => stale_unmatched
|
||||
}
|
||||
stats["stale_unmatched_details"] = stats["stale_unmatched_details"].last(50)
|
||||
rescue => e
|
||||
Rails.logger.warn("SimpleFin: stale unmatched tracking failed for account #{account.id}: #{e.class} - #{e.message}")
|
||||
record_reconciliation_error("stale_unmatched_tracking", account, e)
|
||||
end
|
||||
|
||||
# Record reconciliation errors to sync_stats for UI visibility
|
||||
def record_reconciliation_error(context, account, error)
|
||||
stats["reconciliation_errors"] ||= []
|
||||
stats["reconciliation_errors"] << {
|
||||
"context" => context,
|
||||
"account_id" => account.id,
|
||||
"account_name" => account.name,
|
||||
"error" => "#{error.class}: #{error.message}"
|
||||
}
|
||||
stats["reconciliation_errors"] = stats["reconciliation_errors"].last(20)
|
||||
end
|
||||
end
|
||||
|
||||
@@ -19,6 +19,22 @@ class Transaction < ApplicationRecord
|
||||
one_time: "one_time" # A one-time expense/income, excluded from budget analytics
|
||||
}
|
||||
|
||||
# Pending transaction scopes - filter based on provider pending flags in extra JSONB
|
||||
# Works with any provider that stores pending status in extra["provider_name"]["pending"]
|
||||
scope :pending, -> {
|
||||
where(<<~SQL.squish)
|
||||
(transactions.extra -> 'simplefin' ->> 'pending')::boolean = true
|
||||
OR (transactions.extra -> 'plaid' ->> 'pending')::boolean = true
|
||||
SQL
|
||||
}
|
||||
|
||||
scope :excluding_pending, -> {
|
||||
where(<<~SQL.squish)
|
||||
(transactions.extra -> 'simplefin' ->> 'pending')::boolean IS DISTINCT FROM true
|
||||
AND (transactions.extra -> 'plaid' ->> 'pending')::boolean IS DISTINCT FROM true
|
||||
SQL
|
||||
}
|
||||
|
||||
# Overarching grouping method for all transfer-type transactions
|
||||
def transfer?
|
||||
funds_movement? || cc_payment? || loan_payment?
|
||||
@@ -42,7 +58,85 @@ class Transaction < ApplicationRecord
|
||||
false
|
||||
end
|
||||
|
||||
# Potential duplicate matching methods
|
||||
# These help users review and resolve fuzzy-matched pending/posted pairs
|
||||
|
||||
def has_potential_duplicate?
|
||||
potential_posted_match_data.present? && !potential_duplicate_dismissed?
|
||||
end
|
||||
|
||||
def potential_duplicate_entry
|
||||
return nil unless has_potential_duplicate?
|
||||
Entry.find_by(id: potential_posted_match_data["entry_id"])
|
||||
end
|
||||
|
||||
def potential_duplicate_reason
|
||||
potential_posted_match_data&.dig("reason")
|
||||
end
|
||||
|
||||
def potential_duplicate_confidence
|
||||
potential_posted_match_data&.dig("confidence") || "medium"
|
||||
end
|
||||
|
||||
def low_confidence_duplicate?
|
||||
potential_duplicate_confidence == "low"
|
||||
end
|
||||
|
||||
def potential_duplicate_posted_amount
|
||||
potential_posted_match_data&.dig("posted_amount")&.to_d
|
||||
end
|
||||
|
||||
def potential_duplicate_dismissed?
|
||||
potential_posted_match_data&.dig("dismissed") == true
|
||||
end
|
||||
|
||||
# Merge this pending transaction with its suggested posted match
|
||||
# This DELETES the pending entry since the posted version is canonical
|
||||
def merge_with_duplicate!
|
||||
return false unless has_potential_duplicate?
|
||||
|
||||
posted_entry = potential_duplicate_entry
|
||||
return false unless posted_entry
|
||||
|
||||
pending_entry_id = entry.id
|
||||
pending_entry_name = entry.name
|
||||
|
||||
# Delete this pending entry completely (no need to keep it around)
|
||||
entry.destroy!
|
||||
|
||||
Rails.logger.info("User merged pending entry #{pending_entry_id} (#{pending_entry_name}) with posted entry #{posted_entry.id}")
|
||||
true
|
||||
end
|
||||
|
||||
# Dismiss the duplicate suggestion - user says these are NOT the same transaction
|
||||
def dismiss_duplicate_suggestion!
|
||||
return false unless potential_posted_match_data.present?
|
||||
|
||||
updated_extra = (extra || {}).deep_dup
|
||||
updated_extra["potential_posted_match"]["dismissed"] = true
|
||||
update!(extra: updated_extra)
|
||||
|
||||
Rails.logger.info("User dismissed duplicate suggestion for entry #{entry.id}")
|
||||
true
|
||||
end
|
||||
|
||||
# Clear the duplicate suggestion entirely
|
||||
def clear_duplicate_suggestion!
|
||||
return false unless potential_posted_match_data.present?
|
||||
|
||||
updated_extra = (extra || {}).deep_dup
|
||||
updated_extra.delete("potential_posted_match")
|
||||
update!(extra: updated_extra)
|
||||
true
|
||||
end
|
||||
|
||||
private
|
||||
|
||||
def potential_posted_match_data
|
||||
return nil unless extra.is_a?(Hash)
|
||||
extra["potential_posted_match"]
|
||||
end
|
||||
|
||||
def clear_merchant_unlinked_association
|
||||
return unless merchant_id.present? && merchant.is_a?(ProviderMerchant)
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@ class Transaction::Search
|
||||
attribute :amount, :string
|
||||
attribute :amount_operator, :string
|
||||
attribute :types, array: true
|
||||
attribute :status, array: true
|
||||
attribute :accounts, array: true
|
||||
attribute :account_ids, array: true
|
||||
attribute :start_date, :string
|
||||
@@ -30,6 +31,7 @@ class Transaction::Search
|
||||
query = apply_active_accounts_filter(query, active_accounts_only)
|
||||
query = apply_category_filter(query, categories)
|
||||
query = apply_type_filter(query, types)
|
||||
query = apply_status_filter(query, status)
|
||||
query = apply_merchant_filter(query, merchants)
|
||||
query = apply_tag_filter(query, tags)
|
||||
query = EntrySearch.apply_search_filter(query, search)
|
||||
@@ -153,4 +155,28 @@ class Transaction::Search
|
||||
return query unless tags.present?
|
||||
query.joins(:tags).where(tags: { name: tags })
|
||||
end
|
||||
|
||||
def apply_status_filter(query, statuses)
|
||||
return query unless statuses.present?
|
||||
return query if statuses.uniq.sort == [ "confirmed", "pending" ] # Both selected = no filter
|
||||
|
||||
pending_condition = <<~SQL.squish
|
||||
(transactions.extra -> 'simplefin' ->> 'pending')::boolean = true
|
||||
OR (transactions.extra -> 'plaid' ->> 'pending')::boolean = true
|
||||
SQL
|
||||
|
||||
confirmed_condition = <<~SQL.squish
|
||||
(transactions.extra -> 'simplefin' ->> 'pending')::boolean IS DISTINCT FROM true
|
||||
AND (transactions.extra -> 'plaid' ->> 'pending')::boolean IS DISTINCT FROM true
|
||||
SQL
|
||||
|
||||
case statuses.sort
|
||||
when [ "pending" ]
|
||||
query.where(pending_condition)
|
||||
when [ "confirmed" ]
|
||||
query.where(confirmed_condition)
|
||||
else
|
||||
query
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
30
app/views/settings/hostings/_sync_settings.html.erb
Normal file
30
app/views/settings/hostings/_sync_settings.html.erb
Normal file
@@ -0,0 +1,30 @@
|
||||
<% env_configured = ENV["SIMPLEFIN_INCLUDE_PENDING"].present? || ENV["PLAID_INCLUDE_PENDING"].present? %>
|
||||
<div class="space-y-4">
|
||||
<div class="flex items-center justify-between">
|
||||
<div class="space-y-1">
|
||||
<p class="text-sm"><%= t(".include_pending_label") %></p>
|
||||
<p class="text-secondary text-sm"><%= t(".include_pending_description") %></p>
|
||||
</div>
|
||||
|
||||
<%= styled_form_with model: Setting.new,
|
||||
url: settings_hosting_path,
|
||||
method: :patch,
|
||||
data: { controller: "auto-submit-form", auto_submit_form_trigger_event_value: "change" } do |form| %>
|
||||
<%= form.toggle :syncs_include_pending,
|
||||
checked: Setting.syncs_include_pending,
|
||||
disabled: env_configured,
|
||||
data: { auto_submit_form_target: "auto" } %>
|
||||
<% end %>
|
||||
</div>
|
||||
|
||||
<% if env_configured %>
|
||||
<div class="bg-warning-50 border border-warning-200 rounded-lg p-3">
|
||||
<div class="flex items-start gap-2">
|
||||
<%= icon("alert-circle", class: "w-5 h-5 text-warning-600 mt-0.5 shrink-0") %>
|
||||
<p class="text-sm text-warning-800">
|
||||
<%= t(".env_configured_message") %>
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
<% end %>
|
||||
</div>
|
||||
@@ -16,6 +16,9 @@
|
||||
<% end %>
|
||||
</div>
|
||||
<% end %>
|
||||
<%= settings_section title: t(".sync_settings") do %>
|
||||
<%= render "settings/hostings/sync_settings" %>
|
||||
<% end %>
|
||||
<%= settings_section title: t(".invites") do %>
|
||||
<%= render "settings/hostings/invite_code_settings" %>
|
||||
<% end %>
|
||||
|
||||
@@ -104,6 +104,25 @@
|
||||
<%= icon "alert-circle", size: "sm", color: "warning" %>
|
||||
<%= tag.span stale_status[:message], class: "text-sm" %>
|
||||
</div>
|
||||
<% elsif (pending_status = simplefin_item.stale_pending_status)[:count] > 0 %>
|
||||
<div class="text-secondary">
|
||||
<div class="flex items-center gap-1">
|
||||
<%= icon "clock", size: "sm", color: "secondary" %>
|
||||
<%= tag.span pending_status[:message], class: "text-sm" %>
|
||||
<span class="text-xs text-tertiary"><%= t(".stale_pending_note") %></span>
|
||||
</div>
|
||||
<% if pending_status[:accounts]&.any? %>
|
||||
<div class="text-xs text-tertiary ml-5">
|
||||
<%= t(".stale_pending_accounts", accounts: pending_status[:accounts].join(", ")) %>
|
||||
</div>
|
||||
<% end %>
|
||||
</div>
|
||||
<% elsif (reconciled_status = simplefin_item.last_sync_reconciled_status)[:count] > 0 %>
|
||||
<div class="text-success flex items-center gap-1">
|
||||
<%= icon "check-circle", size: "sm", color: "success" %>
|
||||
<%= tag.span reconciled_status[:message], class: "text-sm" %>
|
||||
<span class="text-xs text-tertiary"><%= t(".reconciled_details_note") %></span>
|
||||
</div>
|
||||
<% elsif simplefin_item.rate_limited_message.present? %>
|
||||
<div class="text-warning flex items-center gap-1">
|
||||
<%= icon "clock", size: "sm", color: "warning" %>
|
||||
@@ -117,7 +136,7 @@
|
||||
<% elsif duplicate_only_errors %>
|
||||
<div class="text-secondary flex items-center gap-1">
|
||||
<%= icon "info", size: "sm" %>
|
||||
<%= tag.span "Some accounts were skipped as duplicates — use ‘Link existing accounts’ to merge.", class: "text-secondary" %>
|
||||
<%= tag.span t(".duplicate_accounts_skipped"), class: "text-secondary" %>
|
||||
</div>
|
||||
<% else %>
|
||||
<p class="text-secondary">
|
||||
|
||||
@@ -80,12 +80,27 @@
|
||||
|
||||
<%# Pending indicator %>
|
||||
<% if transaction.pending? %>
|
||||
<span class="inline-flex items-center gap-1 text-xs font-medium rounded-full px-1.5 py-0.5 border border-secondary text-secondary" title="Pending — may change when posted">
|
||||
<span class="inline-flex items-center gap-1 text-xs font-medium rounded-full px-1.5 py-0.5 border border-secondary text-secondary" title="<%= t('transactions.transaction.pending_tooltip') %>">
|
||||
<%= icon "clock", size: "sm", color: "current" %>
|
||||
Pending
|
||||
<%= t("transactions.transaction.pending") %>
|
||||
</span>
|
||||
<% end %>
|
||||
|
||||
<%# Potential duplicate indicator - different styling for low vs medium confidence %>
|
||||
<% if transaction.has_potential_duplicate? %>
|
||||
<% if transaction.low_confidence_duplicate? %>
|
||||
<span class="inline-flex items-center gap-1 text-xs font-medium rounded-full px-1.5 py-0.5 border border-secondary bg-surface-inset text-secondary" title="<%= t('transactions.transaction.review_recommended_tooltip') %>">
|
||||
<%= icon "help-circle", size: "sm", color: "current" %>
|
||||
<%= t("transactions.transaction.review_recommended") %>
|
||||
</span>
|
||||
<% else %>
|
||||
<span class="inline-flex items-center gap-1 text-xs font-medium rounded-full px-1.5 py-0.5 border border-warning bg-warning/10 text-warning" title="<%= t('transactions.transaction.potential_duplicate_tooltip') %>">
|
||||
<%= icon "alert-triangle", size: "sm", color: "current" %>
|
||||
<%= t("transactions.transaction.possible_duplicate") %>
|
||||
</span>
|
||||
<% end %>
|
||||
<% end %>
|
||||
|
||||
<% if transaction.transfer.present? %>
|
||||
<%= render "transactions/transfer_match", transaction: transaction %>
|
||||
<% end %>
|
||||
|
||||
@@ -35,6 +35,11 @@
|
||||
end %>"></div>
|
||||
<p><%= t(".#{param_value.downcase}") %></p>
|
||||
</div>
|
||||
<% elsif param_key == "status" %>
|
||||
<div class="flex items-center gap-2 px-1">
|
||||
<%= icon(param_value.downcase == "pending" ? "clock" : "check", size: "sm") %>
|
||||
<p><%= t(".#{param_value.downcase}") %></p>
|
||||
</div>
|
||||
<% else %>
|
||||
<div class="flex items-center gap-2">
|
||||
<p><%= param_value %></p>
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
<%# locals: (form:) %>
|
||||
|
||||
<div class="p-2 space-y-3">
|
||||
<div class="flex items-center gap-3" data-filter-name="confirmed">
|
||||
<%= form.check_box :status,
|
||||
{
|
||||
multiple: true,
|
||||
checked: @q[:status]&.include?("confirmed"),
|
||||
class: "checkbox checkbox--light"
|
||||
},
|
||||
"confirmed",
|
||||
nil %>
|
||||
<%= form.label :status, t(".confirmed"), value: "confirmed", class: "text-sm text-primary" %>
|
||||
</div>
|
||||
<div class="flex items-center gap-3" data-filter-name="pending">
|
||||
<%= form.check_box :status,
|
||||
{
|
||||
multiple: true,
|
||||
checked: @q[:status]&.include?("pending"),
|
||||
class: "checkbox checkbox--light"
|
||||
},
|
||||
"pending",
|
||||
nil %>
|
||||
<%= form.label :status, t(".pending"), value: "pending", class: "text-sm text-primary" %>
|
||||
</div>
|
||||
</div>
|
||||
@@ -4,6 +4,47 @@
|
||||
<% end %>
|
||||
|
||||
<% dialog.with_body do %>
|
||||
<%# Potential duplicate alert %>
|
||||
<% if @entry.transaction.has_potential_duplicate? %>
|
||||
<% potential_match = @entry.transaction.potential_duplicate_entry %>
|
||||
<% if potential_match %>
|
||||
<div class="mx-4 my-3 p-4 rounded-lg border border-warning bg-warning/5">
|
||||
<div class="flex items-start gap-3">
|
||||
<%= icon "alert-triangle", size: "md", color: "warning" %>
|
||||
<div class="flex-1 space-y-2">
|
||||
<h4 class="text-sm font-medium text-primary"><%= t("transactions.show.potential_duplicate_title") %></h4>
|
||||
<p class="text-sm text-secondary"><%= t("transactions.show.potential_duplicate_description") %></p>
|
||||
|
||||
<div class="mt-3 p-3 rounded bg-container border border-primary">
|
||||
<div class="flex items-center justify-between">
|
||||
<div>
|
||||
<p class="text-sm font-medium text-primary"><%= potential_match.name %></p>
|
||||
<p class="text-xs text-secondary"><%= potential_match.date.strftime("%b %d, %Y") %> • <%= potential_match.account.name %></p>
|
||||
</div>
|
||||
<p class="text-sm font-medium <%= potential_match.amount.negative? ? 'text-green-600' : 'text-primary' %>">
|
||||
<%= format_money(-potential_match.amount_money) %>
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="flex items-center gap-2 mt-3">
|
||||
<%= button_to t("transactions.show.merge_duplicate"),
|
||||
merge_duplicate_transaction_path(@entry.transaction),
|
||||
method: :post,
|
||||
class: "btn btn--primary btn--sm",
|
||||
data: { turbo_frame: "_top" } %>
|
||||
<%= button_to t("transactions.show.keep_both"),
|
||||
dismiss_duplicate_transaction_path(@entry.transaction),
|
||||
method: :post,
|
||||
class: "btn btn--outline btn--sm",
|
||||
data: { turbo_frame: "_top" } %>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<% end %>
|
||||
<% end %>
|
||||
|
||||
<% dialog.with_section(title: t(".overview"), open: true) do %>
|
||||
<div class="pb-4">
|
||||
<%= styled_form_with model: @entry,
|
||||
|
||||
@@ -4,4 +4,13 @@
|
||||
Rails.application.configure do
|
||||
config.plaid = nil
|
||||
config.plaid_eu = nil
|
||||
|
||||
# Plaid pending transaction settings (mirrors SimpleFIN config pattern)
|
||||
falsy = %w[0 false no off]
|
||||
config.x.plaid ||= ActiveSupport::OrderedOptions.new
|
||||
# Default to true - fetch pending transactions for display with "Pending" badge
|
||||
# and reconciliation when posted versions arrive (Plaid provides pending_transaction_id for reliable linking)
|
||||
# Set PLAID_INCLUDE_PENDING=0 to disable if user prefers not to see pending transactions
|
||||
pending_env = ENV["PLAID_INCLUDE_PENDING"].to_s.strip.downcase
|
||||
config.x.plaid.include_pending = pending_env.blank? ? true : !falsy.include?(pending_env)
|
||||
end
|
||||
|
||||
@@ -1,7 +1,15 @@
|
||||
Rails.application.configure do
|
||||
truthy = %w[1 true yes on]
|
||||
falsy = %w[0 false no off]
|
||||
|
||||
config.x.simplefin ||= ActiveSupport::OrderedOptions.new
|
||||
config.x.simplefin.include_pending = truthy.include?(ENV["SIMPLEFIN_INCLUDE_PENDING"].to_s.strip.downcase)
|
||||
# Default to true - always fetch pending transactions so they can be:
|
||||
# - Displayed with a "Pending" badge
|
||||
# - Excluded from budgets (but included in net worth)
|
||||
# - Reconciled when posted versions arrive (avoiding duplicates)
|
||||
# - Auto-excluded after 8 days if they remain stale
|
||||
# Set SIMPLEFIN_INCLUDE_PENDING=0 to disable if a bank's integration causes issues
|
||||
pending_env = ENV["SIMPLEFIN_INCLUDE_PENDING"].to_s.strip.downcase
|
||||
config.x.simplefin.include_pending = pending_env.blank? ? true : !falsy.include?(pending_env)
|
||||
config.x.simplefin.debug_raw = truthy.include?(ENV["SIMPLEFIN_DEBUG_RAW"].to_s.strip.downcase)
|
||||
end
|
||||
|
||||
@@ -50,15 +50,19 @@ en:
|
||||
activity:
|
||||
amount: Amount
|
||||
balance: Balance
|
||||
confirmed: Confirmed
|
||||
date: Date
|
||||
entries: entries
|
||||
entry: entry
|
||||
filter: Filter
|
||||
new: New
|
||||
new_balance: New balance
|
||||
new_transaction: New transaction
|
||||
no_entries: No entries found
|
||||
pending: Pending
|
||||
search:
|
||||
placeholder: Search entries by name
|
||||
status: Status
|
||||
title: Activity
|
||||
chart:
|
||||
balance: Balance
|
||||
|
||||
@@ -24,6 +24,28 @@ en:
|
||||
rate_limited: "Rate limited %{time_ago}"
|
||||
recently: recently
|
||||
errors: "Errors: %{count}"
|
||||
pending_reconciled:
|
||||
one: "%{count} duplicate pending transaction reconciled"
|
||||
other: "%{count} duplicate pending transactions reconciled"
|
||||
view_reconciled: View reconciled transactions
|
||||
duplicate_suggestions:
|
||||
one: "%{count} possible duplicate needs review"
|
||||
other: "%{count} possible duplicates need review"
|
||||
view_duplicate_suggestions: View suggested duplicates
|
||||
stale_pending:
|
||||
one: "%{count} stale pending transaction (excluded from budgets)"
|
||||
other: "%{count} stale pending transactions (excluded from budgets)"
|
||||
view_stale_pending: View affected accounts
|
||||
stale_pending_count:
|
||||
one: "%{count} transaction"
|
||||
other: "%{count} transactions"
|
||||
stale_unmatched:
|
||||
one: "%{count} pending transaction needs manual review"
|
||||
other: "%{count} pending transactions need manual review"
|
||||
view_stale_unmatched: View transactions needing review
|
||||
stale_unmatched_count:
|
||||
one: "%{count} transaction"
|
||||
other: "%{count} transactions"
|
||||
data_warnings: "Data warnings: %{count}"
|
||||
notices: "Notices: %{count}"
|
||||
view_data_quality: View data quality details
|
||||
|
||||
@@ -17,6 +17,7 @@ en:
|
||||
show:
|
||||
general: General Settings
|
||||
financial_data_providers: Financial Data Providers
|
||||
sync_settings: Sync Settings
|
||||
invites: Invite Codes
|
||||
title: Self-Hosting
|
||||
danger_zone: Danger Zone
|
||||
@@ -77,3 +78,7 @@ en:
|
||||
clear_cache:
|
||||
cache_cleared: Data cache has been cleared. This may take a few moments to complete.
|
||||
not_authorized: You are not authorized to perform this action
|
||||
sync_settings:
|
||||
include_pending_label: Include pending transactions
|
||||
include_pending_description: When enabled, pending (uncleared) transactions will be imported and automatically reconciled when they post. Disable if your bank provides unreliable pending data.
|
||||
env_configured_message: This setting is disabled because a provider environment variable (SIMPLEFIN_INCLUDE_PENDING or PLAID_INCLUDE_PENDING) is set. Remove it to enable this setting.
|
||||
|
||||
@@ -70,6 +70,10 @@ en:
|
||||
status_with_summary: "Last synced %{timestamp} ago • %{summary}"
|
||||
syncing: Syncing...
|
||||
update: Update
|
||||
stale_pending_note: "(excluded from budgets)"
|
||||
stale_pending_accounts: "in: %{accounts}"
|
||||
reconciled_details_note: "(see sync summary for details)"
|
||||
duplicate_accounts_skipped: "Some accounts were skipped as duplicates — use 'Link existing accounts' to merge."
|
||||
select_existing_account:
|
||||
title: "Link %{account_name} to SimpleFIN"
|
||||
description: Select a SimpleFIN account to link to your existing account
|
||||
@@ -86,3 +90,11 @@ en:
|
||||
errors:
|
||||
only_manual: Only manual accounts can be linked
|
||||
invalid_simplefin_account: Invalid SimpleFIN account selected
|
||||
reconciled_status:
|
||||
message:
|
||||
one: "%{count} duplicate pending transaction reconciled"
|
||||
other: "%{count} duplicate pending transactions reconciled"
|
||||
stale_pending_status:
|
||||
message:
|
||||
one: "%{count} pending transaction older than %{days} days"
|
||||
other: "%{count} pending transactions older than %{days} days"
|
||||
|
||||
@@ -44,6 +44,23 @@ en:
|
||||
settings: Settings
|
||||
tags_label: Tags
|
||||
uncategorized: "(uncategorized)"
|
||||
potential_duplicate_title: Possible duplicate detected
|
||||
potential_duplicate_description: This pending transaction may be the same as the posted transaction below. If so, merge them to avoid double-counting.
|
||||
merge_duplicate: Yes, merge them
|
||||
keep_both: No, keep both
|
||||
transaction:
|
||||
pending: Pending
|
||||
pending_tooltip: Pending transaction — may change when posted
|
||||
possible_duplicate: Duplicate?
|
||||
potential_duplicate_tooltip: This may be a duplicate of another transaction
|
||||
review_recommended: Review
|
||||
review_recommended_tooltip: Large amount difference — review recommended to check if this is a duplicate
|
||||
merge_duplicate:
|
||||
success: Transactions merged successfully
|
||||
failure: Could not merge transactions
|
||||
dismiss_duplicate:
|
||||
success: Kept as separate transactions
|
||||
failure: Could not dismiss duplicate suggestion
|
||||
header:
|
||||
edit_categories: Edit categories
|
||||
edit_imports: Edit imports
|
||||
@@ -55,6 +72,16 @@ en:
|
||||
transactions: transactions
|
||||
import: Import
|
||||
toggle_recurring_section: Toggle upcoming recurring transactions
|
||||
search:
|
||||
filters:
|
||||
account: Account
|
||||
date: Date
|
||||
type: Type
|
||||
status: Status
|
||||
amount: Amount
|
||||
category: Category
|
||||
tag: Tag
|
||||
merchant: Merchant
|
||||
searches:
|
||||
filters:
|
||||
amount_filter:
|
||||
@@ -68,10 +95,15 @@ en:
|
||||
on_or_after: on or after %{date}
|
||||
on_or_before: on or before %{date}
|
||||
transfer: Transfer
|
||||
confirmed: Confirmed
|
||||
pending: Pending
|
||||
type_filter:
|
||||
expense: Expense
|
||||
income: Income
|
||||
transfer: Transfer
|
||||
status_filter:
|
||||
confirmed: Confirmed
|
||||
pending: Pending
|
||||
menu:
|
||||
account_filter: Account
|
||||
amount_filter: Amount
|
||||
@@ -81,6 +113,7 @@ en:
|
||||
clear_filters: Clear filters
|
||||
date_filter: Date
|
||||
merchant_filter: Merchant
|
||||
status_filter: Status
|
||||
tag_filter: Tag
|
||||
type_filter: Type
|
||||
search:
|
||||
|
||||
@@ -191,6 +191,8 @@ Rails.application.routes.draw do
|
||||
|
||||
member do
|
||||
post :mark_as_recurring
|
||||
post :merge_duplicate
|
||||
post :dismiss_duplicate
|
||||
end
|
||||
end
|
||||
|
||||
|
||||
158
lib/tasks/simplefin_pending_cleanup.rake
Normal file
158
lib/tasks/simplefin_pending_cleanup.rake
Normal file
@@ -0,0 +1,158 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
namespace :simplefin do
|
||||
desc "Find and optionally remove duplicate pending transactions that have matching posted versions"
|
||||
task pending_cleanup: :environment do
|
||||
dry_run = ENV["DRY_RUN"] != "false"
|
||||
date_window = (ENV["DATE_WINDOW"] || 8).to_i
|
||||
|
||||
puts "SimpleFIN Pending Transaction Cleanup"
|
||||
puts "======================================"
|
||||
puts "Mode: #{dry_run ? 'DRY RUN (no changes)' : 'LIVE (will delete duplicates)'}"
|
||||
puts "Date window: #{date_window} days"
|
||||
puts ""
|
||||
|
||||
# Find all pending SimpleFIN transactions
|
||||
pending_entries = Entry.joins(
|
||||
"INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'"
|
||||
).where(source: "simplefin")
|
||||
.where("transactions.extra -> 'simplefin' ->> 'pending' = ?", "true")
|
||||
.includes(:account)
|
||||
|
||||
puts "Found #{pending_entries.count} pending SimpleFIN transactions"
|
||||
puts ""
|
||||
|
||||
duplicates_found = 0
|
||||
duplicates_to_delete = []
|
||||
|
||||
pending_entries.find_each do |pending_entry|
|
||||
# Look for a matching posted transaction
|
||||
posted_match = Entry.joins(
|
||||
"INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'"
|
||||
).where(account_id: pending_entry.account_id)
|
||||
.where(source: "simplefin")
|
||||
.where(amount: pending_entry.amount)
|
||||
.where(currency: pending_entry.currency)
|
||||
.where(date: pending_entry.date..(pending_entry.date + date_window.days)) # Posted must be ON or AFTER pending
|
||||
.where.not(id: pending_entry.id)
|
||||
.where("transactions.extra -> 'simplefin' ->> 'pending' != ? OR transactions.extra -> 'simplefin' ->> 'pending' IS NULL", "true")
|
||||
.first
|
||||
|
||||
if posted_match
|
||||
duplicates_found += 1
|
||||
duplicates_to_delete << pending_entry
|
||||
|
||||
puts "DUPLICATE FOUND:"
|
||||
puts " Pending: ID=#{pending_entry.id} | #{pending_entry.date} | #{pending_entry.name} | #{pending_entry.amount} #{pending_entry.currency}"
|
||||
puts " Posted: ID=#{posted_match.id} | #{posted_match.date} | #{posted_match.name} | #{posted_match.amount} #{posted_match.currency}"
|
||||
puts " Account: #{pending_entry.account.name}"
|
||||
puts ""
|
||||
end
|
||||
end
|
||||
|
||||
puts "======================================"
|
||||
puts "Summary: #{duplicates_found} duplicate pending transactions found"
|
||||
puts ""
|
||||
|
||||
if duplicates_found > 0
|
||||
if dry_run
|
||||
puts "To delete these duplicates, run:"
|
||||
puts " rails simplefin:pending_cleanup DRY_RUN=false"
|
||||
puts ""
|
||||
puts "To adjust the date matching window (default 8 days):"
|
||||
puts " rails simplefin:pending_cleanup DATE_WINDOW=14"
|
||||
else
|
||||
print "Deleting #{duplicates_to_delete.count} duplicate pending entries... "
|
||||
Entry.where(id: duplicates_to_delete.map(&:id)).destroy_all
|
||||
puts "Done!"
|
||||
end
|
||||
else
|
||||
puts "No duplicates found. Nothing to clean up."
|
||||
end
|
||||
end
|
||||
|
||||
desc "Un-exclude pending transactions that were wrongly matched (fixes direction bug)"
|
||||
task pending_restore: :environment do
|
||||
dry_run = ENV["DRY_RUN"] != "false"
|
||||
date_window = (ENV["DATE_WINDOW"] || 8).to_i
|
||||
|
||||
puts "Restore Wrongly Excluded Pending Transactions"
|
||||
puts "=============================================="
|
||||
puts "Mode: #{dry_run ? 'DRY RUN (no changes)' : 'LIVE (will restore)'}"
|
||||
puts "Date window: #{date_window} days (forward only)"
|
||||
puts ""
|
||||
|
||||
# Find all EXCLUDED pending transactions (these may have been wrongly excluded)
|
||||
excluded_pending = Entry.joins(
|
||||
"INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'"
|
||||
).where(excluded: true)
|
||||
.where(<<~SQL.squish)
|
||||
(transactions.extra -> 'simplefin' ->> 'pending')::boolean = true
|
||||
OR (transactions.extra -> 'plaid' ->> 'pending')::boolean = true
|
||||
SQL
|
||||
|
||||
puts "Found #{excluded_pending.count} excluded pending transactions to evaluate"
|
||||
puts ""
|
||||
|
||||
to_restore = []
|
||||
|
||||
excluded_pending.includes(:account).find_each do |pending_entry|
|
||||
# Check if there's a VALID posted match using CORRECT logic (forward-only dates)
|
||||
# Posted date must be ON or AFTER pending date
|
||||
valid_match = pending_entry.account.entries
|
||||
.joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'")
|
||||
.where.not(id: pending_entry.id)
|
||||
.where(currency: pending_entry.currency)
|
||||
.where(amount: pending_entry.amount)
|
||||
.where(date: pending_entry.date..(pending_entry.date + date_window.days))
|
||||
.where(<<~SQL.squish)
|
||||
(transactions.extra -> 'simplefin' ->> 'pending')::boolean IS NOT TRUE
|
||||
AND (transactions.extra -> 'plaid' ->> 'pending')::boolean IS NOT TRUE
|
||||
SQL
|
||||
.exists?
|
||||
|
||||
unless valid_match
|
||||
to_restore << pending_entry
|
||||
puts "SHOULD RESTORE (no valid match):"
|
||||
puts " ID=#{pending_entry.id} | #{pending_entry.date} | #{pending_entry.name} | #{pending_entry.amount} #{pending_entry.currency}"
|
||||
puts " Account: #{pending_entry.account.name}"
|
||||
puts ""
|
||||
end
|
||||
end
|
||||
|
||||
puts "=============================================="
|
||||
puts "Summary: #{to_restore.count} transactions should be restored"
|
||||
puts ""
|
||||
|
||||
if to_restore.any?
|
||||
if dry_run
|
||||
puts "To restore these transactions, run:"
|
||||
puts " rails simplefin:pending_restore DRY_RUN=false"
|
||||
else
|
||||
Entry.where(id: to_restore.map(&:id)).update_all(excluded: false)
|
||||
puts "Restored #{to_restore.count} transactions!"
|
||||
end
|
||||
else
|
||||
puts "No wrongly excluded transactions found."
|
||||
end
|
||||
end
|
||||
|
||||
desc "List all pending SimpleFIN transactions (for review)"
|
||||
task pending_list: :environment do
|
||||
pending_entries = Entry.joins(
|
||||
"INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'"
|
||||
).where(source: "simplefin")
|
||||
.where("transactions.extra -> 'simplefin' ->> 'pending' = ?", "true")
|
||||
.includes(:account)
|
||||
.order(date: :desc)
|
||||
|
||||
puts "All Pending SimpleFIN Transactions"
|
||||
puts "==================================="
|
||||
puts "Total: #{pending_entries.count}"
|
||||
puts ""
|
||||
|
||||
pending_entries.find_each do |entry|
|
||||
puts "ID=#{entry.id} | #{entry.date} | #{entry.name.truncate(40)} | #{entry.amount} #{entry.currency} | Account: #{entry.account.name}"
|
||||
end
|
||||
end
|
||||
end
|
||||
@@ -785,4 +785,482 @@ class Account::ProviderImportAdapterTest < ActiveSupport::TestCase
|
||||
assert_nil newer_entry.reload.external_id
|
||||
end
|
||||
end
|
||||
|
||||
# ============================================================================
|
||||
# Pending→Posted Transaction Reconciliation Tests
|
||||
# ============================================================================
|
||||
|
||||
test "reconciles pending transaction when posted version arrives with different external_id" do
|
||||
# Simulate SimpleFIN giving different IDs for pending vs posted transactions
|
||||
# First, import a pending transaction
|
||||
pending_entry = @adapter.import_transaction(
|
||||
external_id: "simplefin_pending_abc",
|
||||
amount: 99.99,
|
||||
currency: "USD",
|
||||
date: Date.today - 2.days,
|
||||
name: "Coffee Shop",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => true } }
|
||||
)
|
||||
|
||||
assert pending_entry.transaction.pending?, "Entry should be marked pending"
|
||||
original_id = pending_entry.id
|
||||
|
||||
# Now import the posted version with a DIFFERENT external_id
|
||||
# This should claim the pending entry, not create a duplicate
|
||||
assert_no_difference "@account.entries.count" do
|
||||
posted_entry = @adapter.import_transaction(
|
||||
external_id: "simplefin_posted_xyz",
|
||||
amount: 99.99,
|
||||
currency: "USD",
|
||||
date: Date.today,
|
||||
name: "Coffee Shop - Posted",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => false } }
|
||||
)
|
||||
|
||||
# Should be the same entry, now with updated external_id
|
||||
assert_equal original_id, posted_entry.id
|
||||
assert_equal "simplefin_posted_xyz", posted_entry.external_id
|
||||
assert_not posted_entry.transaction.pending?, "Entry should no longer be pending"
|
||||
end
|
||||
end
|
||||
|
||||
test "does not reconcile when posted transaction has same external_id as pending" do
|
||||
# When external_id matches, normal dedup should handle it
|
||||
pending_entry = @adapter.import_transaction(
|
||||
external_id: "simplefin_same_id",
|
||||
amount: 50.00,
|
||||
currency: "USD",
|
||||
date: Date.today - 1.day,
|
||||
name: "Gas Station",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => true } }
|
||||
)
|
||||
|
||||
# Import posted version with SAME external_id
|
||||
assert_no_difference "@account.entries.count" do
|
||||
posted_entry = @adapter.import_transaction(
|
||||
external_id: "simplefin_same_id",
|
||||
amount: 50.00,
|
||||
currency: "USD",
|
||||
date: Date.today,
|
||||
name: "Gas Station - Posted",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => false } }
|
||||
)
|
||||
|
||||
assert_equal pending_entry.id, posted_entry.id
|
||||
assert_not posted_entry.transaction.pending?
|
||||
end
|
||||
end
|
||||
|
||||
test "fuzzy amount match creates suggestion instead of auto-claiming" do
|
||||
# Import pending transaction (pre-tip authorization)
|
||||
pending_entry = @adapter.import_transaction(
|
||||
external_id: "simplefin_pending_amount_test",
|
||||
amount: 100.00,
|
||||
currency: "USD",
|
||||
date: Date.today - 1.day,
|
||||
name: "Restaurant",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => true } }
|
||||
)
|
||||
|
||||
# Import posted with tip added - should NOT auto-claim, but should store suggestion
|
||||
# Fuzzy matches now create suggestions for user review instead of auto-reconciling
|
||||
assert_difference "@account.entries.count", 1 do
|
||||
posted_entry = @adapter.import_transaction(
|
||||
external_id: "simplefin_posted_amount_test",
|
||||
amount: 105.00, # 5% tip added - within 25% tolerance
|
||||
currency: "USD",
|
||||
date: Date.today,
|
||||
name: "Restaurant",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => false } }
|
||||
)
|
||||
|
||||
# Should be a NEW entry (not claimed)
|
||||
assert_not_equal pending_entry.id, posted_entry.id
|
||||
assert_equal "simplefin_posted_amount_test", posted_entry.external_id
|
||||
|
||||
# The PENDING entry should now have a potential_posted_match suggestion
|
||||
pending_entry.reload
|
||||
assert pending_entry.transaction.has_potential_duplicate?
|
||||
assert_equal posted_entry.id, pending_entry.transaction.potential_duplicate_entry.id
|
||||
end
|
||||
end
|
||||
|
||||
test "does not reconcile pending when amount difference exceeds tolerance" do
|
||||
# Import pending transaction
|
||||
pending_entry = @adapter.import_transaction(
|
||||
external_id: "simplefin_pending_big_diff",
|
||||
amount: 50.00,
|
||||
currency: "USD",
|
||||
date: Date.today - 1.day,
|
||||
name: "Store",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => true } }
|
||||
)
|
||||
|
||||
# Import posted with amount >25% different - should NOT match
|
||||
# $100 posted / 1.25 = $80 minimum pending, but pending is only $50
|
||||
assert_difference "@account.entries.count", 1 do
|
||||
posted_entry = @adapter.import_transaction(
|
||||
external_id: "simplefin_posted_big_diff",
|
||||
amount: 100.00, # 100% increase - way outside 25% tolerance
|
||||
currency: "USD",
|
||||
date: Date.today,
|
||||
name: "Store",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => false } }
|
||||
)
|
||||
|
||||
assert_not_equal pending_entry.id, posted_entry.id
|
||||
end
|
||||
end
|
||||
|
||||
test "does not reconcile pending when date is outside window" do
|
||||
# Import pending transaction
|
||||
pending_entry = @adapter.import_transaction(
|
||||
external_id: "simplefin_pending_date_test",
|
||||
amount: 25.00,
|
||||
currency: "USD",
|
||||
date: Date.today - 15.days, # 15 days ago
|
||||
name: "Subscription",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => true } }
|
||||
)
|
||||
|
||||
# Import posted with date outside 7-day window - should NOT match
|
||||
assert_difference "@account.entries.count", 1 do
|
||||
posted_entry = @adapter.import_transaction(
|
||||
external_id: "simplefin_posted_date_test",
|
||||
amount: 25.00,
|
||||
currency: "USD",
|
||||
date: Date.today,
|
||||
name: "Subscription",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => false } }
|
||||
)
|
||||
|
||||
assert_not_equal pending_entry.id, posted_entry.id
|
||||
end
|
||||
end
|
||||
|
||||
test "reconciles pending within 7 day window" do
|
||||
# Import pending transaction
|
||||
pending_entry = @adapter.import_transaction(
|
||||
external_id: "simplefin_pending_window_test",
|
||||
amount: 75.00,
|
||||
currency: "USD",
|
||||
date: Date.today - 5.days,
|
||||
name: "Online Order",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => true } }
|
||||
)
|
||||
|
||||
# Import posted within 7-day window - should match
|
||||
assert_no_difference "@account.entries.count" do
|
||||
posted_entry = @adapter.import_transaction(
|
||||
external_id: "simplefin_posted_window_test",
|
||||
amount: 75.00,
|
||||
currency: "USD",
|
||||
date: Date.today,
|
||||
name: "Online Order - Posted",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => false } }
|
||||
)
|
||||
|
||||
assert_equal pending_entry.id, posted_entry.id
|
||||
end
|
||||
end
|
||||
|
||||
test "does not reconcile pending from different source" do
|
||||
# Import pending from SimpleFIN
|
||||
pending_entry = @adapter.import_transaction(
|
||||
external_id: "simplefin_pending_source_test",
|
||||
amount: 30.00,
|
||||
currency: "USD",
|
||||
date: Date.today - 1.day,
|
||||
name: "Pharmacy",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => true } }
|
||||
)
|
||||
|
||||
# Import from different source (plaid) - should NOT match SimpleFIN pending
|
||||
assert_difference "@account.entries.count", 1 do
|
||||
plaid_entry = @adapter.import_transaction(
|
||||
external_id: "plaid_posted_source_test",
|
||||
amount: 30.00,
|
||||
currency: "USD",
|
||||
date: Date.today,
|
||||
name: "Pharmacy",
|
||||
source: "plaid",
|
||||
extra: { "plaid" => { "pending" => false } }
|
||||
)
|
||||
|
||||
assert_not_equal pending_entry.id, plaid_entry.id
|
||||
end
|
||||
end
|
||||
|
||||
test "does not reconcile when incoming transaction is also pending" do
|
||||
# Import first pending transaction
|
||||
pending_entry1 = @adapter.import_transaction(
|
||||
external_id: "simplefin_pending_1",
|
||||
amount: 45.00,
|
||||
currency: "USD",
|
||||
date: Date.today - 1.day,
|
||||
name: "Store",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => true } }
|
||||
)
|
||||
|
||||
# Import another pending transaction with different ID - should NOT match
|
||||
assert_difference "@account.entries.count", 1 do
|
||||
pending_entry2 = @adapter.import_transaction(
|
||||
external_id: "simplefin_pending_2",
|
||||
amount: 45.00,
|
||||
currency: "USD",
|
||||
date: Date.today,
|
||||
name: "Store",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => true } }
|
||||
)
|
||||
|
||||
assert_not_equal pending_entry1.id, pending_entry2.id
|
||||
end
|
||||
end
|
||||
|
||||
test "reconciles most recent pending when multiple exist" do
|
||||
# Create two pending transactions with same amount
|
||||
older_pending = @adapter.import_transaction(
|
||||
external_id: "simplefin_older_pending",
|
||||
amount: 60.00,
|
||||
currency: "USD",
|
||||
date: Date.today - 5.days,
|
||||
name: "Recurring Payment - Old",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => true } }
|
||||
)
|
||||
|
||||
newer_pending = @adapter.import_transaction(
|
||||
external_id: "simplefin_newer_pending",
|
||||
amount: 60.00,
|
||||
currency: "USD",
|
||||
date: Date.today - 1.day,
|
||||
name: "Recurring Payment - New",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => true } }
|
||||
)
|
||||
|
||||
# Import posted - should match the most recent pending (by date)
|
||||
assert_no_difference "@account.entries.count" do
|
||||
posted_entry = @adapter.import_transaction(
|
||||
external_id: "simplefin_posted_recurring",
|
||||
amount: 60.00,
|
||||
currency: "USD",
|
||||
date: Date.today,
|
||||
name: "Recurring Payment - Posted",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => false } }
|
||||
)
|
||||
|
||||
# Should match the newer pending entry
|
||||
assert_equal newer_pending.id, posted_entry.id
|
||||
# Older pending should remain untouched
|
||||
assert_equal "simplefin_older_pending", older_pending.reload.external_id
|
||||
end
|
||||
end
|
||||
|
||||
test "find_pending_transaction returns nil when no pending transactions exist" do
|
||||
# Create a non-pending transaction
|
||||
@adapter.import_transaction(
|
||||
external_id: "simplefin_not_pending",
|
||||
amount: 40.00,
|
||||
currency: "USD",
|
||||
date: Date.today - 1.day,
|
||||
name: "Regular Transaction",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => false } }
|
||||
)
|
||||
|
||||
result = @adapter.find_pending_transaction(
|
||||
date: Date.today,
|
||||
amount: 40.00,
|
||||
currency: "USD",
|
||||
source: "simplefin"
|
||||
)
|
||||
|
||||
assert_nil result
|
||||
end
|
||||
|
||||
# ============================================================================
|
||||
# Critical Direction Fix Tests (CITGO Bug Prevention)
|
||||
# ============================================================================
|
||||
|
||||
test "does not match pending transaction that is AFTER the posted date (direction fix)" do
|
||||
# This is the CITGO bug scenario:
|
||||
# - Posted transaction on Dec 31
|
||||
# - Pending transaction on Jan 8 (AFTER the posted)
|
||||
# - These should NOT match because pending MUST come BEFORE posted
|
||||
|
||||
# First, import a POSTED transaction on an earlier date
|
||||
posted_entry = @adapter.import_transaction(
|
||||
external_id: "simplefin_posted_dec31",
|
||||
amount: 6.67,
|
||||
currency: "USD",
|
||||
date: Date.today - 8.days, # Dec 31 (earlier)
|
||||
name: "CITGO Gas Station",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => false } }
|
||||
)
|
||||
|
||||
# Now import a PENDING transaction on a LATER date
|
||||
# This should NOT be matched because the date direction is wrong
|
||||
assert_difference "@account.entries.count", 1 do
|
||||
pending_entry = @adapter.import_transaction(
|
||||
external_id: "simplefin_pending_jan8",
|
||||
amount: 6.65, # Similar but different amount
|
||||
currency: "USD",
|
||||
date: Date.today, # Jan 8 (later)
|
||||
name: "CITGO Gas Station",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => true } }
|
||||
)
|
||||
|
||||
# Should be a DIFFERENT entry - not matched to the earlier posted one
|
||||
assert_not_equal posted_entry.id, pending_entry.id
|
||||
assert pending_entry.transaction.pending?
|
||||
end
|
||||
end
|
||||
|
||||
test "find_pending_transaction only searches backward in time" do
|
||||
# Create a pending transaction in the FUTURE (after the posted date we'll search from)
|
||||
# This should NOT be found because pending must be ON or BEFORE posted
|
||||
future_pending = @adapter.import_transaction(
|
||||
external_id: "simplefin_future_pending",
|
||||
amount: 50.00,
|
||||
currency: "USD",
|
||||
date: Date.today + 3.days, # Future date
|
||||
name: "Future Transaction",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => true } }
|
||||
)
|
||||
|
||||
# Search from today - should NOT find the future pending
|
||||
result = @adapter.find_pending_transaction(
|
||||
date: Date.today,
|
||||
amount: 50.00,
|
||||
currency: "USD",
|
||||
source: "simplefin"
|
||||
)
|
||||
|
||||
assert_nil result, "Should not find pending transactions that are in the future relative to the posted date"
|
||||
end
|
||||
|
||||
test "find_pending_transaction finds pending transaction that is before posted date" do
|
||||
# Create a pending transaction in the PAST (before the posted date)
|
||||
# This SHOULD be found
|
||||
past_pending = @adapter.import_transaction(
|
||||
external_id: "simplefin_past_pending",
|
||||
amount: 75.00,
|
||||
currency: "USD",
|
||||
date: Date.today - 3.days, # 3 days ago
|
||||
name: "Past Transaction",
|
||||
source: "simplefin",
|
||||
extra: { "simplefin" => { "pending" => true } }
|
||||
)
|
||||
|
||||
# Search from today - should find the past pending
|
||||
result = @adapter.find_pending_transaction(
|
||||
date: Date.today,
|
||||
amount: 75.00,
|
||||
currency: "USD",
|
||||
source: "simplefin"
|
||||
)
|
||||
|
||||
assert_equal past_pending.id, result.id
|
||||
end
|
||||
|
||||
# ============================================================================
|
||||
# Plaid pending_transaction_id Tests
|
||||
# ============================================================================
|
||||
|
||||
test "reconciles pending via Plaid pending_transaction_id" do
|
||||
# Import a pending transaction
|
||||
pending_entry = @adapter.import_transaction(
|
||||
external_id: "plaid_pending_abc",
|
||||
amount: 42.00,
|
||||
currency: "USD",
|
||||
date: Date.today - 2.days,
|
||||
name: "Coffee Shop",
|
||||
source: "plaid",
|
||||
extra: { "plaid" => { "pending" => true } }
|
||||
)
|
||||
|
||||
# Import posted with pending_transaction_id linking to the pending
|
||||
assert_no_difference "@account.entries.count" do
|
||||
posted_entry = @adapter.import_transaction(
|
||||
external_id: "plaid_posted_xyz",
|
||||
amount: 42.00,
|
||||
currency: "USD",
|
||||
date: Date.today,
|
||||
name: "Coffee Shop",
|
||||
source: "plaid",
|
||||
pending_transaction_id: "plaid_pending_abc", # Links to pending
|
||||
extra: { "plaid" => { "pending" => false, "pending_transaction_id" => "plaid_pending_abc" } }
|
||||
)
|
||||
|
||||
# Should claim the pending entry
|
||||
assert_equal pending_entry.id, posted_entry.id
|
||||
assert_equal "plaid_posted_xyz", posted_entry.external_id
|
||||
assert_not posted_entry.transaction.pending?
|
||||
end
|
||||
end
|
||||
|
||||
test "Plaid pending_transaction_id takes priority over amount matching" do
|
||||
# Create TWO pending transactions with same amount
|
||||
pending1 = @adapter.import_transaction(
|
||||
external_id: "plaid_pending_1",
|
||||
amount: 25.00,
|
||||
currency: "USD",
|
||||
date: Date.today - 1.day,
|
||||
name: "Store A",
|
||||
source: "plaid",
|
||||
extra: { "plaid" => { "pending" => true } }
|
||||
)
|
||||
|
||||
pending2 = @adapter.import_transaction(
|
||||
external_id: "plaid_pending_2",
|
||||
amount: 25.00,
|
||||
currency: "USD",
|
||||
date: Date.today - 1.day,
|
||||
name: "Store B",
|
||||
source: "plaid",
|
||||
extra: { "plaid" => { "pending" => true } }
|
||||
)
|
||||
|
||||
# Import posted that explicitly links to pending2 via pending_transaction_id
|
||||
assert_no_difference "@account.entries.count" do
|
||||
posted_entry = @adapter.import_transaction(
|
||||
external_id: "plaid_posted_linked",
|
||||
amount: 25.00,
|
||||
currency: "USD",
|
||||
date: Date.today,
|
||||
name: "Store B",
|
||||
source: "plaid",
|
||||
pending_transaction_id: "plaid_pending_2", # Explicitly links to pending2
|
||||
extra: { "plaid" => { "pending" => false } }
|
||||
)
|
||||
|
||||
# Should claim pending2 specifically (not pending1)
|
||||
assert_equal pending2.id, posted_entry.id
|
||||
assert_equal "plaid_posted_linked", posted_entry.external_id
|
||||
end
|
||||
|
||||
# pending1 should still exist as pending
|
||||
pending1.reload
|
||||
assert_equal "plaid_pending_1", pending1.external_id
|
||||
end
|
||||
end
|
||||
|
||||
@@ -29,8 +29,8 @@ class SimplefinEntry::ProcessorTest < ActiveSupport::TestCase
|
||||
payee: "Pizza Hut",
|
||||
description: "Order #1234",
|
||||
memo: "Carryout",
|
||||
posted: Date.today.to_s,
|
||||
transacted_at: (Date.today - 1).to_s,
|
||||
posted: Date.current.to_s,
|
||||
transacted_at: (Date.current - 1).to_s,
|
||||
extra: { category: "restaurants", check_number: nil }
|
||||
}
|
||||
|
||||
@@ -64,7 +64,7 @@ class SimplefinEntry::ProcessorTest < ActiveSupport::TestCase
|
||||
description: "Latte",
|
||||
memo: "Morning run",
|
||||
posted: nil,
|
||||
transacted_at: (Date.today - 3).to_s
|
||||
transacted_at: (Date.current - 3).to_s
|
||||
}
|
||||
|
||||
SimplefinEntry::Processor.new(tx, simplefin_account: @simplefin_account).process
|
||||
@@ -77,8 +77,8 @@ class SimplefinEntry::ProcessorTest < ActiveSupport::TestCase
|
||||
|
||||
test "captures FX metadata when tx currency differs from account currency" do
|
||||
# Account is USD from setup; use EUR for tx
|
||||
t_date = (Date.today - 5)
|
||||
p_date = Date.today
|
||||
t_date = (Date.current - 5)
|
||||
p_date = Date.current
|
||||
|
||||
tx = {
|
||||
id: "tx_fx_1",
|
||||
@@ -106,8 +106,8 @@ class SimplefinEntry::ProcessorTest < ActiveSupport::TestCase
|
||||
payee: "Test Store",
|
||||
description: "Auth",
|
||||
memo: "",
|
||||
posted: Date.today.to_s, # provider says pending=true should still flag
|
||||
transacted_at: (Date.today - 1).to_s,
|
||||
posted: Date.current.to_s, # provider says pending=true should still flag
|
||||
transacted_at: (Date.current - 1).to_s,
|
||||
pending: true
|
||||
}
|
||||
|
||||
@@ -120,7 +120,7 @@ class SimplefinEntry::ProcessorTest < ActiveSupport::TestCase
|
||||
|
||||
test "posted==0 treated as missing, entry uses transacted_at date and flags pending" do
|
||||
# Simulate provider sending epoch-like zeros for posted and an integer transacted_at
|
||||
t_epoch = (Date.today - 2).to_time.to_i
|
||||
t_epoch = (Date.current - 2).to_time.to_i
|
||||
tx = {
|
||||
id: "tx_pending_zero_posted_1",
|
||||
amount: "-6.48",
|
||||
@@ -141,4 +141,26 @@ class SimplefinEntry::ProcessorTest < ActiveSupport::TestCase
|
||||
sf = entry.transaction.extra.fetch("simplefin")
|
||||
assert_equal true, sf["pending"], "expected pending flag to be true when posted==0 and/or pending=true"
|
||||
end
|
||||
|
||||
test "infers pending when posted is explicitly 0 and transacted_at present (no explicit pending flag)" do
|
||||
# Some SimpleFIN banks indicate pending by sending posted=0 + transacted_at, without pending flag
|
||||
t_epoch = (Date.current - 1).to_time.to_i
|
||||
tx = {
|
||||
id: "tx_inferred_pending_1",
|
||||
amount: "-15.00",
|
||||
currency: "USD",
|
||||
payee: "Gas Station",
|
||||
description: "Fuel",
|
||||
memo: "",
|
||||
posted: 0,
|
||||
transacted_at: t_epoch
|
||||
# Note: NO pending flag set
|
||||
}
|
||||
|
||||
SimplefinEntry::Processor.new(tx, simplefin_account: @simplefin_account).process
|
||||
|
||||
entry = @account.entries.find_by!(external_id: "simplefin_tx_inferred_pending_1", source: "simplefin")
|
||||
sf = entry.transaction.extra.fetch("simplefin")
|
||||
assert_equal true, sf["pending"], "expected pending to be inferred from posted=0 + transacted_at present"
|
||||
end
|
||||
end
|
||||
|
||||
Reference in New Issue
Block a user