4 Commits

Author	SHA1	Message	Date
LPW	3658e812a8	Add pending transaction handling and duplicate reconciliation logic (#602) ... * Add pending transaction handling and duplicate reconciliation logic - Implemented logic to exclude pending transactions from budgets and analytics calculations. - Introduced mechanisms for reconciling pending transactions with posted versions. - Added duplicate detection with support for merging or dismissing matches. - Updated transaction search filters to include a `status_filter` for pending/confirmed transactions. - Introduced UI elements for reviewing and resolving duplicates. - Enhanced `ProviderSyncSummary` with stats for reconciled and stale pending transactions. * Refactor translation handling and enhance transaction and sync logic - Moved hardcoded strings to locale files for improved translation support. - Refined styling for duplicate transaction indicators and sync summaries. - Improved logic for excluding stale pending transactions and updating timestamps on batch exclusion. - Added unique IDs to status filters for better element targeting in UI. - Optimized database queries to avoid N+1 issues in stale pending calculations. * Add sync settings and enhance pending transaction handling - Introduced a new "Sync Settings" section in hosting settings with UI to toggle inclusion of pending transactions. - Updated handling of pending transactions with improved inference logic for `posted=0` and `transacted_at` in processors. - Added priority order for pending transaction inclusion: explicit argument > environment variable > runtime configurable setting. - Refactored settings and controllers to store updated sync preferences. * Refactor sync settings and pending transaction reconciliation - Extracted logic for pending transaction reconciliation, stale exclusion, and unmatched tracking into dedicated methods for better maintainability. - Updated sync settings to infer defaults from multiple provider environment variables (`SIMPLEFIN_INCLUDE_PENDING`, `PLAID_INCLUDE_PENDING`). - Refined UI and messaging to handle multi-provider configurations in sync settings. # Conflicts: # app/models/simplefin_item/importer.rb * Debounce transaction reconciliation during imports - Added per-run reconciliation debouncing to prevent repeated scans for the same account during chunked history imports. - Trimmed size of reconciliation stats to retain recent details only. - Introduced error tracking for reconciliation steps to improve UI visibility of issues. * Apply ABS() in pending transaction queries and improve error handling - Updated pending transaction logic to use ABS() for consistent handling of negative amounts. - Adjusted amount bounds calculations to ensure accuracy for both positive and negative values. - Refined exception handling in `merge_duplicate` to log failures and update user alert. - Replaced `Date.today` with `Date.current` in tests to ensure timezone consistency. - Minor optimization to avoid COUNT queries by loading limited records directly. * Improve error handling in duplicate suggestion and dismissal logic - Added exception handling for `store_duplicate_suggestion` to log failures and prevent crashes during fuzzy/low-confidence matches. - Enhanced `dismiss_duplicate` action to handle `ActiveRecord::RecordInvalid` and display appropriate user alerts. --------- Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>	2026-01-10 20:11:00 +01:00
LPW	c12c585a0e	Harden SimpleFin sync: retries, safer imports, manual relinking, and data-quality reconciliation (#544) ... * Add tests and enhance logic for SimpleFin account synchronization and reconciliation - Added retry logic with exponential backoff for network errors in `Provider::Simplefin`. - Introduced tests to verify retry functionality and error handling for rate-limit, server errors, and stale data. - Updated `SimplefinItem` to detect stale sync status and reconciliation issues. - Enhanced UI to display stale sync warnings and data integrity notices. - Improved SimpleFin account matching during updates with multi-tier strategy (ID, fingerprint, fuzzy match). - Added transaction reconciliation logic to detect data gaps, transaction count drops, and duplicate transaction IDs. * Introduce `SimplefinConnectionUpdateJob` for asynchronous SimpleFin connection updates - Moved SimpleFin connection update logic to `SimplefinConnectionUpdateJob` to improve response times by offloading network retries, data fetching, and reconciliation tasks. - Enhanced SimpleFin account matching with a multi-tier strategy (ID, fingerprint, fuzzy name match). - Added retry logic and bounded latency for token claim requests in `Provider::Simplefin`. - Updated tests to cover the new job flow and ensure correct account reconciliation during updates. * Remove unused SimpleFin account matching logic and improve error handling in `SimplefinConnectionUpdateJob` - Deleted the multi-tier account matching logic from `SimplefinItemsController` as it is no longer used. - Enhanced error handling in `SimplefinConnectionUpdateJob` to gracefully handle import failures, ensuring orphaned items can be manually resolved. - Updated job flow to conditionally set item status based on the success of import operations. * Fix SimpleFin sync: check both legacy FK and AccountProvider for linked accounts * Add crypto, checking, savings, and cash account detection; refine subtype selection and linking - Enhanced `Simplefin::AccountTypeMapper` to include detection for crypto, checking, savings, and standalone cash accounts. - Improved subtype selection UI with validation and warning indicators for missing selections. - Updated SimpleFin account linking to handle both legacy FK and `AccountProvider` associations consistently. - Refined job flow and importer logic for better handling of linked accounts and subtype inference. * Improve `SimplefinConnectionUpdateJob` and holdings processing logic - Fixed race condition in `SimplefinConnectionUpdateJob` by moving `destroy_later` calls outside of transactions. - Updated fuzzy name match logic to use Levenshtein distance for better accuracy. - Enhanced synthetic ticker generation in holdings processor with hash suffix for uniqueness. * Refine SimpleFin entry processing logic and ensure `extra` data persistence - Simplified pending flag determination to rely solely on provider-supplied values. - Fixed potential stale values in `extra` by ensuring deep merge overwrite with `entry.transaction.save!`. * Replace hardcoded fallback transaction description with localized string * Refine pending flag logic in SimpleFin processor tests - Adjust test to prevent falsely inferring pending status from missing posted dates. - Ensure provider explicitly sets pending flag for transactions. * Add `has_many :holdings` association to `AccountProvider` with `dependent: :nullify` --------- Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>	2026-01-05 22:11:47 +01:00
LPW	664c6c2b7c	Pending detection, FX metadata, Pending UI badge. (#374) ... * - Add support for `SIMPLEFIN_INCLUDE_PENDING` to control pending behavior via ENV. - Enhance debug logging for SimpleFin API requests and raw payloads. - Refine pending flag handling in `SimplefinEntry::Processor` based on provider data and inferred conditions. - Improve FX metadata processing for transactions with currency mismatches. - Add new tests for pending detection, FX metadata, and edge cases involving `posted` values. - Add pending indicator UI to transaction view. * Document pending transaction detection, storage, and UI behavior for SimpleFIN and Plaid integrations. Add debug flags for troubleshooting. * Add `pending?` method to `Transaction` model, refactor UI indicator, and centralize SimpleFIN configuration - Introduced `pending?` method in `Transaction` for unified pending state detection. - Refactored transaction pending indicator in the UI to use `pending?` method. - Centralized SimpleFIN configuration in initializer with ENV-backed toggles. - Updated tests for `pending?` behavior and clarified docs for pending detection logic * Add SimpleFIN debug and runtime flags to `.env.local.example` and `.env.test.example` - Introduced `SIMPLEFIN_INCLUDE_PENDING` and `SIMPLEFIN_DEBUG_RAW` flags for controlling pending behavior and debugging. - Updated example environment files with descriptions for new configuration options. * Normalize formatting for `SIMPLEFIN_INCLUDE_PENDING` and `SIMPLEFIN_DEBUG_RAW` flags in `.env.local.example` and `.env.test.example`. --------- Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>	2025-12-19 23:24:48 +01:00
LPW	61eb611529	Simplefin enhancements v2 (#267) ... * SimpleFin: metadata + merge fixes; holdings (incl. crypto) + Day Change; Sync Summary; ops rakes; lint # Conflicts: # db/schema.rb # Conflicts: # app/controllers/simplefin_items_controller.rb * fix testing * fix linting * xfix linting x2 * Review PR #267 on we-promise/sure (SimpleFin enhancements v2). Address all 15 actionable CodeRabbit comments: Add UUID validations in rakes (e.g., simplefin_unlink), swap Ruby pattern matching/loops for efficient DB queries (e.g., where LOWER(name) LIKE ?), generate docstrings for low-coverage areas (31%), consolidate routes for simplefin_items, move view logic to helpers (e.g., format_transaction_extra), strengthen tests with exact assertions/fixtures for dedup/relink failures. Also, check for overlaps with merged #262 (merchants fix): Ensure merchant creation in simplefin_entry/processor.rb aligns with new payee-based flow and MD5 IDs; add tests for edge cases like empty payees or over-merging pendings. Prioritize security (PII redaction in logs, no hardcoded secrets). * SimpleFin: address CodeRabbit comments (batch 1) - Consolidate simplefin_items routes under a single resources block; keep URLs stable - Replace inline JS with Stimulus auto-relink controller; auto-load relink modal via global modal frame - Improve a11y in relink modal by wrapping rows in labels - Harden unlink rake: default dry_run=true, UUID validation, redact PII in outputs, clearer errors - Backfill rake: default dry_run=true, UUID validation; groundwork for per-SFA counters - Fix-was-merged rake: default dry_run=true, UUID validation; clearer outputs - Idempotent transfer auto-match (find_or_create_by! + RecordNotUnique rescue) - Extract SimpleFin error tooltip assembly into helper and use it in view RuboCop: maintain 2-space indentation, spaces inside array brackets, spaces after commas, and no redundant returns * Linter noise * removed filed commited by mistake. * manual relink flow and tighten composite matching * enforce manual relink UI; fix adapter keywords; guarantee extra.simplefin hash * refactor(simplefin): extract relink service; enforce manual relink UI; tighten composite match; migration 7.2 * add provider date parser; refactor rake; move view queries; partial resilience * run balances-only import in background job. make update flow enqueue balances-only job * persists across all update redirects and initialize used_manual_ids to prevent NameError in relink candidate computation. * SimpleFin: metadata + merge fixes; holdings (incl. crypto) + Day Change; Sync Summary; ops rakes; lint * Fixed failed test after rebase. * scan_ruby fix * Calming the rabbit: Fix AccountProvider linking when accounts change Drop the legacy unique index instead of duplicating it Fix dynamic constant assignment Use fixtures consistently; avoid rescue for control flow. Replace bare rescue with explicit exception class. Move business logic out of the view. Critical: Transaction boundary excludes recompute phase, risking data loss. Inconsistency between documentation and implementation for zero-error case. Refactor to use the compute_unlinked_count helper for consistency. Fix cleanup task default: it deletes by default. Move sync stats computation to controller to avoid N+1 queries. Consolidate duplicate sync query. Clarify the intent of setting flash notice on the error path. Fix Date/Time comparison in should_be_inactive?. Move stats retrieval logic to controller. Remove duplicate Sync summary section. Remove the unnecessary sleep statement; use Capybara's built-in waiting. Add label wrappers for accessibility and consistency. * FIX SimpleFIN new account modal Now new account properly loads as a Modal, instead of new page. Fixes also form showing dashboard instead of settings page. * Remove SimpleFin legacy UI components, migrate schema, and refine linking behavior. # Conflicts: # app/helpers/settings_helper.rb * Extract SimpleFin-related logic to `prepare_show_context` helper and refactor for consistency. Adjust conditional checks and ensure controller variables are properly initialized. * Remove unused SimpleFin maps from prepare_show_context; select IDs to avoid N+1 Replace Tailwind bg-green-500 with semantic bg-success in _simplefin_panel/_provider_form Add f.label :setup_token in simplefin_items/new for a11y Remove duplicate require in AccountsControllerSimplefinCtaTest * Remove unnecessary blank lines * Reduce unnecessary changes This reduces the diff against main * Simplefin Account Setup: Display in modal This fixes an issue with the `X` dismiss button in the top right corner * Removed unnecessary comment. * removed unnecessary function. * fixed broken links * Removed unnecessary file * changed to database query * set to use UTC and gaurd against null * set dry_run=true * Fixed comment * Changed to use a database-level query * matched test name to test behavior. * Eliminate code duplication and Time.zone dependency * make final summary surface failures * lint fix * Revised timezone comment. better handle missing selectors. * sanitized LIKE wildcards * Fixed SimpleFin import to avoid “Currency can’t be blank” validation failures when providers return an empty currency string. * Added helper methods for admin and self-hosted checks * Specify exception types in rescue clauses. * Refined logic to determine transaction dates for credit accounts. * Refined stats calculation for `total_accounts` to track the maximum unique accounts per run instead of accumulating totals. * Moved `unlink_all!` logic to `SimplefinItem::Unlinking` concern and deprecated `SimplefinItem::Unlinker`. Updated related references. * Refined legacy unlinking logic, improved `current_holdings` formatting, and added ENV-based overrides for self-hosted checks. * Enhanced `unlink_all!` with explicit error handling, improved transaction safety, and refined ENV-based self-hosted checks. Adjusted exception types and cleaned up private method handling. * Improved currency assignment logic by adding fallback to `current_account` and `family` currencies. * Enhanced error tracking during SimpleFin account imports by adding categorized error buckets, limiting stored errors to the last 5, and improving `stats` calculations. * typo fix * Didn't realize rabbit was still mad... Refactored SimpleFin error handling and CTA logic: centralized duplicate detection and relink visibility into controller, improved task counters, adjusted redirect notices, and fixed form indexing. * Dang rabbit never stops... Centralized SimpleFin maps logic into `MapsHelper` concern and integrated it into relevant controllers and rake tasks. Optimized queries, reduced redundancy, and improved unlinked counts and manual account checks with batch processing. Adjusted task arguments for clarity. * Persistent rabbit. Optimized SimpleFin maps logic by implementing batch queries for manual account and unlinked count checks, reducing N+1 issues. Improved clarity of rake task argument descriptions and error messages for better usability. * Lost a commit somehow, resolved here. Refactored transaction extra details logic by introducing `build_transaction_extra_details` helper to improve clarity, reusability, and reduce view complexity. Enhanced rake tasks with strict dry-run validation and better error handling. Updated schema to allow nullable `merchant_id` and added conditional unique indexes for recurring transactions. * Refactored sensitive data redaction in `simplefin_unlink` task for recursive handling, optimized SQL sanitization in `simplefin_holdings_backfill`, improved error handling in `transactions_helper`, and streamlined day change calculation logic in `Holding` model. * Lint fix * Removed per PR comments. * Also removing per PR comment. * git commit -m "SimpleFIN polish: preserve #manual-accounts wrapper, unify \"manual\" scope, and correct unlinked counts - Preserve #manual-accounts wrapper: switch non-empty updates to turbo_stream.update and background broadcast_update_to; keep empty-path replace to render <div id=\"manual-accounts\"></div> - Unify definition of manual accounts via Account.visible_manual (visible + legacy-nil + no AccountProvider); reuse in controllers, jobs, and helper - Correct setup/unlinked counts: SimplefinItem::Syncer#finalize_setup_counts and maps now consider AccountProvider links (legacy account AND provider must be absent) Deleted: - app/models/simplefin_item/relink_service.rb - app/controllers/concerns/simplefin_items/relink_helpers.rb - app/javascript/controllers/auto_relink_controller.js - app/views/simplefin_items/_relink_modal.html.erb - app/views/simplefin_items/manual_relink.html.erb - app/views/simplefin_items/relink.html.erb - test/services/simplefin_item/relink_service_test.rb Refs: PR #318 unified link/unlink; PR #267 SimpleFIN; follow-up to fix wrapper ID loss and counting drift." * Extend unlinked account check to include "Investment" type * set SimpleFIN item for `balances`, remove redundant unpacking, and improve holdings task error * SimpleFIN: add `errors` action + modal; do not reintroduce legacy relink actions; removed dead helper * FIX simpleFIN linking * Add delay back, tests benefit from it * Put cache back in * Remove empty `rake` task * Small spelling fixes. --------- Signed-off-by: soky srm <sokysrm@gmail.com> Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: sokie <sokysrm@gmail.com> Co-authored-by: Dylan Corrales <deathcamel58@gmail.com>	2025-11-17 21:51:37 +01:00