mirror of
https://github.com/we-promise/sure.git
synced 2026-04-09 07:14:47 +00:00
d4ab29ae1bc03f6bae073d9669aa32f434cc94de
4 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
LPW
|
c12c585a0e |
Harden SimpleFin sync: retries, safer imports, manual relinking, and data-quality reconciliation (#544)
* Add tests and enhance logic for SimpleFin account synchronization and reconciliation - Added retry logic with exponential backoff for network errors in `Provider::Simplefin`. - Introduced tests to verify retry functionality and error handling for rate-limit, server errors, and stale data. - Updated `SimplefinItem` to detect stale sync status and reconciliation issues. - Enhanced UI to display stale sync warnings and data integrity notices. - Improved SimpleFin account matching during updates with multi-tier strategy (ID, fingerprint, fuzzy match). - Added transaction reconciliation logic to detect data gaps, transaction count drops, and duplicate transaction IDs. * Introduce `SimplefinConnectionUpdateJob` for asynchronous SimpleFin connection updates - Moved SimpleFin connection update logic to `SimplefinConnectionUpdateJob` to improve response times by offloading network retries, data fetching, and reconciliation tasks. - Enhanced SimpleFin account matching with a multi-tier strategy (ID, fingerprint, fuzzy name match). - Added retry logic and bounded latency for token claim requests in `Provider::Simplefin`. - Updated tests to cover the new job flow and ensure correct account reconciliation during updates. * Remove unused SimpleFin account matching logic and improve error handling in `SimplefinConnectionUpdateJob` - Deleted the multi-tier account matching logic from `SimplefinItemsController` as it is no longer used. - Enhanced error handling in `SimplefinConnectionUpdateJob` to gracefully handle import failures, ensuring orphaned items can be manually resolved. - Updated job flow to conditionally set item status based on the success of import operations. * Fix SimpleFin sync: check both legacy FK and AccountProvider for linked accounts * Add crypto, checking, savings, and cash account detection; refine subtype selection and linking - Enhanced `Simplefin::AccountTypeMapper` to include detection for crypto, checking, savings, and standalone cash accounts. - Improved subtype selection UI with validation and warning indicators for missing selections. - Updated SimpleFin account linking to handle both legacy FK and `AccountProvider` associations consistently. - Refined job flow and importer logic for better handling of linked accounts and subtype inference. * Improve `SimplefinConnectionUpdateJob` and holdings processing logic - Fixed race condition in `SimplefinConnectionUpdateJob` by moving `destroy_later` calls outside of transactions. - Updated fuzzy name match logic to use Levenshtein distance for better accuracy. - Enhanced synthetic ticker generation in holdings processor with hash suffix for uniqueness. * Refine SimpleFin entry processing logic and ensure `extra` data persistence - Simplified pending flag determination to rely solely on provider-supplied values. - Fixed potential stale values in `extra` by ensuring deep merge overwrite with `entry.transaction.save!`. * Replace hardcoded fallback transaction description with localized string * Refine pending flag logic in SimpleFin processor tests - Adjust test to prevent falsely inferring pending status from missing posted dates. - Ensure provider explicitly sets pending flag for transactions. * Add `has_many :holdings` association to `AccountProvider` with `dependent: :nullify` --------- Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com> |
||
|
LPW
|
61eb611529 |
Simplefin enhancements v2 (#267)
* SimpleFin: metadata + merge fixes; holdings (incl. crypto) + Day Change; Sync Summary; ops rakes; lint # Conflicts: # db/schema.rb # Conflicts: # app/controllers/simplefin_items_controller.rb * fix testing * fix linting * xfix linting x2 * Review PR #267 on we-promise/sure (SimpleFin enhancements v2). Address all 15 actionable CodeRabbit comments: Add UUID validations in rakes (e.g., simplefin_unlink), swap Ruby pattern matching/loops for efficient DB queries (e.g., where LOWER(name) LIKE ?), generate docstrings for low-coverage areas (31%), consolidate routes for simplefin_items, move view logic to helpers (e.g., format_transaction_extra), strengthen tests with exact assertions/fixtures for dedup/relink failures. Also, check for overlaps with merged #262 (merchants fix): Ensure merchant creation in simplefin_entry/processor.rb aligns with new payee-based flow and MD5 IDs; add tests for edge cases like empty payees or over-merging pendings. Prioritize security (PII redaction in logs, no hardcoded secrets). * SimpleFin: address CodeRabbit comments (batch 1) - Consolidate simplefin_items routes under a single resources block; keep URLs stable - Replace inline JS with Stimulus auto-relink controller; auto-load relink modal via global modal frame - Improve a11y in relink modal by wrapping rows in labels - Harden unlink rake: default dry_run=true, UUID validation, redact PII in outputs, clearer errors - Backfill rake: default dry_run=true, UUID validation; groundwork for per-SFA counters - Fix-was-merged rake: default dry_run=true, UUID validation; clearer outputs - Idempotent transfer auto-match (find_or_create_by! + RecordNotUnique rescue) - Extract SimpleFin error tooltip assembly into helper and use it in view RuboCop: maintain 2-space indentation, spaces inside array brackets, spaces after commas, and no redundant returns * Linter noise * removed filed commited by mistake. * manual relink flow and tighten composite matching * enforce manual relink UI; fix adapter keywords; guarantee extra.simplefin hash * refactor(simplefin): extract relink service; enforce manual relink UI; tighten composite match; migration 7.2 * add provider date parser; refactor rake; move view queries; partial resilience * run balances-only import in background job. make update flow enqueue balances-only job * persists across all update redirects and initialize used_manual_ids to prevent NameError in relink candidate computation. * SimpleFin: metadata + merge fixes; holdings (incl. crypto) + Day Change; Sync Summary; ops rakes; lint * Fixed failed test after rebase. * scan_ruby fix * Calming the rabbit: Fix AccountProvider linking when accounts change Drop the legacy unique index instead of duplicating it Fix dynamic constant assignment Use fixtures consistently; avoid rescue for control flow. Replace bare rescue with explicit exception class. Move business logic out of the view. Critical: Transaction boundary excludes recompute phase, risking data loss. Inconsistency between documentation and implementation for zero-error case. Refactor to use the compute_unlinked_count helper for consistency. Fix cleanup task default: it deletes by default. Move sync stats computation to controller to avoid N+1 queries. Consolidate duplicate sync query. Clarify the intent of setting flash notice on the error path. Fix Date/Time comparison in should_be_inactive?. Move stats retrieval logic to controller. Remove duplicate Sync summary section. Remove the unnecessary sleep statement; use Capybara's built-in waiting. Add label wrappers for accessibility and consistency. * FIX SimpleFIN new account modal Now new account properly loads as a Modal, instead of new page. Fixes also form showing dashboard instead of settings page. * Remove SimpleFin legacy UI components, migrate schema, and refine linking behavior. # Conflicts: # app/helpers/settings_helper.rb * Extract SimpleFin-related logic to `prepare_show_context` helper and refactor for consistency. Adjust conditional checks and ensure controller variables are properly initialized. * Remove unused SimpleFin maps from prepare_show_context; select IDs to avoid N+1 Replace Tailwind bg-green-500 with semantic bg-success in _simplefin_panel/_provider_form Add f.label :setup_token in simplefin_items/new for a11y Remove duplicate require in AccountsControllerSimplefinCtaTest * Remove unnecessary blank lines * Reduce unnecessary changes This reduces the diff against main * Simplefin Account Setup: Display in modal This fixes an issue with the `X` dismiss button in the top right corner * Removed unnecessary comment. * removed unnecessary function. * fixed broken links * Removed unnecessary file * changed to database query * set to use UTC and gaurd against null * set dry_run=true * Fixed comment * Changed to use a database-level query * matched test name to test behavior. * Eliminate code duplication and Time.zone dependency * make final summary surface failures * lint fix * Revised timezone comment. better handle missing selectors. * sanitized LIKE wildcards * Fixed SimpleFin import to avoid “Currency can’t be blank” validation failures when providers return an empty currency string. * Added helper methods for admin and self-hosted checks * Specify exception types in rescue clauses. * Refined logic to determine transaction dates for credit accounts. * Refined stats calculation for `total_accounts` to track the maximum unique accounts per run instead of accumulating totals. * Moved `unlink_all!` logic to `SimplefinItem::Unlinking` concern and deprecated `SimplefinItem::Unlinker`. Updated related references. * Refined legacy unlinking logic, improved `current_holdings` formatting, and added ENV-based overrides for self-hosted checks. * Enhanced `unlink_all!` with explicit error handling, improved transaction safety, and refined ENV-based self-hosted checks. Adjusted exception types and cleaned up private method handling. * Improved currency assignment logic by adding fallback to `current_account` and `family` currencies. * Enhanced error tracking during SimpleFin account imports by adding categorized error buckets, limiting stored errors to the last 5, and improving `stats` calculations. * typo fix * Didn't realize rabbit was still mad... Refactored SimpleFin error handling and CTA logic: centralized duplicate detection and relink visibility into controller, improved task counters, adjusted redirect notices, and fixed form indexing. * Dang rabbit never stops... Centralized SimpleFin maps logic into `MapsHelper` concern and integrated it into relevant controllers and rake tasks. Optimized queries, reduced redundancy, and improved unlinked counts and manual account checks with batch processing. Adjusted task arguments for clarity. * Persistent rabbit. Optimized SimpleFin maps logic by implementing batch queries for manual account and unlinked count checks, reducing N+1 issues. Improved clarity of rake task argument descriptions and error messages for better usability. * Lost a commit somehow, resolved here. Refactored transaction extra details logic by introducing `build_transaction_extra_details` helper to improve clarity, reusability, and reduce view complexity. Enhanced rake tasks with strict dry-run validation and better error handling. Updated schema to allow nullable `merchant_id` and added conditional unique indexes for recurring transactions. * Refactored sensitive data redaction in `simplefin_unlink` task for recursive handling, optimized SQL sanitization in `simplefin_holdings_backfill`, improved error handling in `transactions_helper`, and streamlined day change calculation logic in `Holding` model. * Lint fix * Removed per PR comments. * Also removing per PR comment. * git commit -m "SimpleFIN polish: preserve #manual-accounts wrapper, unify \"manual\" scope, and correct unlinked counts - Preserve #manual-accounts wrapper: switch non-empty updates to turbo_stream.update and background broadcast_update_to; keep empty-path replace to render <div id=\"manual-accounts\"></div> - Unify definition of manual accounts via Account.visible_manual (visible + legacy-nil + no AccountProvider); reuse in controllers, jobs, and helper - Correct setup/unlinked counts: SimplefinItem::Syncer#finalize_setup_counts and maps now consider AccountProvider links (legacy account AND provider must be absent) Deleted: - app/models/simplefin_item/relink_service.rb - app/controllers/concerns/simplefin_items/relink_helpers.rb - app/javascript/controllers/auto_relink_controller.js - app/views/simplefin_items/_relink_modal.html.erb - app/views/simplefin_items/manual_relink.html.erb - app/views/simplefin_items/relink.html.erb - test/services/simplefin_item/relink_service_test.rb Refs: PR #318 unified link/unlink; PR #267 SimpleFIN; follow-up to fix wrapper ID loss and counting drift." * Extend unlinked account check to include "Investment" type * set SimpleFIN item for `balances`, remove redundant unpacking, and improve holdings task error * SimpleFIN: add `errors` action + modal; do not reintroduce legacy relink actions; removed dead helper * FIX simpleFIN linking * Add delay back, tests benefit from it * Put cache back in * Remove empty `rake` task * Small spelling fixes. --------- Signed-off-by: soky srm <sokysrm@gmail.com> Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: sokie <sokysrm@gmail.com> Co-authored-by: Dylan Corrales <deathcamel58@gmail.com> |
||
|
soky srm
|
4fb0a3856e |
Providers factory (#250)
* Implement providers factory * Multiple providers sync support - Proper Multi-Provider Syncing: When you click sync on an account with multiple providers (e.g., both Plaid and SimpleFin), all provider items are synced - Better API: The existing account.providers method already returns all providers, and account.provider returns the first one for backward compatibility - Correct Holdings Deletion Logic: Holdings can only be deleted if ALL providers allow it, preventing accidental deletions that would be recreated on next sync TODO: validate this is the way we want to go? We would need to check holdings belong to which account, and then check provider allows deletion. More complex - Database Constraints: The existing validations ensure an account can have at most one provider of each type (one PlaidAccount, one SimplefinAccount, etc.) * Add generic provider_import_adapter * Finish unified import strategy * Update app/models/plaid_account.rb Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: soky srm <sokysrm@gmail.com> * Update app/models/provider/factory.rb Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: soky srm <sokysrm@gmail.com> * Fix account linked by plaid_id instead of external_id * Parse numerics to BigDecimal Parse numerics to BigDecimal before computing amount; guard nils. Avoid String * String and float drift; also normalize date. * Fix incorrect usage of assert_raises. * Fix linter * Fix processor test. * Update current_balance_manager.rb * Test fixes * Fix plaid linked account test * Add support for holding per account_provider * Fix proper account access Also fix account deletion for simpefin too * FIX match tests for consistency * Some more factory updates * Fix account schema for multipe providers Can do: - Account #1 → PlaidAccount + SimplefinAccount (multiple different providers) - Account #2 → PlaidAccount only - Account #3 → SimplefinAccount only Cannot do: - Account #1 → PlaidAccount + PlaidAccount (duplicate provider type) - PlaidAccount #123 → Account #1 + Account #2 (provider linked to multiple accounts) * Fix account setup - An account CAN have multiple providers (the schema shows account_providers with unique index on [account_id, provider_type]) - Each provider should maintain its own separate entries - We should NOT update one provider's entry when another provider syncs * Fix linter and guard migration * FIX linter issues. * Fixes - Remove duplicated index - Pass account_provider_id - Guard holdings call to avoid NoMethodError * Update schema and provider import fix * Plaid doesn't allow holdings deletion * Use ClimateControl for proper env setup * No need for this in .git --------- Signed-off-by: soky srm <sokysrm@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
Himmelschmidt
|
7e36b1c7c5 |
Feature/simplefin integration (#94)
* Add HTTParty gem for SimpleFin API integration - Add HTTParty gem for making HTTP requests to SimpleFin API - Required for SimpleFin protocol implementation * Add SimpleFin database schema - Create simplefin_items table for SimpleFin connections - Create simplefin_accounts table for account metadata - Add simplefin_account_id to accounts table for linking - Add external_id to transactions for deduplication - Enable encrypted storage of SimpleFin access URLs * Implement SimpleFin API client and data models - Add SimplefinItem model with sync capabilities and encryption - Add SimplefinAccount model for account data mapping - Implement Provider::Simplefin API client with token exchange - Add SimpleFin protocol support with proper error handling - Include sync jobs, importers, and processors for data flow - Add family SimpleFin connectivity mixin * Update core models for SimpleFin integration - Add SimpleFin account creation methods to Account model - Implement intelligent account type mapping from names - Add SimpleFin linkable functionality to Account - Include SimpleFin items in Family model associations - Support account creation with user-selected types * Add SimpleFin controllers and routing - Create SimplefinItemsController with CRUD operations - Add account setup flow with user type selection - Include sync management and error handling - Update AccountsController to display SimpleFin items - Add routes for SimpleFin item management and setup * Add SimpleFin user interface components - Create SimpleFin connection management views - Add account setup modal with type selection - Include connection form with token input and instructions - Update accounts index to display SimpleFin items - Add SimpleFin option to account method selector - Include SimpleFin in settings navigation * Add user account type selection workflow - Add pending_account_setup field to SimpleFin items - Enable pausing sync for user account type selection - Allow users to choose account types during import - Prevent automatic account creation until user confirms * Add tests for SimpleFin integration - Add SimplefinItem model tests with fixtures - Add SimplefinAccount model tests - Add SimplefinItemsController tests - Include test coverage for sync and account creation * Fix account show page for SimpleFin accounts - Update sync button routing to handle SimpleFin accounts - Add SimpleFin item sync path alongside existing Plaid support - Prevent NoMethodError when viewing SimpleFin-linked accounts - Support proper sync routing for Plaid, SimpleFin, and manual accounts * Complete subtype selection for SimpleFin accounts - Add subtype database columns to all accountable models - Create Stimulus controller for dynamic subtype dropdown interaction - Add delegation from Account to accountable subtype for clean API access - Update SimpleFin account setup form with working subtype selection - Fix account display to show proper subtype labels instead of generic "Cash" Users can now select both account type and subtype during SimpleFin import, and the selected subtypes are properly saved and displayed in the UI. * Fix dark mode compatibility for SimpleFin UI components - Replace hardcoded colors with design system tokens throughout SimpleFin views - Fix method selector hover states to use bg-surface instead of bg-gray-50 - Update SimpleFin form to use styled_form_with and standard form patterns - Replace custom button styling with design system button components - Fix info boxes and containers to use bg-surface and border-primary - Replace hardcoded green/blue colors with text-primary, text-secondary, text-link - Remove custom text area styling to allow form builder defaults (dark mode support) All SimpleFin components now properly adapt to both light and dark themes with correct contrast and visibility. * Fix SimpleFin integration bugs and improve code quality - Fix upsert method to handle string/symbol keys with indifferent access - Add missing show route and view for SimpleFin items - Fix test fixtures to use correct user references - Update test data to match real-world JSON format (string keys, BigDecimal) - Apply code formatting and linting fixes (rubocop, erb_lint) - Ensure all SimpleFin tests pass (16/16 passing) * Remove SimpleFin demo file with outdated setup token * Update SimpleFin User-Agent to use Sure Finance branding * Remove unused SimpleFin account type mapping logic - Remove map_simplefin_type_to_accountable_type method (no longer needed) - Remove create_from_simplefin_account method (manual setup only) - Simplify account type selection UI to not pre-select defaults - Update processor to log error if account missing (safety check) - All account creation now goes through manual user selection flow * Gate SimpleFin option behind US region check SimpleFin is primarily for North American financial institutions, so only show the option when US banking connections are available. * Refactor SimpleFin controller to use model method - Move SimpleFin item creation logic from controller to Family#create_simplefin_item! - Remove duplication between controller and model - Simplify controller to focus on web request/response handling - Remove unused simplefin_provider method - Follow Rails best practices for fat models, skinny controllers * Fix critical data integrity issue in SimpleFin date parsing - Remove fallback to Date.current when transaction dates fail to parse - Raise ArgumentError instead to ensure data integrity - Log detailed error messages for debugging - Skip transactions with invalid dates rather than using incorrect dates - Prevents hard-to-debug issues with balances and financial reports * Address all Gemini code review feedback for SimpleFin integration - Remove debug console.log statements from JavaScript controller - Consolidate duplicate SimpleFin account creation methods into single method - Refactor SimplefinItemsController to reduce complexity with helper methods - Fix HTTParty thread-safety by moving SSL options to class level - Remove redundant HTTParty options from individual requests - Add proper error logging for invalid currency URIs - Extract sync button path logic to AccountsHelper#sync_path_for method - DRY up repeated subtype dropdown code with reusable partial and data structure All SimpleFin tests passing (16/16). Code quality improvements maintain backward compatibility while following Rails best practices. * Fix tests for subtype delegation to accountable models The subtype attribute was moved from Account to individual accountable models to enable users to select specific subtypes during SimpleFin account import. This change allows for better account categorization and more precise display of account types (e.g., "HSA" instead of generic "Cash"). However, tests and the PlaidAccount processor weren't updated to work with the new delegation pattern. This commit fixes: - PlaidAccount::Processor now sets subtype on accountable and uses enrichable pattern to respect user locks - PropertiesController updated to handle subtype via accountable_attributes - Test fixtures corrected to set subtype on accountable models not Account - Tests updated to work with the delegated subtype pattern All originally failing tests now pass: - PropertiesControllerTest#test_updates_property_overview - PlaidAccount::ProcessorTest (2 failing tests) - AccountTest#test_gets_short/long_subtype_label * Fix trailing whitespace (rubocop auto-fix) * Add option to "skip" adding an account * Revert "Gate SimpleFin option behind US region check" This reverts commit |