Files
sure/app/models/account.rb
Himmelschmidt 4cd737b5d9 Fix SimpleFin investment holdings and comprehensive integration improvements (#104)
* Remove skipped account functionality from SimpleFin

- Remove "Skip - don't add" option from account setup
- Simplify account setup flow to require all accounts be assigned types
- Update controller logic to handle all accounts without skipping
- Redirect to accounts page instead of SimpleFin items page
- Add I18N message support with t(".success")

* Simplify SimpleFin sync logic by removing skipped accounts

- Remove skipped account filtering from syncer
- All unlinked accounts now block sync until setup is complete
- Remove skipped account UI elements from setup view
- Streamline sync flow without complex skipped/non-skipped logic

* Fix cash balance calculation for SimpleFin investment accounts

- Update SimplefinAccount::Processor to recalculate balances during sync
- Properly calculate cash_balance for investment accounts using BalanceCalculator
- Handle negative balances for credit cards and loans correctly
- Ensure account balance and cash balance are updated from latest SimpleFin data

* Add I18N translations and edit view for SimpleFin

- Add comprehensive English translations for SimpleFin UI
- Create edit view for SimpleFin re-authentication
- Support status messages, errors, and user feedback
- Match translation structure with Plaid integration

* Add specialized SimpleFin data processors

- Add investment processors for transactions, holdings, and balance calculation
- Add liability processors for credit cards and loans
- Add transaction processor for standard account transactions
- Add account importer for SimpleFin account data
- Organize processors by account type for maintainable architecture

* Add loading button controller for SimpleFin forms

- Add Stimulus controller to show loading state during form submission
- Disable button and show loading text to prevent double submissions
- Improve user experience during SimpleFin account setup

* Add SimpleFin edit and update routes

- Add edit and update actions to SimpleFin items routes
- Enable re-authentication flow for expired SimpleFin connections
- Match route structure with Plaid items for consistency

* Add institution metadata fields to SimpleFin items

- Add institution_id, institution_name, institution_domain fields
- Add institution_url, institution_color for UI customization
- Add raw_institution_payload for complete institution data storage
- Enable better institution organization and display

* Enhance SimpleFin item with institution support and metadata

- Add institution summary and connected institutions methods
- Store and retrieve institution metadata from SimpleFin API
- Add institution-aware import functionality
- Support multiple institutions per SimpleFin connection

* Fix account creation and Plaid provider issues

- Fix cash balance calculation in Account.create_from_simplefin_account
- Add nil check for plaid_provider in remove_plaid_item method
- Ensure proper balance handling for investment accounts during creation

* Improve sync parent relationship handling

- Add parent sync assignment for existing syncs when parent_sync is provided
- Ensure sync hierarchy is maintained when expanding sync windows
- Fix sync relationship consistency in nested sync operations

* Update SimpleFin item view with enhanced UI

- Improve SimpleFin connection display and status information
- Add better visual styling and user feedback
- Match UI consistency with Plaid item views

* Update database schema with institution fields

- Add institution metadata fields to simplefin_items table
- Support institution tracking and organization features

* Update SimpleFin tests for new functionality

- Update controller tests for edit/update actions and removed skip functionality
- Update model tests for institution metadata and enhanced features
- Ensure test coverage for SimpleFin improvements

* Add migration to remove old institution fields

* Fix linting issues

* Apply suggestion from @coderabbitai[bot]

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: Himmelschmidt <46351743+Himmelschmidt@users.noreply.github.com>

* Fix 2 failing tests

* Wrap SimpleFin account transfer in database transaction

* Make loading button controller more reusable

- Add loadingText Stimulus value for configurable loading text
- Remove unused originalText variable
- Update view to pass loading text via data attribute

* Remove unused SimplefinAccount::Importer class

This class was added in the PR but is never called anywhere in the codebase.
The actual SimpleFin account processing is handled by SimplefinAccount::Processor
and its specialized sub-processors.

* Fix SimpleFin account transfer bug during token updates

- Call import_latest_simplefin_data before account transfer to ensure
  new SimpleFin accounts exist with account_id populated
- Prevents silent failure where accounts become orphaned when
  refreshing expired SimpleFin tokens

* Fix SimpleFin error handling to render correct template and use i18n

- Update render_error method to accept context parameter for template selection
- Fix update action to render :edit template instead of :new on errors
- Replace hardcoded error messages with localized strings using t() calls
- Add comprehensive error message keys to SimpleFin locale file

* Improve loading button accessibility and HTML semantics

- Add aria-disabled and aria-busy attributes for screen readers
- Use semantic span elements instead of divs for button content
- Add aria-hidden to decorative spinner element

* Fix SimpleFin SSL verification to use OpenSSL constant

* Remove HTTParty streaming to prevent empty response body and PII logging

* Use BigDecimal zero for consistent numeric types in balance calculator

* Add investment account guard to holdings processor

* Remove duplicate balance normalization from SimpleFin loan processor

* Fix critical account deletion bug in SimpleFin token update

* Fix linting issues in SimpleFin controller tests

* Replace hardcoded colors with design system tokens in SimpleFin views

* Gate investment processors to Investment accounts only

Prevents investment processors from running on non-Investment account types,
matching the pattern used by liability processors.

* Localize hardcoded strings in SimpleFin edit form

* Adding the error message to a hover state.

* Use only 1 month for sync_start_date, new account restriction?

* Harden investment cash_balance calculation with error handling

- Add try/catch around SimplefinAccount::Investments::BalanceCalculator
- Fallback to zero on calculation errors or nil results
- Log warning with error details for debugging
- Prevents nil cash_balance that could cause downstream issues

* Fix SimpleFin institution fields migration and add DB constraints

- Remove destructive migration that dropped existing institution fields
- Add only new fields (institution_domain, institution_color)
- Add indexes on institution fields for query performance
- Add NOT NULL constraints on required fields (institution_id, institution_name)
- Fix schema jsonb consistency for raw_institution_payload

* Improve SimpleFin holdings error handling and BigDecimal consistency

* Add security attribute to external link in SimpleFin edit form

* Improve SimpleFin sync timing and add user-configurable date range

- Fix initial sync timing issue: change from 1 month to 7 days default lookback
- Add user-selectable sync start date in account setup UI
- Implement chunked historical sync that respects user-selected date range
- Add sync_start_date field to SimpleFin items
- Handle new accounts on existing connections with same date picker

This addresses SimpleFin API limitations and gives users control over
how much transaction history to sync during initial setup.

* Fix SimpleFin sync status to show detailed results instead of "Never synced"

- Modify sync completion logic to always complete even with unlinked accounts
- Add sync_stats column to track account sync statistics during sync process
- Update sync status display to show "X synced, Y need setup" instead of "Never synced"
- Store detailed sync statistics (total, linked, unlinked accounts) in sync record
- Add sync_status_summary method to provide meaningful status text
- Remove early return that prevented sync completion when accounts need setup

Resolves issue where successful account syncing still showed "Never synced" status.

* Fix Transaction persistence before Entry creation in SimpleFin processor

Persist Transaction with create! instead of new to ensure it has an ID before
creating Entry that references it as entryable. Prevents foreign key errors.

* Fix indifferent access for SimpleFin institution data extraction

The accounts_snapshot parameter comes from JSON with string keys, but the
code was accessing with symbol keys which could silently fail. Convert to
indifferent access to handle both string and symbol keys properly.

* Localize hardcoded deletion in progress string

Replace hardcoded "(deletion in progress...)" text with I18n translation
to maintain consistency with the rest of the view.

* Fix SimpleFin item update test to properly verify rebind/destroy behavior

The test now creates a different SimplefinItem instance and mocks
create_simplefin_item! to return it, ensuring the controller operates
on a new record instead of the same instance. This properly exercises
the rebind/destroy logic and verifies the original item is scheduled
for deletion.

* Fix potential transaction data loss in SimpleFin importer

Prevent wiping stored transactions when API omits transaction data by only
updating raw_transactions_payload when transactions are actually present
in the response, preserving existing transaction data when API doesn't
include transactions key.

* Fix SimpleFin sync chunking and enforce 3-year historical limit

- Fix SimpleFin's actual API limit from 365 days to 60 days per request
- Implement proper backward-walking chunked sync for historical data
- Enforce 3-year maximum historical data limit (60 days × 22 requests)
- Update date picker to reflect 3-year limit and better defaults
- Add comprehensive logging for debugging sync issues

* Add dedicated raw_holdings_payload storage for SimpleFin accounts

- Add raw_holdings_payload column to simplefin_accounts table
- Separates holdings data from general account data for cleaner processing
- Follows same pattern as raw_transactions_payload for consistency
- Enables proper SimpleFin holdings processing pipeline

* Enhance SimpleFin holdings storage with external ID tracking

- Add external_id and cost_basis columns to holdings table
- Update holdings processor to use external_id for precise matching
- Capture all available SimpleFin holdings data (shares, market_value, cost_basis, etc.)
- Use SimpleFin holding ID as external_id with "simplefin_" prefix
- Calculate price from market_value/shares when available
- Store raw holdings payload in simplefin_accounts for complete data retention

This enables better holdings tracking than composite key approach and ensures
we capture all SimpleFin data even if not immediately used in our models.

* Simplify SimpleFin transaction enrichment

- Add MerchantDetector that uses payee field directly for merchant creation
- Enhance SimpleFin entry name generation combining payee + description
- Remove transaction processor category matching logic
- Create dedicated SimpleFin entry processor

Uses SimpleFin's clean payee data without unnecessary filtering.

* Add source field to ProviderMerchant and fix data enrichment

- Add source field to ProviderMerchant model for provider-specific merchant tracking
- Fix DataEnrichment to handle string transaction IDs correctly with to_i conversion

Enables per-provider merchant deduplication and fixes transaction lookups.

* Fix SimpleFin controller parameter handling

- Convert string account_ids to integers for proper account lookup
- Ensure account selection works correctly with form submissions

Fixes account filtering when setting up SimpleFin sync.

* Fix linting issues - auto-corrected whitespace and formatting

* Derive institution domain from URL when missing in SimpleFin items

* Fix render_error to preserve persisted record for edit context

* Add unique index to prevent duplicate holdings

* Fix potential NameError in holdings processor rescue block

* Guard against missing SimpleFin transaction IDs

* Fix SimpleFin amount parsing error handling

Re-raise ArgumentError instead of silently returning BigDecimal("0")
to prevent misleading $0 transactions from invalid amount data.

* Fix SimpleFin chunked import data loss bug

Merge transaction arrays instead of overwriting to prevent data loss
during chunked imports. Preserve most recent holdings data only.

* Add external_id uniqueness validation to Holding model

* Fix holdings cost_basis precision and add external_id unique constraint

* Fix SimpleFin test mock expectations and remove debug statements

- Fixed SimplefinItemsControllerTest by properly mocking Provider::Simplefin
  instead of over-mocking the create_simplefin_item! method
- Removed DEBUG puts statements from SimplefinItem::Importer

* Fix linting issues - auto-corrected whitespace and formatting

---------

Signed-off-by: Himmelschmidt <46351743+Himmelschmidt@users.noreply.github.com>
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2025-10-22 19:51:24 +02:00

231 lines
7.4 KiB
Ruby

class Account < ApplicationRecord
include AASM, Syncable, Monetizable, Chartable, Linkable, Enrichable, Anchorable, Reconcileable
validates :name, :balance, :currency, presence: true
belongs_to :family
belongs_to :import, optional: true
belongs_to :simplefin_account, optional: true
has_many :import_mappings, as: :mappable, dependent: :destroy, class_name: "Import::Mapping"
has_many :entries, dependent: :destroy
has_many :transactions, through: :entries, source: :entryable, source_type: "Transaction"
has_many :valuations, through: :entries, source: :entryable, source_type: "Valuation"
has_many :trades, through: :entries, source: :entryable, source_type: "Trade"
has_many :holdings, dependent: :destroy
has_many :balances, dependent: :destroy
monetize :balance, :cash_balance
enum :classification, { asset: "asset", liability: "liability" }, validate: { allow_nil: true }
scope :visible, -> { where(status: [ "draft", "active" ]) }
scope :assets, -> { where(classification: "asset") }
scope :liabilities, -> { where(classification: "liability") }
scope :alphabetically, -> { order(:name) }
scope :manual, -> { where(plaid_account_id: nil, simplefin_account_id: nil) }
has_one_attached :logo
delegated_type :accountable, types: Accountable::TYPES, dependent: :destroy
delegate :subtype, to: :accountable, allow_nil: true
accepts_nested_attributes_for :accountable, update_only: true
# Account state machine
aasm column: :status, timestamps: true do
state :active, initial: true
state :draft
state :disabled
state :pending_deletion
event :activate do
transitions from: [ :draft, :disabled ], to: :active
end
event :disable do
transitions from: [ :draft, :active ], to: :disabled
end
event :enable do
transitions from: :disabled, to: :active
end
event :mark_for_deletion do
transitions from: [ :draft, :active, :disabled ], to: :pending_deletion
end
end
class << self
def create_and_sync(attributes)
attributes[:accountable_attributes] ||= {} # Ensure accountable is created, even if empty
account = new(attributes.merge(cash_balance: attributes[:balance]))
initial_balance = attributes.dig(:accountable_attributes, :initial_balance)&.to_d
transaction do
account.save!
manager = Account::OpeningBalanceManager.new(account)
result = manager.set_opening_balance(balance: initial_balance || account.balance)
raise result.error if result.error
end
account.sync_later
account
end
def create_from_simplefin_account(simplefin_account, account_type, subtype = nil)
# Get the balance from SimpleFin
balance = simplefin_account.current_balance || simplefin_account.available_balance || 0
# SimpleFin returns negative balances for credit cards (liabilities)
# But Sure expects positive balances for liabilities
if account_type == "CreditCard" || account_type == "Loan"
balance = balance.abs
end
# Calculate cash balance correctly for investment accounts
cash_balance = balance
if account_type == "Investment"
begin
calculator = SimplefinAccount::Investments::BalanceCalculator.new(simplefin_account)
calculated = calculator.cash_balance
cash_balance = calculated unless calculated.nil?
rescue => e
Rails.logger.warn(
"Investment cash_balance calculation failed for " \
"SimpleFin account #{simplefin_account.id}: #{e.class} - #{e.message}"
)
# Fallback to zero as suggested
cash_balance = 0
end
end
attributes = {
family: simplefin_account.simplefin_item.family,
name: simplefin_account.name,
balance: balance,
cash_balance: cash_balance,
currency: simplefin_account.currency,
accountable_type: account_type,
accountable_attributes: build_simplefin_accountable_attributes(simplefin_account, account_type, subtype),
simplefin_account_id: simplefin_account.id
}
create_and_sync(attributes)
end
private
def build_simplefin_accountable_attributes(simplefin_account, account_type, subtype)
attributes = {}
attributes[:subtype] = subtype if subtype.present?
# Set account-type-specific attributes from SimpleFin data
case account_type
when "CreditCard"
# For credit cards, available_balance often represents available credit
if simplefin_account.available_balance.present? && simplefin_account.available_balance > 0
attributes[:available_credit] = simplefin_account.available_balance
end
when "Loan"
# For loans, we might get additional data from the raw_payload
# This is where loan-specific information could be extracted if available
# Currently we don't have specific loan fields from SimpleFin protocol
end
attributes
end
end
def institution_domain
url_string = plaid_account&.plaid_item&.institution_url
return nil unless url_string.present?
begin
uri = URI.parse(url_string)
# Use safe navigation on .host before calling gsub
uri.host&.gsub(/^www\./, "")
rescue URI::InvalidURIError
# Log a warning if the URL is invalid and return nil
Rails.logger.warn("Invalid institution URL encountered for account #{id}: #{url_string}")
nil
end
end
def destroy_later
mark_for_deletion!
DestroyJob.perform_later(self)
end
# Override destroy to handle error recovery for accounts
def destroy
super
rescue => e
# If destruction fails, transition back to disabled state
# This provides a cleaner recovery path than the generic scheduled_for_deletion flag
disable! if may_disable?
raise e
end
def current_holdings
holdings.where(currency: currency)
.where.not(qty: 0)
.where(
id: holdings.select("DISTINCT ON (security_id) id")
.where(currency: currency)
.order(:security_id, date: :desc)
)
.order(amount: :desc)
end
def start_date
first_entry_date = entries.minimum(:date) || Date.current
first_entry_date - 1.day
end
def lock_saved_attributes!
super
accountable.lock_saved_attributes!
end
def first_valuation
entries.valuations.order(:date).first
end
def first_valuation_amount
first_valuation&.amount_money || balance_money
end
# Get short version of the subtype label
def short_subtype_label
accountable_class.short_subtype_label_for(subtype) || accountable_class.display_name
end
# Get long version of the subtype label
def long_subtype_label
accountable_class.long_subtype_label_for(subtype) || accountable_class.display_name
end
# The balance type determines which "component" of balance is being tracked.
# This is primarily used for balance related calculations and updates.
#
# "Cash" = "Liquid"
# "Non-cash" = "Illiquid"
# "Investment" = A mix of both, including brokerage cash (liquid) and holdings (illiquid)
def balance_type
case accountable_type
when "Depository", "CreditCard"
:cash
when "Property", "Vehicle", "OtherAsset", "Loan", "OtherLiability"
:non_cash
when "Investment", "Crypto"
:investment
else
raise "Unknown account type: #{accountable_type}"
end
end
end