* perf(sync): reduce per-job memory peak in Balance/Holding materialization
Profiling of SyncJob (StackProf object mode + Sidekiq memory middleware)
showed peaks of 600k-1.1M live heap slots per job and ~196k retained
ActiveModel::Attribute::FromUser objects post-GC, driven by full-history
in-memory accumulation in the balance/holding sync pipeline.
Changes:
- Replace Holding.new / Balance.new in calculators with lightweight
Struct-based HoldingData / BalanceData. Skips AR attribute sets,
belongs_to proxies, dirty tracking, type casting, and callbacks
that were never used (upsert_all bypasses validations/callbacks
anyway). Eliminates ~30% of allocations and the bulk of retained
ActiveModel::Attribute::* instances.
- Build upsert payloads directly from struct fields instead of
Holding/Balance#attributes.slice(...).
- Batch upsert_all in PERSIST_BATCH_SIZE (2,000) slices in both
Balance::Materializer and Holding::Materializer so the intermediate
attribute-hash array is bounded instead of holding the full
multi-year history alongside the calculator output.
- Replace account.holdings.reload with account.holdings.reset in
Holding::Materializer. Same cache invalidation, no eager re-query;
the next consumer (Balance::SyncCache) loads on demand.
- Mutate entries in place in Balance::SyncCache#converted_entries
instead of Entry#dup. The instances are scoped to the throwaway
sync-cache and never persisted, so dup'ing was producing tens of
thousands of unused FromUser/FromDatabase attribute wrappers per
sync.
All persist paths run inside the existing Balance.transaction wrapper,
so batched upserts retain transactional atomicity. No production caller
of Balance::SyncCache or Holding::Materializer reuses the affected
instances outside the materializer's lifetime.
Test coverage: balance/{sync_cache,materializer,forward_calculator,
reverse_calculator} and holding/{materializer,forward_calculator,
reverse_calculator} plus account/syncer and sync (82 runs, 2,548
assertions, 0 failures).
* refactor(holding): stream materializer upserts to bound peak memory
Replace full-array accumulation + each_slice in Materializer#persist_holdings
with two flush-on-fill buffers (holdings_buffer_to_upsert_with_cost /
holdings_buffer_to_upsert_without_cost) that upsert and clear at
PERSIST_BATCH_SIZE, keeping peak RSS bounded to ~2x batch size.
Also add assert_not_nil guards in ReverseCalculatorTest before
dereferencing calculated.find results to surface clear failures
instead of NoMethodError.
* refactor(balance): promote BalanceData to Balance namespace and document sync mutation safety
- Extract Balance::BalanceData struct into its own file (app/models/balance/balance_data.rb)
so it is discoverable without knowing it lived inside BaseCalculator
- Remove inline Struct definition from Balance::BaseCalculator; update build_balance
to reference Balance::BalanceData explicitly (required because class Foo::Bar syntax
does not nest Foo in constant lookup)
- Add comment to SyncCache#converted_entries clarifying that to_a materialises
independent AR instances with no identity map active, making in-place mutation safe
- Update all Balance::BaseCalculator::BalanceData references in materializer_test
* test(balance): use to_h instead of attributes on BalanceData struct in waypoint test
* perf(balance,market_data): replace sort_by + first/last with minmax_by and push account-entry join to SQL
- Balance::Materializer#purge_stale_balances: replace sort_by(&:date) + first/last with minmax_by(&:date) to avoid full sort when only min/max are needed
- MarketDataImporter: replace Entry.group(:account_id).minimum(:date) (which loads all account IDs into a Hash) with a LEFT JOIN subquery that computes MIN(date) per account in SQL and exposes it as first_entry_date on the Account relation
* test(balance): use BalanceData struct in materializer purge test
* fix(holding): carry cost_basis forward onto gap-filled dates
* feat(balance): incremental ForwardCalculator — only recalculate from changed date forward
When a Sync record carries a window_start_date, ForwardCalculator now
seeds its starting balances from the persisted DB balance for
window_start_date - 1, then iterates only from window_start_date to
calc_end_date. This avoids recomputing every daily balance on a
long-lived account when a single transaction changes.
Key changes:
- Account::Syncer passes sync.window_start_date to Balance::Materializer
- Balance::Materializer accepts window_start_date and forwards it to
ForwardCalculator; purge_stale_balances uses opening_anchor_date as the
lower bound in incremental mode so pre-window balances are not deleted
- Balance::ForwardCalculator accepts window_start_date; resolve_starting_balances
loads end_cash_balance/end_non_cash_balance from the prior DB record and
falls back to full recalculation when no prior record exists
- Tests added for incremental correctness, fallback behaviour, and purge safety
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
# Conflicts:
# app/models/balance/materializer.rb
* Enhance fallback logic on ForwardCalculator and Materializer
* fix(balance): address CodeRabbit review issues on incremental ForwardCalculator
- materializer.rb: handle empty sorted_balances in incremental mode by still
purging stale tail balances beyond window_start_date - 1, preventing orphaned
future rows when a transaction is deleted and the recalc window produces no rows
- materializer_test.rb: stub incremental? alongside calculate in the incremental
sync test so the guard in ForwardCalculator#incremental? doesn't raise when
@fell_back is nil (never set because calculate was stubbed out)
- materializer_test.rb: correct window_start_date in the fallback test from
3.days.ago to 2.days.ago so window_start_date - 1 hits a date with no
persisted balance, correctly triggering full recalculation instead of
accidentally seeding from the stale wrong_pre_window balance
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(balance): multi-currency fallback to full recalculation and add corresponding tests
* address coderabbit comment about test
* Make the foreign-currency precondition explicit in the test setup.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Feat: Implement manual sync prices functionality and enhance holdings display
* Feat: Enhance sync prices functionality with error handling and update UI components
* Feat: Update sync prices error handling and enhance Spanish locale messages
* Fix: Address CodeRabbit review feedback
- Set fallback @provider_error when prices_updated == 0 so turbo stream
never fails silently without a visible error message
- Move attr_reader :provider_error to class header in Price::Importer
for conventional placement alongside other attribute declarations
- Precompute @last_price_updated in controller (show + sync_prices)
instead of running a DB query directly inside ERB templates
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: Replace bare rescue with explicit exception handling in turbo stream view
Bare `rescue` silently swallows all exceptions, making debugging impossible.
Match the pattern already used in show.html.erb: rescue ActiveRecord::RecordInvalid
explicitly, then catch StandardError with logging (message + backtrace) before
falling back to the unknown label.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: Update test assertion to expect actual provider error message
The stub returns "Yahoo Finance rate limit exceeded" as the provider error.
After the @provider_error fallback fix, the controller now correctly surfaces
the real provider error when present (using .presence || fallback), so the
flash[:alert] is the actual error string, not the generic fallback.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: Assert scoped security_ids in sync_prices materializer test
Replace loose stub with constructor expectation to verify that
Balance::Materializer is instantiated with the single-security scope.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix: Assert holding remap in remap_security test
Add assertion that @holding.security_id is updated to the target
security after remap, covering the core command outcome.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix: CI test failure - Update disconnect external assistant test to use env overrides
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Balance reconcilations with new components
* Fix materializer and test assumptions
* Fix investment valuation calculations and recon display
* Lint fixes
* Balance series uses new component fields
Abhinav Dhiman