* perf(sync): reduce per-job memory peak in Balance/Holding materialization
Profiling of SyncJob (StackProf object mode + Sidekiq memory middleware)
showed peaks of 600k-1.1M live heap slots per job and ~196k retained
ActiveModel::Attribute::FromUser objects post-GC, driven by full-history
in-memory accumulation in the balance/holding sync pipeline.
Changes:
- Replace Holding.new / Balance.new in calculators with lightweight
Struct-based HoldingData / BalanceData. Skips AR attribute sets,
belongs_to proxies, dirty tracking, type casting, and callbacks
that were never used (upsert_all bypasses validations/callbacks
anyway). Eliminates ~30% of allocations and the bulk of retained
ActiveModel::Attribute::* instances.
- Build upsert payloads directly from struct fields instead of
Holding/Balance#attributes.slice(...).
- Batch upsert_all in PERSIST_BATCH_SIZE (2,000) slices in both
Balance::Materializer and Holding::Materializer so the intermediate
attribute-hash array is bounded instead of holding the full
multi-year history alongside the calculator output.
- Replace account.holdings.reload with account.holdings.reset in
Holding::Materializer. Same cache invalidation, no eager re-query;
the next consumer (Balance::SyncCache) loads on demand.
- Mutate entries in place in Balance::SyncCache#converted_entries
instead of Entry#dup. The instances are scoped to the throwaway
sync-cache and never persisted, so dup'ing was producing tens of
thousands of unused FromUser/FromDatabase attribute wrappers per
sync.
All persist paths run inside the existing Balance.transaction wrapper,
so batched upserts retain transactional atomicity. No production caller
of Balance::SyncCache or Holding::Materializer reuses the affected
instances outside the materializer's lifetime.
Test coverage: balance/{sync_cache,materializer,forward_calculator,
reverse_calculator} and holding/{materializer,forward_calculator,
reverse_calculator} plus account/syncer and sync (82 runs, 2,548
assertions, 0 failures).
* refactor(holding): stream materializer upserts to bound peak memory
Replace full-array accumulation + each_slice in Materializer#persist_holdings
with two flush-on-fill buffers (holdings_buffer_to_upsert_with_cost /
holdings_buffer_to_upsert_without_cost) that upsert and clear at
PERSIST_BATCH_SIZE, keeping peak RSS bounded to ~2x batch size.
Also add assert_not_nil guards in ReverseCalculatorTest before
dereferencing calculated.find results to surface clear failures
instead of NoMethodError.
* refactor(balance): promote BalanceData to Balance namespace and document sync mutation safety
- Extract Balance::BalanceData struct into its own file (app/models/balance/balance_data.rb)
so it is discoverable without knowing it lived inside BaseCalculator
- Remove inline Struct definition from Balance::BaseCalculator; update build_balance
to reference Balance::BalanceData explicitly (required because class Foo::Bar syntax
does not nest Foo in constant lookup)
- Add comment to SyncCache#converted_entries clarifying that to_a materialises
independent AR instances with no identity map active, making in-place mutation safe
- Update all Balance::BaseCalculator::BalanceData references in materializer_test
* test(balance): use to_h instead of attributes on BalanceData struct in waypoint test
* perf(balance,market_data): replace sort_by + first/last with minmax_by and push account-entry join to SQL
- Balance::Materializer#purge_stale_balances: replace sort_by(&:date) + first/last with minmax_by(&:date) to avoid full sort when only min/max are needed
- MarketDataImporter: replace Entry.group(:account_id).minimum(:date) (which loads all account IDs into a Hash) with a LEFT JOIN subquery that computes MIN(date) per account in SQL and exposes it as first_entry_date on the Account relation
* test(balance): use BalanceData struct in materializer purge test
* fix(holding): carry cost_basis forward onto gap-filled dates
Income trades created via Trade::CreateForm#create_income_trade set
price: 0 on the Trade record. This zero price was ingested by
PortfolioCache#load_prices as a valid price source, causing
ForwardCalculator#build_holdings to compute holding amount = qty * 0 = $0,
and overriding the security's market price on that date.
Additionally, Balance::BaseCalculator#flows_for_date misclassified income
trades (qty=0) as sells, producing spurious non_cash_outflows.
Fixes:
- PortfolioCache#load_prices: filter out zero-price trades from trade
price sources using .select { |t| t.entryable.price&.positive? }
- Balance::BaseCalculator#flows_for_date: separate income trades from
regular trades using their qty (income trades have qty=0), so they
contribute only to cash flows, not non-cash flows
- Balance::SyncCache#converted_entries: preserve the entryable association
target on duped entries so that e.entryable.qty access doesn't N+1
Fixes#2672
* Performance improvements in balance sync cache
Balance::SyncCache#converted_holdings called account.holdings.map { |h| h.dup }
which duplicated every holding record into a new ActiveRecord object, converted
its currency, and stored the full object in a holdings_by_date array hash.
For an investment account with years of history this allocates 100,000+
AR objects on every sync - one per holding row - creating proportional GC
pressure that scaled with account age.
The only consumer of get_holdings(date) was BaseCalculator#holdings_value_for_date,
which immediately discarded the objects after calling .sum(&:amount). The
individual holding objects were never accessed for any other attribute.
Replace the dup-and-group approach with a single aggregation pass that stores
only the per-date sum:
holdings_value_by_date: account.holdings.each_with_object(Hash.new(0)) do |h, totals|
converted = Money.new(h.amount, h.currency).exchange_to(account.currency, date: h.date).amount
totals[h.date] += converted
end
Interface change: get_holdings(date) -> get_holdings_value(date) returns a
Numeric directly rather than an Array. BaseCalculator#holdings_value_for_date
is updated accordingly, and its own per-date memoization layer is removed
since holdings_value_by_date is already fully memoized at the SyncCache level.
* fall back to 1:1 rate in SyncCache when holding exchange rate is missing; update tests to use investment class
* Perf: Index Balance::SyncCache lookups by date to eliminate O(N×D) scans
Each call to get_holdings(date) and get_entries(date) previously did a
linear scan over the full converted_holdings / converted_entries arrays.
The balance calculators call these once per day across the full account
history, making the overall complexity O(N×D) where N is the total number
of holding/entry rows and D is the number of days in the account history.
For a typical investment account (20 securities, 2 years of history):
- Holdings: 20 × 730 = 14,600 rows
- Balance loop: 730 date iterations
- Comparisons: 14,600 × 730 ≈ 10.7 million per materialise run
This change builds a hash index (grouped by date) once on first access and
reuses it for all subsequent lookups, reducing per-call complexity to O(1).
Total complexity becomes O(N) — load once, look up cheaply.
Observed wall-clock improvement on a real account: ~36 s → ~5 s for a full
Balance::Materializer run. The nightly sync benefits equally.
No behavioural change: get_holdings, get_entries, and get_valuation return
identical data — they are now just fetched via a hash key rather than a
repeated array scan.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: Return defensive copy from get_holdings to prevent cache mutation
get_holdings was returning a direct reference to the internal cached
array from holdings_by_date. A caller appending to the result (e.g.
via <<) would silently corrupt the cache for all subsequent date
lookups in the same materialise run.
Use &.dup to return a shallow copy of the group array. Callers only
read from the result (sum, map, etc.) so this has no behavioural
impact and negligible performance cost.
get_entries is already safe — Array#select always returns a new array.
get_valuation returns a single object, not an array, so no issue there.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Remove unnecessary dup in get_holdings for consistency
No caller mutates the returned array (only .sum is called), so the
defensive copy is unnecessary overhead. This aligns get_holdings with
get_entries and get_valuation which also return cached references directly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* Initial split transaction support
* Add support to unsplit and edit split
* Update show.html.erb
* FIX address reviews
* Improve UX
* Update show.html.erb
* Reviews
* Update edit.html.erb
* Add parent category to dialog
* Update en.yml
* Add UI indication to totals
* FIX ui update
* Add category select like rest of app
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
Abhinav Dhiman