mirror of
https://github.com/we-promise/sure.git
synced 2026-09-05 14:51:15 +00:00
Abhinav Dhiman
e3d46021c2
* perf(sync): reduce per-job memory peak in Balance/Holding materialization
Profiling of SyncJob (StackProf object mode + Sidekiq memory middleware)
showed peaks of 600k-1.1M live heap slots per job and ~196k retained
ActiveModel::Attribute::FromUser objects post-GC, driven by full-history
in-memory accumulation in the balance/holding sync pipeline.
Changes:
- Replace Holding.new / Balance.new in calculators with lightweight
Struct-based HoldingData / BalanceData. Skips AR attribute sets,
belongs_to proxies, dirty tracking, type casting, and callbacks
that were never used (upsert_all bypasses validations/callbacks
anyway). Eliminates ~30% of allocations and the bulk of retained
ActiveModel::Attribute::* instances.
- Build upsert payloads directly from struct fields instead of
Holding/Balance#attributes.slice(...).
- Batch upsert_all in PERSIST_BATCH_SIZE (2,000) slices in both
Balance::Materializer and Holding::Materializer so the intermediate
attribute-hash array is bounded instead of holding the full
multi-year history alongside the calculator output.
- Replace account.holdings.reload with account.holdings.reset in
Holding::Materializer. Same cache invalidation, no eager re-query;
the next consumer (Balance::SyncCache) loads on demand.
- Mutate entries in place in Balance::SyncCache#converted_entries
instead of Entry#dup. The instances are scoped to the throwaway
sync-cache and never persisted, so dup'ing was producing tens of
thousands of unused FromUser/FromDatabase attribute wrappers per
sync.
All persist paths run inside the existing Balance.transaction wrapper,
so batched upserts retain transactional atomicity. No production caller
of Balance::SyncCache or Holding::Materializer reuses the affected
instances outside the materializer's lifetime.
Test coverage: balance/{sync_cache,materializer,forward_calculator,
reverse_calculator} and holding/{materializer,forward_calculator,
reverse_calculator} plus account/syncer and sync (82 runs, 2,548
assertions, 0 failures).
* refactor(holding): stream materializer upserts to bound peak memory
Replace full-array accumulation + each_slice in Materializer#persist_holdings
with two flush-on-fill buffers (holdings_buffer_to_upsert_with_cost /
holdings_buffer_to_upsert_without_cost) that upsert and clear at
PERSIST_BATCH_SIZE, keeping peak RSS bounded to ~2x batch size.
Also add assert_not_nil guards in ReverseCalculatorTest before
dereferencing calculated.find results to surface clear failures
instead of NoMethodError.
* refactor(balance): promote BalanceData to Balance namespace and document sync mutation safety
- Extract Balance::BalanceData struct into its own file (app/models/balance/balance_data.rb)
so it is discoverable without knowing it lived inside BaseCalculator
- Remove inline Struct definition from Balance::BaseCalculator; update build_balance
to reference Balance::BalanceData explicitly (required because class Foo::Bar syntax
does not nest Foo in constant lookup)
- Add comment to SyncCache#converted_entries clarifying that to_a materialises
independent AR instances with no identity map active, making in-place mutation safe
- Update all Balance::BaseCalculator::BalanceData references in materializer_test
* test(balance): use to_h instead of attributes on BalanceData struct in waypoint test
* perf(balance,market_data): replace sort_by + first/last with minmax_by and push account-entry join to SQL
- Balance::Materializer#purge_stale_balances: replace sort_by(&:date) + first/last with minmax_by(&:date) to avoid full sort when only min/max are needed
- MarketDataImporter: replace Entry.group(:account_id).minimum(:date) (which loads all account IDs into a Hash) with a LEFT JOIN subquery that computes MIN(date) per account in SQL and exposes it as first_entry_date on the Account relation
* test(balance): use BalanceData struct in materializer purge test
* fix(holding): carry cost_basis forward onto gap-filled dates
58 lines
1.9 KiB
Ruby
58 lines
1.9 KiB
Ruby
class Balance::SyncCache
|
|
def initialize(account)
|
|
@account = account
|
|
end
|
|
|
|
def get_valuation(date)
|
|
entries_by_date[date]&.find { |e| e.valuation? }
|
|
end
|
|
|
|
def get_holdings_value(date)
|
|
holdings_value_by_date[date] || 0
|
|
end
|
|
|
|
def get_entries(date)
|
|
entries_by_date[date]&.select { |e| e.transaction? || e.trade? } || []
|
|
end
|
|
|
|
private
|
|
attr_reader :account
|
|
|
|
def entries_by_date
|
|
@entries_by_date ||= converted_entries.group_by(&:date)
|
|
end
|
|
|
|
def holdings_value_by_date
|
|
@holdings_value_by_date ||= account.holdings.each_with_object(Hash.new(0)) do |h, totals|
|
|
begin
|
|
converted = Money.new(h.amount, h.currency).exchange_to(account.currency, date: h.date).amount
|
|
rescue Money::ConversionError
|
|
converted = h.amount # fallback to 1:1 conversion rate if exchange rate unavailable
|
|
end
|
|
totals[h.date] += converted
|
|
end
|
|
end
|
|
|
|
def converted_entries
|
|
@converted_entries ||= account.entries.excluding_split_parents.includes(:entryable).order(:date).to_a.map do |e|
|
|
custom_rate = e.entryable.exchange_rate if e.entryable.respond_to?(:exchange_rate)
|
|
|
|
# Use Money#exchange_to with custom rate if available, standard lookup otherwise.
|
|
# Mutate the entry in place rather than dup'ing — these instances are scoped to
|
|
# this sync-cache only and never persisted, so avoiding the dup eliminates a
|
|
# large amount of ActiveModel::Attribute allocations during sync.
|
|
# to_a materializes independent instances; no AR identity map is active during sync,
|
|
# so callers holding a reference to the same association will never see these mutations.
|
|
new_amount = e.amount_money.exchange_to(
|
|
account.currency,
|
|
date: e.date,
|
|
custom_rate: custom_rate
|
|
).amount
|
|
|
|
e.amount = new_amount
|
|
e.currency = account.currency
|
|
e
|
|
end
|
|
end
|
|
end
|