QT/sure
1
0
Fork 0
mirror of https://github.com/we-promise/sure.git synced 2026-09-05 14:51:15 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
fd6f4ff078ea30751069e2de59f4fbdf2510c57a
sure/test/models/holding/reverse_calculator_test.rb
T
Abhinav Dhiman e3d46021c2 fix: memory leak in sidekiq (#1940) 
* perf(sync): reduce per-job memory peak in Balance/Holding materialization

Profiling of SyncJob (StackProf object mode + Sidekiq memory middleware)
showed peaks of 600k-1.1M live heap slots per job and ~196k retained
ActiveModel::Attribute::FromUser objects post-GC, driven by full-history
in-memory accumulation in the balance/holding sync pipeline.

Changes:
- Replace Holding.new / Balance.new in calculators with lightweight
  Struct-based HoldingData / BalanceData. Skips AR attribute sets,
  belongs_to proxies, dirty tracking, type casting, and callbacks
  that were never used (upsert_all bypasses validations/callbacks
  anyway). Eliminates ~30% of allocations and the bulk of retained
  ActiveModel::Attribute::* instances.
- Build upsert payloads directly from struct fields instead of
  Holding/Balance#attributes.slice(...).
- Batch upsert_all in PERSIST_BATCH_SIZE (2,000) slices in both
  Balance::Materializer and Holding::Materializer so the intermediate
  attribute-hash array is bounded instead of holding the full
  multi-year history alongside the calculator output.
- Replace account.holdings.reload with account.holdings.reset in
  Holding::Materializer. Same cache invalidation, no eager re-query;
  the next consumer (Balance::SyncCache) loads on demand.
- Mutate entries in place in Balance::SyncCache#converted_entries
  instead of Entry#dup. The instances are scoped to the throwaway
  sync-cache and never persisted, so dup'ing was producing tens of
  thousands of unused FromUser/FromDatabase attribute wrappers per
  sync.

All persist paths run inside the existing Balance.transaction wrapper,
so batched upserts retain transactional atomicity. No production caller
of Balance::SyncCache or Holding::Materializer reuses the affected
instances outside the materializer's lifetime.

Test coverage: balance/{sync_cache,materializer,forward_calculator,
reverse_calculator} and holding/{materializer,forward_calculator,
reverse_calculator} plus account/syncer and sync (82 runs, 2,548
assertions, 0 failures).

* refactor(holding): stream materializer upserts to bound peak memory

Replace full-array accumulation + each_slice in Materializer#persist_holdings
with two flush-on-fill buffers (holdings_buffer_to_upsert_with_cost /
holdings_buffer_to_upsert_without_cost) that upsert and clear at
PERSIST_BATCH_SIZE, keeping peak RSS bounded to ~2x batch size.

Also add assert_not_nil guards in ReverseCalculatorTest before
dereferencing calculated.find results to surface clear failures
instead of NoMethodError.

* refactor(balance): promote BalanceData to Balance namespace and document sync mutation safety

- Extract Balance::BalanceData struct into its own file (app/models/balance/balance_data.rb)
  so it is discoverable without knowing it lived inside BaseCalculator
- Remove inline Struct definition from Balance::BaseCalculator; update build_balance
  to reference Balance::BalanceData explicitly (required because class Foo::Bar syntax
  does not nest Foo in constant lookup)
- Add comment to SyncCache#converted_entries clarifying that to_a materialises
  independent AR instances with no identity map active, making in-place mutation safe
- Update all Balance::BaseCalculator::BalanceData references in materializer_test

* test(balance): use to_h instead of attributes on BalanceData struct in waypoint test

* perf(balance,market_data): replace sort_by + first/last with minmax_by and push account-entry join to SQL

- Balance::Materializer#purge_stale_balances: replace sort_by(&:date) + first/last with minmax_by(&:date) to avoid full sort when only min/max are needed
- MarketDataImporter: replace Entry.group(:account_id).minimum(:date) (which loads all account IDs into a Hash) with a LEFT JOIN subquery that computes MIN(date) per account in SQL and exposes it as first_entry_date on the Account relation

* test(balance): use BalanceData struct in materializer purge test

* fix(holding): carry cost_basis forward onto gap-filled dates
2026-08-20 06:41:42 +02:00

322 lines
15 KiB
Ruby
Raw Blame History

require "test_helper"
class Holding::ReverseCalculatorTest < ActiveSupport::TestCase
include EntriesTestHelper
setup do
@account = families(:empty).accounts.create!(
name: "Test",
balance: 20000,
cash_balance: 20000,
currency: "USD",
accountable: Investment.new
)
end
test "no holdings" do
empty_snapshot = OpenStruct.new(to_h: {})
calculated = Holding::ReverseCalculator.new(@account, portfolio_snapshot: empty_snapshot).calculate
assert_equal [], calculated
end
test "holding generation respects user timezone and last generated date is current user date" do
# Simulate user in EST timezone
Time.use_zone("America/New_York") do
# Set current time to 1am UTC on Jan 5, 2025
# This would be 8pm EST on Jan 4, 2025 (user's time, and the last date we should generate holdings for)
travel_to Time.utc(2025, 01, 05, 1, 0, 0)
voo = Security.create!(ticker: "VOO", name: "Vanguard S&P 500 ETF")
Security::Price.create!(security: voo, date: "2025-01-02", price: 500)
Security::Price.create!(security: voo, date: "2025-01-03", price: 500)
Security::Price.create!(security: voo, date: "2025-01-04", price: 500)
# Today's holdings (provided)
@account.holdings.create!(security: voo, date: "2025-01-04", qty: 10, price: 500, amount: 5000, currency: "USD")
create_trade(voo, qty: 10, date: "2025-01-03", price: 500, account: @account)
expected = [ [ "2025-01-02", 0 ], [ "2025-01-03", 5000 ], [ "2025-01-04", 5000 ] ]
# Mock snapshot with the holdings we created
snapshot = OpenStruct.new(to_h: { voo.id => 10 })
calculated = Holding::ReverseCalculator.new(@account, portfolio_snapshot: snapshot).calculate
assert_equal expected, calculated.sort_by(&:date).map { |b| [ b.date.to_s, b.amount ] }
end
end
# Should be able to handle this case, although we should not be reverse-syncing an account without provided current day holdings
test "reverse portfolio with trades but without current day holdings" do
voo = Security.create!(ticker: "VOO", name: "Vanguard S&P 500 ETF")
Security::Price.create!(security: voo, date: Date.current, price: 470)
Security::Price.create!(security: voo, date: 1.day.ago.to_date, price: 470)
create_trade(voo, qty: -10, date: Date.current, price: 470, account: @account)
# Mock empty portfolio since no current day holdings
snapshot = OpenStruct.new(to_h: { voo.id => 0 })
calculated = Holding::ReverseCalculator.new(@account, portfolio_snapshot: snapshot).calculate
assert_equal 2, calculated.length
end
test "reverse portfolio calculation" do
load_today_portfolio
# Build up to 10 shares of VOO (current value $5000)
create_trade(@voo, qty: 20, date: 3.days.ago.to_date, price: 470, account: @account)
create_trade(@voo, qty: -15, date: 2.days.ago.to_date, price: 480, account: @account)
create_trade(@voo, qty: 5, date: 1.day.ago.to_date, price: 490, account: @account)
# Amazon won't exist in current holdings because qty is zero, but should show up in historical portfolio
create_trade(@amzn, qty: 1, date: 2.days.ago.to_date, price: 200, account: @account)
create_trade(@amzn, qty: -1, date: 1.day.ago.to_date, price: 200, account: @account)
# Build up to 100 shares of WMT (current value $10000)
create_trade(@wmt, qty: 100, date: 1.day.ago.to_date, price: 100, account: @account)
expected = [
# 4 days ago
Holding.new(security: @voo, date: 4.days.ago.to_date, qty: 0, price: 460, amount: 0),
Holding.new(security: @wmt, date: 4.days.ago.to_date, qty: 0, price: 100, amount: 0),
Holding.new(security: @amzn, date: 4.days.ago.to_date, qty: 0, price: 200, amount: 0),
# 3 days ago
Holding.new(security: @voo, date: 3.days.ago.to_date, qty: 20, price: 470, amount: 9400),
Holding.new(security: @wmt, date: 3.days.ago.to_date, qty: 0, price: 100, amount: 0),
Holding.new(security: @amzn, date: 3.days.ago.to_date, qty: 0, price: 200, amount: 0),
# 2 days ago
Holding.new(security: @voo, date: 2.days.ago.to_date, qty: 5, price: 480, amount: 2400),
Holding.new(security: @wmt, date: 2.days.ago.to_date, qty: 0, price: 100, amount: 0),
Holding.new(security: @amzn, date: 2.days.ago.to_date, qty: 1, price: 200, amount: 200),
# 1 day ago
Holding.new(security: @voo, date: 1.day.ago.to_date, qty: 10, price: 490, amount: 4900),
Holding.new(security: @wmt, date: 1.day.ago.to_date, qty: 100, price: 100, amount: 10000),
Holding.new(security: @amzn, date: 1.day.ago.to_date, qty: 0, price: 200, amount: 0),
# Today
Holding.new(security: @voo, date: Date.current, qty: 10, price: 500, amount: 5000),
Holding.new(security: @wmt, date: Date.current, qty: 100, price: 100, amount: 10000),
Holding.new(security: @amzn, date: Date.current, qty: 0, price: 200, amount: 0)
]
# Mock snapshot with today's portfolio from load_today_portfolio
snapshot = OpenStruct.new(to_h: { @voo.id => 10, @wmt.id => 100, @amzn.id => 0 })
calculated = Holding::ReverseCalculator.new(@account, portfolio_snapshot: snapshot).calculate
assert_equal expected.length, calculated.length
expected.each do |expected_entry|
calculated_entry = calculated.find { |c| c.security_id == expected_entry.security_id && c.date == expected_entry.date }
assert_not_nil calculated_entry, "No calculated entry for security_id=#{expected_entry.security_id} on #{expected_entry.date}"
assert_equal expected_entry.qty, calculated_entry.qty, "Qty mismatch for security_id=#{expected_entry.security_id} on #{expected_entry.date}"
assert_equal expected_entry.price, calculated_entry.price, "Price mismatch for security_id=#{expected_entry.security_id} on #{expected_entry.date}"
assert_equal expected_entry.amount, calculated_entry.amount, "Amount mismatch for security_id=#{expected_entry.security_id} on #{expected_entry.date}"
end
end
# For a reverse sync, Plaid will provide today's holdings + prices. We need to match those exactly so balances match in net worth rollups.
test "current day holdings always match provided holdings and prices" do
# Provider gives us total value of $10,000 ($5,000 cash, $5,000 in holdings)
@account.update!(balance: 10000, cash_balance: 5000)
wmt = Security.create!(ticker: "WMT", name: "Walmart Inc.")
create_trade(wmt, qty: 50, date: 1.day.ago.to_date, price: 98, account: @account)
@account.holdings.create!(
date: Date.current,
price: 100,
qty: 50,
amount: 5000,
currency: "USD",
security: wmt
)
Security::Price.create!(security: wmt, date: Date.current, price: 102) # This price should be ignored on current day
Security::Price.create!(security: wmt, date: 1.day.ago, price: 98) # This price will be used for historical holding calculation
Security::Price.create!(security: wmt, date: 2.days.ago, price: 95) # This price will be used for historical holding calculation
expected = [
Holding.new(security: wmt, date: 2.days.ago.to_date, qty: 0, price: 95, amount: 0), # Uses market price, empty holding
Holding.new(security: wmt, date: 1.day.ago.to_date, qty: 50, price: 98, amount: 4900), # Uses market price
Holding.new(security: wmt, date: Date.current, qty: 50, price: 100, amount: 5000) # Uses holding price, not market price
]
# Mock snapshot with WMT holding from the test setup
snapshot = OpenStruct.new(to_h: { wmt.id => 50 })
calculated = Holding::ReverseCalculator.new(@account, portfolio_snapshot: snapshot).calculate
assert_equal expected.length, calculated.length
expected.each do |expected_entry|
calculated_entry = calculated.find { |c| c.security_id == expected_entry.security_id && c.date == expected_entry.date }
assert_not_nil calculated_entry, "No calculated entry for security_id=#{expected_entry.security_id} on #{expected_entry.date}"
assert_equal expected_entry.qty, calculated_entry.qty, "Qty mismatch for security_id=#{expected_entry.security_id} on #{expected_entry.date}"
assert_equal expected_entry.price, calculated_entry.price, "Price mismatch for security_id=#{expected_entry.security_id} on #{expected_entry.date}"
assert_equal expected_entry.amount, calculated_entry.amount, "Amount mismatch for security_id=#{expected_entry.security_id} on #{expected_entry.date}"
end
end
# cost_basis_for
test "cost_basis_for returns nil when there are no buy trades" do
security = Security.create!(ticker: "TST", name: "Test")
calc = calculator_with_trades(security)
assert_nil cost_basis_for(calc, security, Date.current)
end
test "cost_basis_for returns nil for dates before the first buy" do
security = Security.create!(ticker: "TST", name: "Test")
buy_date = 5.days.ago.to_date
calc = calculator_with_trades(security) do
create_trade(security, account: @account, qty: 10, price: 100, date: buy_date)
end
assert_nil cost_basis_for(calc, security, buy_date - 1)
end
test "cost_basis_for returns weighted average cost on buy date" do
security = Security.create!(ticker: "TST", name: "Test")
buy_date = 5.days.ago.to_date
calc = calculator_with_trades(security) do
create_trade(security, account: @account, qty: 10, price: 100, date: buy_date)
end
assert_in_delta 100.0, cost_basis_for(calc, security, buy_date).to_f, 1e-6
end
test "cost_basis_for carries forward to dates between buys" do
security = Security.create!(ticker: "TST", name: "Test")
first_buy = 10.days.ago.to_date
second_buy = 3.days.ago.to_date
calc = calculator_with_trades(security) do
create_trade(security, account: @account, qty: 10, price: 100, date: first_buy)
create_trade(security, account: @account, qty: 5, price: 130, date: second_buy)
end
# Between the two buys, cost basis is from the first buy only
assert_in_delta 100.0, cost_basis_for(calc, security, first_buy + 1).to_f, 1e-6
assert_in_delta 100.0, cost_basis_for(calc, security, second_buy - 1).to_f, 1e-6
# After second buy: WAC = (10*100 + 5*130) / 15 = 110.0
assert_in_delta 110.0, cost_basis_for(calc, security, second_buy).to_f, 1e-6
assert_in_delta 110.0, cost_basis_for(calc, security, Date.current).to_f, 1e-6
end
test "cost_basis_for accumulates multiple buys on the same date" do
security = Security.create!(ticker: "TST", name: "Test")
buy_date = 5.days.ago.to_date
calc = calculator_with_trades(security) do
create_trade(security, account: @account, qty: 10, price: 100, date: buy_date)
create_trade(security, account: @account, qty: 5, price: 130, date: buy_date)
end
# WAC = (10*100 + 5*130) / 15 = 110.0 — not the intermediate value after only the first trade
assert_in_delta 110.0, cost_basis_for(calc, security, buy_date).to_f, 1e-6
end
test "cost_basis_for ignores sell trades" do
security = Security.create!(ticker: "TST", name: "Test")
buy_date = 10.days.ago.to_date
sell_date = 5.days.ago.to_date
calc = calculator_with_trades(security) do
create_trade(security, account: @account, qty: 10, price: 100, date: buy_date)
create_trade(security, account: @account, qty: -5, price: 120, date: sell_date)
end
# Sell does not change cost basis
assert_in_delta 100.0, cost_basis_for(calc, security, sell_date).to_f, 1e-6
assert_in_delta 100.0, cost_basis_for(calc, security, Date.current).to_f, 1e-6
end
private
def assert_holdings(expected, calculated)
expected.each do |expected_entry|
calculated_entry = calculated.find { |c| c.security_id == expected_entry.security_id && c.date == expected_entry.date }
assert_not_nil calculated_entry, "No calculated entry for security_id=#{expected_entry.security_id} on #{expected_entry.date}"
assert_equal expected_entry.qty, calculated_entry.qty, "Qty mismatch for security_id=#{expected_entry.security_id} on #{expected_entry.date}"
assert_equal expected_entry.price, calculated_entry.price, "Price mismatch for security_id=#{expected_entry.security_id} on #{expected_entry.date}"
assert_equal expected_entry.amount, calculated_entry.amount, "Amount mismatch for security_id=#{expected_entry.security_id} on #{expected_entry.date}"
end
end
def load_prices
@voo = Security.create!(ticker: "VOO", name: "Vanguard S&P 500 ETF")
Security::Price.create!(security: @voo, date: 4.days.ago.to_date, price: 460)
Security::Price.create!(security: @voo, date: 3.days.ago.to_date, price: 470)
Security::Price.create!(security: @voo, date: 2.days.ago.to_date, price: 480)
Security::Price.create!(security: @voo, date: 1.day.ago.to_date, price: 490)
Security::Price.create!(security: @voo, date: Date.current, price: 500)
@wmt = Security.create!(ticker: "WMT", name: "Walmart Inc.")
Security::Price.create!(security: @wmt, date: 4.days.ago.to_date, price: 100)
Security::Price.create!(security: @wmt, date: 3.days.ago.to_date, price: 100)
Security::Price.create!(security: @wmt, date: 2.days.ago.to_date, price: 100)
Security::Price.create!(security: @wmt, date: 1.day.ago.to_date, price: 100)
Security::Price.create!(security: @wmt, date: Date.current, price: 100)
@amzn = Security.create!(ticker: "AMZN", name: "Amazon.com Inc.")
Security::Price.create!(security: @amzn, date: 4.days.ago.to_date, price: 200)
Security::Price.create!(security: @amzn, date: 3.days.ago.to_date, price: 200)
Security::Price.create!(security: @amzn, date: 2.days.ago.to_date, price: 200)
Security::Price.create!(security: @amzn, date: 1.day.ago.to_date, price: 200)
Security::Price.create!(security: @amzn, date: Date.current, price: 200)
end
# Portfolio holdings:
# +--------+-----+--------+---------+
# | Ticker | Qty | Price | Amount |
# +--------+-----+--------+---------+
# | VOO | 10 | $500 | $5,000 |
# | WMT | 100 | $100 | $10,000 |
# +--------+-----+--------+---------+
# Brokerage Cash: $5,000
# Holdings Value: $15,000
# Total Balance: $20,000
def calculator_with_trades(security)
yield if block_given?
snapshot = OpenStruct.new(to_h: { security.id => 10 })
calc = Holding::ReverseCalculator.new(@account, portfolio_snapshot: snapshot)
calc.send(:precompute_cost_basis)
calc
end
def cost_basis_for(calc, security, date)
calc.send(:cost_basis_for, security.id, date)
end
def load_today_portfolio
@account.update!(cash_balance: 5000)
load_prices
@account.holdings.create!(
date: Date.current,
price: 500,
qty: 10,
amount: 5000,
currency: "USD",
security: @voo
)
@account.holdings.create!(
date: Date.current,
price: 100,
qty: 100,
amount: 10000,
currency: "USD",
security: @wmt
)
end
end
Reference in New Issue View Git Blame Copy Permalink