QT/sure
1
0
Fork 0
mirror of https://github.com/we-promise/sure.git synced 2026-05-29 23:39:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat(ibkr): compute net_market_flows from IBKR equity equity delta and trade flows (#1970)

Browse Source 
* feat(ibkr): compute net_market_flows from IBKR equity delta and trade flows

Replace the hardcoded net_market_flows: 0 in HistoricalBalancesSync with an
exact derivation from IBKR's own equity summary data, eliminating any
dependency on third-party security price providers for Period Return.

Formula: nmf = Δnon_cash - net_buy_sell
  - non_cash = IBKR equity total - materializer cash (exact per IBKR)
  - net_buy_sell = sum of trade amounts converted to base currency using
    the stored fx_rate_to_base (IBKR's own FX rate, already on Trade#exchange_rate)

Sets non_cash_adjustments = net_buy_sell so the virtual column identity
(end_non_cash_balance = start + nmf + adjustments) resolves to IBKR's
exact equity figure.

* test(ibkr): add sell-trade and no-trade nmf tests; fix memoization guard

- Add test: sell trades (negative amount) correctly isolate market loss in nmf
- Add test: no-trade scenario produces nmf = full Δnon_cash
- Fix: `return {} unless account` inside ||= exited the method without memoizing;
  restructure to `if account ... else {} end` so the result is always cached

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(ibkr): exclude dividend/interest trades from net_buy_sell; use historical FX date

Addresses two issues flagged in code review:

- P1: trades with qty=0 (Dividend, Interest) were included in net_buy_sell,
  inflating/deflating nmf on dates with income events. Filter to qty != 0 at
  the SQL level so only buy/sell trades affect the market-flow calculation.

- P2: Money#exchange_to defaulted to Date.current when no custom_rate was
  stored, causing historical nmf to drift as FX rates change over time.
  Pass date: entry.date so the fallback lookup uses the trade's own date.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* test(ibkr): cover Money::ConversionError fallback in trade_flows_by_date

Adds a test that stubs Money#exchange_to to raise ConversionError for a
cross-currency trade with no stored exchange_rate, verifying that the
rescue clause falls back to entry.amount and that nmf and
end_non_cash_balance still resolve correctly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(ibkr): log warning when FX conversion falls back to unconverted amount

When Money::ConversionError is raised for a cross-currency trade with no
stored exchange_rate, warn with entry currency, account currency, date,
amount, and entry/account IDs so the silent fallback is visible in logs.
Same-currency ConversionErrors (unexpected but possible) stay silent.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(ibkr): skip unconvertible FX trades, redact log, tighten join

- On Money::ConversionError, skip the entry from net_buy_sell rather
  than falling back to the raw amount (which treated e.g. EUR as CHF);
  nmf now absorbs the full Δnon_cash for that date instead of silently
  misstating period return
- Remove entry amount, entry ID, and account ID from the FX warning log
  to avoid exposing financial data in log output
- Consolidate entryable_type guard into the JOIN condition rather than a
  separate WHERE clause
- Add inline comment on the first-day zero case to distinguish intent
  from a bug
- Update ConversionError test to assert skip behavior (nmf=200, not 50)

* fix(ibkr): exclude dates with unconvertible FX trades from balance upsert

* fix(ibkr): skip upsert_all when all balance rows are filtered by failed FX dates

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
CrossDrain
2026-05-26 20:48:23 +00:00
committed by GitHub
parent bc3e5a824f
commit 3e2990a52c
2 changed files with 218 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files

65
app/models/ibkr_account/historical_balances_sync.rb
View File

@@ -11,8 +11,11 @@ class IbkrAccount::HistoricalBalancesSync
return unless account.present?
return if normalized_rows.empty?
rows = balance_rows
return if rows.empty?
account.balances.upsert_all(
balance_rows,
rows,
unique_by: %i[account_id date currency]
)
end
@@ -109,12 +112,35 @@ class IbkrAccount::HistoricalBalancesSync
def balance_rows
current_time = Time.current
trade_flows_by_date # ensure @failed_fx_dates is populated before iterating
normalized_rows.each_with_index.map do |row, index|
normalized_rows.each_with_index.filter_map do |row, index|
next if @failed_fx_dates.include?(row[:date])
previous_row = index.zero? ? nil : normalized_rows[index - 1]
start_cash_balance = previous_row ? previous_row[:cash] : row[:cash]
start_non_cash_balance = previous_row ? previous_row[:non_cash] : row[:non_cash]
# Derive market return directly from IBKR's equity data so Period Return
# matches IBKR without requiring third-party security price providers.
#
# nmf = Δnon_cash - net_buy_sell
# Δnon_cash : change in holdings value per IBKR equity summary (exact)
# net_buy_sell: sum of trade entry amounts converted to base currency
# (positive = buy, negative = sell; IBKR fx_rate_to_base applied)
#
# non_cash_adjustments absorbs net_buy_sell so the virtual column
# end_non_cash_balance = start + nmf + adjustments stays equal to row[:non_cash].
if previous_row
net_buy_sell = trade_flows_by_date[row[:date]] || 0
nmf = row[:non_cash] - start_non_cash_balance - net_buy_sell
non_cash_adj = net_buy_sell
else
# First-day row has no prior period to diff against, so both values are
# intentionally zero — not a bug, just an unavoidable bootstrap constraint.
nmf = 0
non_cash_adj = 0
end
{
account_id: account.id,
date: row[:date],
@@ -127,13 +153,44 @@ class IbkrAccount::HistoricalBalancesSync
cash_outflows: 0,
non_cash_inflows: 0,
non_cash_outflows: 0,
net_market_flows: 0,
net_market_flows: nmf,
cash_adjustments: row[:cash] - start_cash_balance,
non_cash_adjustments: row[:non_cash] - start_non_cash_balance,
non_cash_adjustments: non_cash_adj,
flows_factor: 1,
created_at: current_time,
updated_at: current_time
}
end
end
# Net value of all trades on each date, in account base currency.
# Uses the IBKR-provided fx_rate_to_base stored on each Trade entry so the
# conversion is exact and consistent with IBKR's own calculations.
# Positive = net buy (cash out), negative = net sell (cash in).
def trade_flows_by_date
@trade_flows_by_date ||= begin
@failed_fx_dates = []
if account
account.entries
.joins("INNER JOIN trades ON trades.id = entries.entryable_id AND entries.entryable_type = 'Trade'")
.where.not(trades: { qty: 0 })
.includes(:entryable)
.each_with_object(Hash.new(0)) do |entry, flows|
custom_rate = entry.entryable.exchange_rate
base_amount = Money.new(entry.amount, entry.currency)
.exchange_to(account_currency, custom_rate: custom_rate, date: entry.date)
.amount
flows[entry.date] += base_amount
rescue Money::ConversionError
Rails.logger.warn(
"IbkrAccount::HistoricalBalancesSync - No FX rate for #{entry.currency}→#{account_currency} " \
"on #{entry.date}; balance row for this date will not be persisted"
)
@failed_fx_dates << entry.date
end
else
{}
end
end
end
end

157
test/models/ibkr_account/historical_balances_sync_test.rb
View File

@@ -213,6 +213,163 @@ class IbkrAccount::HistoricalBalancesSyncTest < ActiveSupport::TestCase
assert_not_nil @account.balances.find_by(date: Date.new(2026, 5, 8), currency: "CHF")
end
test "computes net_market_flows from equity delta minus trade flows" do
# Day 1: total=3000, cash=500, non_cash=2500
# Day 2: total=3200, cash=500, non_cash=2700 (Δnon_cash=200)
# Buy trade on Day 2: CHF 150 (same currency as account, no FX)
# nmf = 200 - 150 = 50
@ibkr_account.update!(
raw_equity_summary_payload: [
{ report_date: "2026-05-07", total: "3000.00" },
{ report_date: "2026-05-08", total: "3200.00" }
]
)
seed_balance(date: Date.new(2026, 5, 7), balance: 3000.00, cash_balance: 500.00)
seed_balance(date: Date.new(2026, 5, 8), balance: 3200.00, cash_balance: 500.00)
security = Security.create!(ticker: "TEST", name: "Test Stock")
@account.entries.create!(
name: "Buy 100 TEST",
date: Date.new(2026, 5, 8),
amount: 150.00,
currency: "CHF",
entryable: Trade.new(security: security, qty: 100, price: 1.5, currency: "CHF")
)
IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync!
day1 = @account.balances.find_by!(date: Date.new(2026, 5, 7), currency: "CHF")
day2 = @account.balances.find_by!(date: Date.new(2026, 5, 8), currency: "CHF")
assert_equal BigDecimal("0"), day1.net_market_flows
assert_equal BigDecimal("50"), day2.net_market_flows
# Virtual column must still resolve to IBKR's equity total minus cash
assert_equal BigDecimal("2500.00"), day1.end_non_cash_balance
assert_equal BigDecimal("2700.00"), day2.end_non_cash_balance
end
test "applies fx_rate_to_base when trade currency differs from account currency" do
# Trade in EUR with fx_rate_to_base=1.1 → CHF 165, not CHF 150
# Day 1: non_cash=2500, Day 2: non_cash=2700 (Δ=200)
# nmf = 200 - 165 = 35
@ibkr_account.update!(
raw_equity_summary_payload: [
{ report_date: "2026-05-07", total: "3000.00" },
{ report_date: "2026-05-08", total: "3200.00" }
]
)
seed_balance(date: Date.new(2026, 5, 7), balance: 3000.00, cash_balance: 500.00)
seed_balance(date: Date.new(2026, 5, 8), balance: 3200.00, cash_balance: 500.00)
security = Security.create!(ticker: "TEST2", name: "Test Stock EUR")
trade = Trade.new(security: security, qty: 100, price: 1.5, currency: "EUR")
trade.exchange_rate = 1.1
@account.entries.create!(
name: "Buy 100 TEST2",
date: Date.new(2026, 5, 8),
amount: 150.00,
currency: "EUR",
entryable: trade
)
IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync!
day2 = @account.balances.find_by!(date: Date.new(2026, 5, 8), currency: "CHF")
assert_in_delta 35.0, day2.net_market_flows.to_f, 0.01
assert_equal BigDecimal("2700.00"), day2.end_non_cash_balance
end
test "excludes balance row from upsert when Money::ConversionError prevents FX conversion" do
# EUR trade with no exchange_rate stored → custom_rate=nil → ConversionError raised.
# The affected date is excluded from the upsert entirely so net_market_flows is not
# silently wrong (the trade's value would otherwise flow into market appreciation).
# The seeded day2 balance is intentionally different from IBKR's total (3150 vs 3200)
# so we can assert the row was not overwritten by sync.
@ibkr_account.update!(
raw_equity_summary_payload: [
{ report_date: "2026-05-07", total: "3000.00" },
{ report_date: "2026-05-08", total: "3200.00" }
]
)
seed_balance(date: Date.new(2026, 5, 7), balance: 3000.00, cash_balance: 500.00)
seed_balance(date: Date.new(2026, 5, 8), balance: 3150.00, cash_balance: 500.00)
security = Security.create!(ticker: "NORATE", name: "No Rate EUR Stock")
@account.entries.create!(
name: "Buy 100 NORATE",
date: Date.new(2026, 5, 8),
amount: 150.00,
currency: "EUR",
entryable: Trade.new(security: security, qty: 100, price: 1.5, currency: "EUR")
)
Money.any_instance.stubs(:exchange_to).raises(
Money::ConversionError.new(from_currency: "EUR", to_currency: "CHF", date: Date.new(2026, 5, 8))
)
IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync!
# Day 1 is unaffected — still synced normally
day1 = @account.balances.find_by!(date: Date.new(2026, 5, 7), currency: "CHF")
assert_equal BigDecimal("3000.00"), day1.balance
# Day 2 was excluded from the upsert — seeded values are preserved, not overwritten
day2 = @account.balances.find_by!(date: Date.new(2026, 5, 8), currency: "CHF")
assert_equal BigDecimal("3150.00"), day2.balance # seeded, not IBKR's 3200
assert_equal BigDecimal("0"), day2.net_market_flows # seeded, not recomputed
end
test "net_market_flows equals full non_cash delta when account has no trades" do
@ibkr_account.update!(
raw_equity_summary_payload: [
{ report_date: "2026-05-07", total: "3000.00" },
{ report_date: "2026-05-08", total: "3300.00" }
]
)
seed_balance(date: Date.new(2026, 5, 7), balance: 3000.00, cash_balance: 500.00)
seed_balance(date: Date.new(2026, 5, 8), balance: 3300.00, cash_balance: 500.00)
IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync!
day1 = @account.balances.find_by!(date: Date.new(2026, 5, 7), currency: "CHF")
day2 = @account.balances.find_by!(date: Date.new(2026, 5, 8), currency: "CHF")
assert_equal BigDecimal("0"), day1.net_market_flows
assert_equal BigDecimal("300"), day2.net_market_flows
assert_equal BigDecimal("2800.00"), day2.end_non_cash_balance
end
test "sell trades reduce net_buy_sell so market loss is isolated in net_market_flows" do
# Day 1: total=3000, cash=500, non_cash=2500
# Day 2: total=2700, cash=700, non_cash=2000 (Δnon_cash=-500)
# Sell 100 at CHF 1.50: entry.amount=-150 (negative = proceeds received)
# net_buy_sell=-150; nmf = -500 - (-150) = -350 (market caused -350 loss)
@ibkr_account.update!(
raw_equity_summary_payload: [
{ report_date: "2026-05-07", total: "3000.00" },
{ report_date: "2026-05-08", total: "2700.00" }
]
)
seed_balance(date: Date.new(2026, 5, 7), balance: 3000.00, cash_balance: 500.00)
seed_balance(date: Date.new(2026, 5, 8), balance: 2700.00, cash_balance: 700.00)
security = Security.create!(ticker: "SELL_TEST", name: "Sell Test Stock")
@account.entries.create!(
name: "Sell 100 SELL_TEST",
date: Date.new(2026, 5, 8),
amount: -150.00,
currency: "CHF",
entryable: Trade.new(security: security, qty: -100, price: 1.5, currency: "CHF")
)
IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync!
day2 = @account.balances.find_by!(date: Date.new(2026, 5, 8), currency: "CHF")
assert_equal BigDecimal("-350"), day2.net_market_flows
assert_equal BigDecimal("2000.00"), day2.end_non_cash_balance
end
test "writes balance row with zero total for fully liquidated dates" do
@ibkr_account.update!(
raw_equity_summary_payload: [