mirror of
https://github.com/we-promise/sure.git
synced 2026-05-30 07:49:01 +00:00
feat(ibkr): compute net_market_flows from IBKR equity equity delta and trade flows (#1970)
* feat(ibkr): compute net_market_flows from IBKR equity delta and trade flows
Replace the hardcoded net_market_flows: 0 in HistoricalBalancesSync with an
exact derivation from IBKR's own equity summary data, eliminating any
dependency on third-party security price providers for Period Return.
Formula: nmf = Δnon_cash - net_buy_sell
- non_cash = IBKR equity total - materializer cash (exact per IBKR)
- net_buy_sell = sum of trade amounts converted to base currency using
the stored fx_rate_to_base (IBKR's own FX rate, already on Trade#exchange_rate)
Sets non_cash_adjustments = net_buy_sell so the virtual column identity
(end_non_cash_balance = start + nmf + adjustments) resolves to IBKR's
exact equity figure.
* test(ibkr): add sell-trade and no-trade nmf tests; fix memoization guard
- Add test: sell trades (negative amount) correctly isolate market loss in nmf
- Add test: no-trade scenario produces nmf = full Δnon_cash
- Fix: `return {} unless account` inside ||= exited the method without memoizing;
restructure to `if account ... else {} end` so the result is always cached
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(ibkr): exclude dividend/interest trades from net_buy_sell; use historical FX date
Addresses two issues flagged in code review:
- P1: trades with qty=0 (Dividend, Interest) were included in net_buy_sell,
inflating/deflating nmf on dates with income events. Filter to qty != 0 at
the SQL level so only buy/sell trades affect the market-flow calculation.
- P2: Money#exchange_to defaulted to Date.current when no custom_rate was
stored, causing historical nmf to drift as FX rates change over time.
Pass date: entry.date so the fallback lookup uses the trade's own date.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* test(ibkr): cover Money::ConversionError fallback in trade_flows_by_date
Adds a test that stubs Money#exchange_to to raise ConversionError for a
cross-currency trade with no stored exchange_rate, verifying that the
rescue clause falls back to entry.amount and that nmf and
end_non_cash_balance still resolve correctly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(ibkr): log warning when FX conversion falls back to unconverted amount
When Money::ConversionError is raised for a cross-currency trade with no
stored exchange_rate, warn with entry currency, account currency, date,
amount, and entry/account IDs so the silent fallback is visible in logs.
Same-currency ConversionErrors (unexpected but possible) stay silent.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(ibkr): skip unconvertible FX trades, redact log, tighten join
- On Money::ConversionError, skip the entry from net_buy_sell rather
than falling back to the raw amount (which treated e.g. EUR as CHF);
nmf now absorbs the full Δnon_cash for that date instead of silently
misstating period return
- Remove entry amount, entry ID, and account ID from the FX warning log
to avoid exposing financial data in log output
- Consolidate entryable_type guard into the JOIN condition rather than a
separate WHERE clause
- Add inline comment on the first-day zero case to distinguish intent
from a bug
- Update ConversionError test to assert skip behavior (nmf=200, not 50)
* fix(ibkr): exclude dates with unconvertible FX trades from balance upsert
* fix(ibkr): skip upsert_all when all balance rows are filtered by failed FX dates
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
CrossDrain
committed by
Juan José Mata
Juan José Mata
parent
af024a89cb
commit
33cc3508b8
@@ -11,8 +11,11 @@ class IbkrAccount::HistoricalBalancesSync
|
||||
return unless account.present?
|
||||
return if normalized_rows.empty?
|
||||
|
||||
rows = balance_rows
|
||||
return if rows.empty?
|
||||
|
||||
account.balances.upsert_all(
|
||||
balance_rows,
|
||||
rows,
|
||||
unique_by: %i[account_id date currency]
|
||||
)
|
||||
end
|
||||
@@ -109,12 +112,35 @@ class IbkrAccount::HistoricalBalancesSync
|
||||
|
||||
def balance_rows
|
||||
current_time = Time.current
|
||||
trade_flows_by_date # ensure @failed_fx_dates is populated before iterating
|
||||
|
||||
normalized_rows.each_with_index.map do |row, index|
|
||||
normalized_rows.each_with_index.filter_map do |row, index|
|
||||
next if @failed_fx_dates.include?(row[:date])
|
||||
previous_row = index.zero? ? nil : normalized_rows[index - 1]
|
||||
start_cash_balance = previous_row ? previous_row[:cash] : row[:cash]
|
||||
start_non_cash_balance = previous_row ? previous_row[:non_cash] : row[:non_cash]
|
||||
|
||||
# Derive market return directly from IBKR's equity data so Period Return
|
||||
# matches IBKR without requiring third-party security price providers.
|
||||
#
|
||||
# nmf = Δnon_cash - net_buy_sell
|
||||
# Δnon_cash : change in holdings value per IBKR equity summary (exact)
|
||||
# net_buy_sell: sum of trade entry amounts converted to base currency
|
||||
# (positive = buy, negative = sell; IBKR fx_rate_to_base applied)
|
||||
#
|
||||
# non_cash_adjustments absorbs net_buy_sell so the virtual column
|
||||
# end_non_cash_balance = start + nmf + adjustments stays equal to row[:non_cash].
|
||||
if previous_row
|
||||
net_buy_sell = trade_flows_by_date[row[:date]] || 0
|
||||
nmf = row[:non_cash] - start_non_cash_balance - net_buy_sell
|
||||
non_cash_adj = net_buy_sell
|
||||
else
|
||||
# First-day row has no prior period to diff against, so both values are
|
||||
# intentionally zero — not a bug, just an unavoidable bootstrap constraint.
|
||||
nmf = 0
|
||||
non_cash_adj = 0
|
||||
end
|
||||
|
||||
{
|
||||
account_id: account.id,
|
||||
date: row[:date],
|
||||
@@ -127,13 +153,44 @@ class IbkrAccount::HistoricalBalancesSync
|
||||
cash_outflows: 0,
|
||||
non_cash_inflows: 0,
|
||||
non_cash_outflows: 0,
|
||||
net_market_flows: 0,
|
||||
net_market_flows: nmf,
|
||||
cash_adjustments: row[:cash] - start_cash_balance,
|
||||
non_cash_adjustments: row[:non_cash] - start_non_cash_balance,
|
||||
non_cash_adjustments: non_cash_adj,
|
||||
flows_factor: 1,
|
||||
created_at: current_time,
|
||||
updated_at: current_time
|
||||
}
|
||||
end
|
||||
end
|
||||
|
||||
# Net value of all trades on each date, in account base currency.
|
||||
# Uses the IBKR-provided fx_rate_to_base stored on each Trade entry so the
|
||||
# conversion is exact and consistent with IBKR's own calculations.
|
||||
# Positive = net buy (cash out), negative = net sell (cash in).
|
||||
def trade_flows_by_date
|
||||
@trade_flows_by_date ||= begin
|
||||
@failed_fx_dates = []
|
||||
if account
|
||||
account.entries
|
||||
.joins("INNER JOIN trades ON trades.id = entries.entryable_id AND entries.entryable_type = 'Trade'")
|
||||
.where.not(trades: { qty: 0 })
|
||||
.includes(:entryable)
|
||||
.each_with_object(Hash.new(0)) do |entry, flows|
|
||||
custom_rate = entry.entryable.exchange_rate
|
||||
base_amount = Money.new(entry.amount, entry.currency)
|
||||
.exchange_to(account_currency, custom_rate: custom_rate, date: entry.date)
|
||||
.amount
|
||||
flows[entry.date] += base_amount
|
||||
rescue Money::ConversionError
|
||||
Rails.logger.warn(
|
||||
"IbkrAccount::HistoricalBalancesSync - No FX rate for #{entry.currency}→#{account_currency} " \
|
||||
"on #{entry.date}; balance row for this date will not be persisted"
|
||||
)
|
||||
@failed_fx_dates << entry.date
|
||||
end
|
||||
else
|
||||
{}
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
@@ -213,6 +213,163 @@ class IbkrAccount::HistoricalBalancesSyncTest < ActiveSupport::TestCase
|
||||
assert_not_nil @account.balances.find_by(date: Date.new(2026, 5, 8), currency: "CHF")
|
||||
end
|
||||
|
||||
test "computes net_market_flows from equity delta minus trade flows" do
|
||||
# Day 1: total=3000, cash=500, non_cash=2500
|
||||
# Day 2: total=3200, cash=500, non_cash=2700 (Δnon_cash=200)
|
||||
# Buy trade on Day 2: CHF 150 (same currency as account, no FX)
|
||||
# nmf = 200 - 150 = 50
|
||||
@ibkr_account.update!(
|
||||
raw_equity_summary_payload: [
|
||||
{ report_date: "2026-05-07", total: "3000.00" },
|
||||
{ report_date: "2026-05-08", total: "3200.00" }
|
||||
]
|
||||
)
|
||||
seed_balance(date: Date.new(2026, 5, 7), balance: 3000.00, cash_balance: 500.00)
|
||||
seed_balance(date: Date.new(2026, 5, 8), balance: 3200.00, cash_balance: 500.00)
|
||||
|
||||
security = Security.create!(ticker: "TEST", name: "Test Stock")
|
||||
@account.entries.create!(
|
||||
name: "Buy 100 TEST",
|
||||
date: Date.new(2026, 5, 8),
|
||||
amount: 150.00,
|
||||
currency: "CHF",
|
||||
entryable: Trade.new(security: security, qty: 100, price: 1.5, currency: "CHF")
|
||||
)
|
||||
|
||||
IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync!
|
||||
|
||||
day1 = @account.balances.find_by!(date: Date.new(2026, 5, 7), currency: "CHF")
|
||||
day2 = @account.balances.find_by!(date: Date.new(2026, 5, 8), currency: "CHF")
|
||||
|
||||
assert_equal BigDecimal("0"), day1.net_market_flows
|
||||
assert_equal BigDecimal("50"), day2.net_market_flows
|
||||
|
||||
# Virtual column must still resolve to IBKR's equity total minus cash
|
||||
assert_equal BigDecimal("2500.00"), day1.end_non_cash_balance
|
||||
assert_equal BigDecimal("2700.00"), day2.end_non_cash_balance
|
||||
end
|
||||
|
||||
test "applies fx_rate_to_base when trade currency differs from account currency" do
|
||||
# Trade in EUR with fx_rate_to_base=1.1 → CHF 165, not CHF 150
|
||||
# Day 1: non_cash=2500, Day 2: non_cash=2700 (Δ=200)
|
||||
# nmf = 200 - 165 = 35
|
||||
@ibkr_account.update!(
|
||||
raw_equity_summary_payload: [
|
||||
{ report_date: "2026-05-07", total: "3000.00" },
|
||||
{ report_date: "2026-05-08", total: "3200.00" }
|
||||
]
|
||||
)
|
||||
seed_balance(date: Date.new(2026, 5, 7), balance: 3000.00, cash_balance: 500.00)
|
||||
seed_balance(date: Date.new(2026, 5, 8), balance: 3200.00, cash_balance: 500.00)
|
||||
|
||||
security = Security.create!(ticker: "TEST2", name: "Test Stock EUR")
|
||||
trade = Trade.new(security: security, qty: 100, price: 1.5, currency: "EUR")
|
||||
trade.exchange_rate = 1.1
|
||||
@account.entries.create!(
|
||||
name: "Buy 100 TEST2",
|
||||
date: Date.new(2026, 5, 8),
|
||||
amount: 150.00,
|
||||
currency: "EUR",
|
||||
entryable: trade
|
||||
)
|
||||
|
||||
IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync!
|
||||
|
||||
day2 = @account.balances.find_by!(date: Date.new(2026, 5, 8), currency: "CHF")
|
||||
assert_in_delta 35.0, day2.net_market_flows.to_f, 0.01
|
||||
assert_equal BigDecimal("2700.00"), day2.end_non_cash_balance
|
||||
end
|
||||
|
||||
test "excludes balance row from upsert when Money::ConversionError prevents FX conversion" do
|
||||
# EUR trade with no exchange_rate stored → custom_rate=nil → ConversionError raised.
|
||||
# The affected date is excluded from the upsert entirely so net_market_flows is not
|
||||
# silently wrong (the trade's value would otherwise flow into market appreciation).
|
||||
# The seeded day2 balance is intentionally different from IBKR's total (3150 vs 3200)
|
||||
# so we can assert the row was not overwritten by sync.
|
||||
@ibkr_account.update!(
|
||||
raw_equity_summary_payload: [
|
||||
{ report_date: "2026-05-07", total: "3000.00" },
|
||||
{ report_date: "2026-05-08", total: "3200.00" }
|
||||
]
|
||||
)
|
||||
seed_balance(date: Date.new(2026, 5, 7), balance: 3000.00, cash_balance: 500.00)
|
||||
seed_balance(date: Date.new(2026, 5, 8), balance: 3150.00, cash_balance: 500.00)
|
||||
|
||||
security = Security.create!(ticker: "NORATE", name: "No Rate EUR Stock")
|
||||
@account.entries.create!(
|
||||
name: "Buy 100 NORATE",
|
||||
date: Date.new(2026, 5, 8),
|
||||
amount: 150.00,
|
||||
currency: "EUR",
|
||||
entryable: Trade.new(security: security, qty: 100, price: 1.5, currency: "EUR")
|
||||
)
|
||||
|
||||
Money.any_instance.stubs(:exchange_to).raises(
|
||||
Money::ConversionError.new(from_currency: "EUR", to_currency: "CHF", date: Date.new(2026, 5, 8))
|
||||
)
|
||||
|
||||
IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync!
|
||||
|
||||
# Day 1 is unaffected — still synced normally
|
||||
day1 = @account.balances.find_by!(date: Date.new(2026, 5, 7), currency: "CHF")
|
||||
assert_equal BigDecimal("3000.00"), day1.balance
|
||||
|
||||
# Day 2 was excluded from the upsert — seeded values are preserved, not overwritten
|
||||
day2 = @account.balances.find_by!(date: Date.new(2026, 5, 8), currency: "CHF")
|
||||
assert_equal BigDecimal("3150.00"), day2.balance # seeded, not IBKR's 3200
|
||||
assert_equal BigDecimal("0"), day2.net_market_flows # seeded, not recomputed
|
||||
end
|
||||
|
||||
test "net_market_flows equals full non_cash delta when account has no trades" do
|
||||
@ibkr_account.update!(
|
||||
raw_equity_summary_payload: [
|
||||
{ report_date: "2026-05-07", total: "3000.00" },
|
||||
{ report_date: "2026-05-08", total: "3300.00" }
|
||||
]
|
||||
)
|
||||
seed_balance(date: Date.new(2026, 5, 7), balance: 3000.00, cash_balance: 500.00)
|
||||
seed_balance(date: Date.new(2026, 5, 8), balance: 3300.00, cash_balance: 500.00)
|
||||
|
||||
IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync!
|
||||
|
||||
day1 = @account.balances.find_by!(date: Date.new(2026, 5, 7), currency: "CHF")
|
||||
day2 = @account.balances.find_by!(date: Date.new(2026, 5, 8), currency: "CHF")
|
||||
|
||||
assert_equal BigDecimal("0"), day1.net_market_flows
|
||||
assert_equal BigDecimal("300"), day2.net_market_flows
|
||||
assert_equal BigDecimal("2800.00"), day2.end_non_cash_balance
|
||||
end
|
||||
|
||||
test "sell trades reduce net_buy_sell so market loss is isolated in net_market_flows" do
|
||||
# Day 1: total=3000, cash=500, non_cash=2500
|
||||
# Day 2: total=2700, cash=700, non_cash=2000 (Δnon_cash=-500)
|
||||
# Sell 100 at CHF 1.50: entry.amount=-150 (negative = proceeds received)
|
||||
# net_buy_sell=-150; nmf = -500 - (-150) = -350 (market caused -350 loss)
|
||||
@ibkr_account.update!(
|
||||
raw_equity_summary_payload: [
|
||||
{ report_date: "2026-05-07", total: "3000.00" },
|
||||
{ report_date: "2026-05-08", total: "2700.00" }
|
||||
]
|
||||
)
|
||||
seed_balance(date: Date.new(2026, 5, 7), balance: 3000.00, cash_balance: 500.00)
|
||||
seed_balance(date: Date.new(2026, 5, 8), balance: 2700.00, cash_balance: 700.00)
|
||||
|
||||
security = Security.create!(ticker: "SELL_TEST", name: "Sell Test Stock")
|
||||
@account.entries.create!(
|
||||
name: "Sell 100 SELL_TEST",
|
||||
date: Date.new(2026, 5, 8),
|
||||
amount: -150.00,
|
||||
currency: "CHF",
|
||||
entryable: Trade.new(security: security, qty: -100, price: 1.5, currency: "CHF")
|
||||
)
|
||||
|
||||
IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync!
|
||||
|
||||
day2 = @account.balances.find_by!(date: Date.new(2026, 5, 8), currency: "CHF")
|
||||
assert_equal BigDecimal("-350"), day2.net_market_flows
|
||||
assert_equal BigDecimal("2000.00"), day2.end_non_cash_balance
|
||||
end
|
||||
|
||||
test "writes balance row with zero total for fully liquidated dates" do
|
||||
@ibkr_account.update!(
|
||||
raw_equity_summary_payload: [
|
||||
|
||||
Reference in New Issue
Block a user