mirror of
https://github.com/we-promise/sure.git
synced 2026-05-24 21:14:56 +00:00
6262b0a493
* fix(ibkr): resolve weekend balance oscillations and improve data processing Address issues where IBKR weekend/holiday data caused incorrect balance calculations and improve the robustness of IBKR account processing. - Fix historical balance oscillations by ignoring anomalous weekend rows and filling gaps by carrying forward the last known trading day value. - Normalize report dates to the last trading day to ensure consistency. - Improve `HoldingsProcessor` to skip individual bad lots instead of failing the entire group. - Refactor `ActivitiesProcessor` to accumulate fee counts locally via return values instead of using instance variables. - Add support for accounting parentheses notation in `DataHelpers`. - Memoize the account object in `IbkrAccount::Processor` to reduce database queries. - Update tests to reflect date normalization and improved precision assertions. * fix(ibkr): derive historical cash from materializer balances, not equity summary Real IBKR Flex exports do not include a reliable cash/stock breakdown in EquitySummaryByReportDateInBase — only the total is consistently present. The previous implementation parsed the missing cash field as zero and wrote cash_balance=0 for every historical date, causing negative and wildly incorrect cash values throughout the account history. Instead, read the materializer's already-computed cash_balance for each date (derived from holdings via the reverse calculator) and use only IBKR's total as an authoritative balance anchor. This is consistent with how present-day balances are handled and requires no weekend/holiday filtering since IBKR does not emit weekend rows and holiday totals are legitimate data points. Also accept equity summary rows without an explicit currency field (some Flex configurations omit it) and explicitly reject BASE_SUMMARY aggregate rows. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: simplify boolean coercion in import_commission_transaction Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(ibkr): cover trailing weekend gap and align qty with valid lots HistoricalBalancesSync: extend fill_gaps to account.current_anchor_date so days after the last equity summary row (e.g. Saturday/Sunday when a sync runs over the weekend) are also overridden rather than left with the materializer's stale total=cash value. HoldingsProcessor: replace separate quantity sum + weighted_cost_basis_for with a single valid_lots method that computes both from the same set of parseable lots. Previously a lot with a valid position but unparseable cost_basis_price was excluded from the cost basis calculation but still counted in quantity, producing inconsistent qty/cost_basis values. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * review: address PR feedback on ibkr fix branch - Remove all "Fix N:" review-artifact comment labels - Add Sentry.capture_message for silenced anchor repair failure so it surfaces in production monitoring - Add Rails.logger.warn for zero/nil total rows skipped in HistoricalBalancesSync - Document normalize_to_last_trading_day holiday limitation and why gap-fill covers it - Rewrite two non-obvious comments to stand alone without the label prefix Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * style: remove alignment padding in balance_rows hash Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(ibkr): address two P1 review findings - Allow zero and negative equity summary totals through HistoricalBalancesSync so fully-liquidated and margin accounts are not silently skipped (which would cause fill_gaps to propagate a stale non-zero total forward). - Remove normalize_to_last_trading_day from HoldingsProcessor: shifting weekend report_dates to Friday caused Balance::SyncCache#get_holdings_value to find no holdings on Saturday/Sunday (exact-date lookup), collapsing non_cash to zero — reintroducing the very oscillation the fix was meant to prevent. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(ibkr): add tests for historical balances sync and data helpers - Add test case to verify non-cash balance calculation in historical balances sync - Add test case to ensure rows with unparseable or nil totals are skipped - Add new test file for IBKR data helpers * fix(ibkr): prevent date range overflow during historical sync Adjust the calculation of `last_date` in `HistoricalBalancesSync` to ensure it does not exceed the current anchor date or today's date. This prevents the sync process from attempting to fetch or process future dates, which was causing oscillations in weekend data. Also remove the conditional check for Sentry before capturing error messages in the account processor. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
282 lines
9.5 KiB
Ruby
282 lines
9.5 KiB
Ruby
require "test_helper"
|
|
|
|
class IbkrAccountProcessorTest < ActiveSupport::TestCase
|
|
fixtures :families, :ibkr_items, :ibkr_accounts, :accounts, :securities
|
|
|
|
setup do
|
|
@family = families(:dylan_family)
|
|
@ibkr_account = ibkr_accounts(:main_account)
|
|
|
|
@account = @family.accounts.create!(
|
|
name: "IBKR Investment",
|
|
balance: 0,
|
|
cash_balance: 0,
|
|
currency: "CHF",
|
|
accountable: Investment.new(subtype: "brokerage")
|
|
)
|
|
@ibkr_account.ensure_account_provider!(@account)
|
|
@ibkr_account.update!(
|
|
raw_holdings_payload: [
|
|
{
|
|
"asset_category" => "STK",
|
|
"conid" => "265598",
|
|
"security_id" => "US0378331005",
|
|
"security_id_type" => "ISIN",
|
|
"symbol" => securities(:aapl).ticker,
|
|
"position" => "10",
|
|
"mark_price" => "150.00",
|
|
"currency" => "USD",
|
|
"fx_rate_to_base" => "0.90",
|
|
"cost_basis_price" => "125.50",
|
|
"report_date" => Date.current.to_s,
|
|
"side" => "Long"
|
|
}
|
|
],
|
|
raw_activities_payload: {
|
|
trades: [
|
|
{
|
|
"asset_category" => "STK",
|
|
"trade_id" => "1001",
|
|
"transaction_id" => "1001a",
|
|
"conid" => "265598",
|
|
"symbol" => securities(:aapl).ticker,
|
|
"quantity" => "2",
|
|
"trade_price" => "140.00",
|
|
"currency" => "USD",
|
|
"fx_rate_to_base" => "0.90",
|
|
"buy_sell" => "BUY",
|
|
"trade_date" => Date.current.to_s,
|
|
"ib_commission" => "-1.25",
|
|
"ib_commission_currency" => "USD"
|
|
},
|
|
{
|
|
"asset_category" => "STK",
|
|
"trade_id" => "1002",
|
|
"transaction_id" => "1002a",
|
|
"conid" => "265598",
|
|
"symbol" => securities(:aapl).ticker,
|
|
"quantity" => "-1",
|
|
"trade_price" => "155.00",
|
|
"currency" => "USD",
|
|
"fx_rate_to_base" => "0.92",
|
|
"buy_sell" => "SELL",
|
|
"trade_date" => Date.current.to_s,
|
|
"ib_commission" => "-1.10",
|
|
"ib_commission_currency" => "USD"
|
|
}
|
|
],
|
|
cash_transactions: [
|
|
{
|
|
"transaction_id" => "4001",
|
|
"type" => "Deposits/Withdrawals",
|
|
"amount" => "500.00",
|
|
"currency" => "CHF",
|
|
"fx_rate_to_base" => "1",
|
|
"report_date" => Date.current.to_s
|
|
},
|
|
{
|
|
"transaction_id" => "4002",
|
|
"type" => "Dividends",
|
|
"amount" => "2.50",
|
|
"currency" => "USD",
|
|
"fx_rate_to_base" => "0.91",
|
|
"report_date" => Date.current.to_s,
|
|
"conid" => "265598"
|
|
}
|
|
]
|
|
},
|
|
report_date: Date.current,
|
|
current_balance: BigDecimal("3351.00"),
|
|
cash_balance: BigDecimal("1000.50"),
|
|
currency: "CHF"
|
|
)
|
|
end
|
|
|
|
test "processor imports holdings, trades, cash transactions, and commissions" do
|
|
IbkrAccount::Processor.new(@ibkr_account).process
|
|
|
|
@account.reload
|
|
assert_equal BigDecimal("3351.00"), @account.balance
|
|
assert_equal BigDecimal("1000.50"), @account.cash_balance
|
|
assert_equal "CHF", @account.currency
|
|
|
|
holding = @account.holdings.find_by(security: securities(:aapl), date: Date.current)
|
|
assert_not_nil holding
|
|
assert_equal BigDecimal("10"), holding.qty
|
|
assert_equal BigDecimal("150.00"), holding.price
|
|
assert_equal BigDecimal("125.50"), holding.cost_basis
|
|
assert_equal "USD", holding.currency
|
|
|
|
buy_trade = @account.entries.find_by(external_id: "ibkr_trade_1001")
|
|
sell_trade = @account.entries.find_by(external_id: "ibkr_trade_1002")
|
|
assert_not_nil buy_trade
|
|
assert_not_nil sell_trade
|
|
assert_equal "Buy", buy_trade.entryable.investment_activity_label
|
|
assert_equal "Sell", sell_trade.entryable.investment_activity_label
|
|
assert_equal BigDecimal("2"), buy_trade.entryable.qty
|
|
assert_equal BigDecimal("-1"), sell_trade.entryable.qty
|
|
assert_equal BigDecimal("280.0"), buy_trade.amount
|
|
assert_equal BigDecimal("-155.0"), sell_trade.amount
|
|
assert_equal "USD", buy_trade.currency
|
|
assert_equal "USD", sell_trade.currency
|
|
assert_equal 0.9, buy_trade.entryable.exchange_rate
|
|
assert_equal 0.92, sell_trade.entryable.exchange_rate
|
|
|
|
dividend = @account.entries.find_by(external_id: "ibkr_cash_4002")
|
|
assert_not_nil dividend
|
|
assert_equal "Dividend", dividend.entryable.investment_activity_label
|
|
assert_equal BigDecimal("-2.5"), dividend.amount
|
|
assert_equal securities(:aapl).id, dividend.entryable.extra["security_id"]
|
|
|
|
commission_one = @account.entries.find_by(external_id: "ibkr_trade_fee_1001")
|
|
commission_two = @account.entries.find_by(external_id: "ibkr_trade_fee_1002")
|
|
assert_not_nil commission_one
|
|
assert_not_nil commission_two
|
|
assert_equal BigDecimal("1.25"), commission_one.amount
|
|
assert_equal BigDecimal("1.1"), commission_two.amount
|
|
assert_equal "USD", commission_one.currency
|
|
assert_equal "USD", commission_two.currency
|
|
assert_equal securities(:aapl).id, commission_one.entryable.extra["security_id"]
|
|
assert_equal securities(:aapl).id, commission_two.entryable.extra["security_id"]
|
|
|
|
deposit = @account.entries.find_by(external_id: "ibkr_cash_4001")
|
|
|
|
assert_not_nil deposit
|
|
assert_equal "Contribution", deposit.entryable.investment_activity_label
|
|
assert_equal BigDecimal("-500"), deposit.amount
|
|
assert_equal "CHF", deposit.currency
|
|
|
|
assert_equal "USD", dividend.currency
|
|
end
|
|
|
|
test "processor computes weighted provider cost basis for grouped lots" do
|
|
@ibkr_account.update!(
|
|
raw_holdings_payload: [
|
|
{
|
|
"asset_category" => "STK",
|
|
"conid" => "265598",
|
|
"security_id" => "US0378331005",
|
|
"security_id_type" => "ISIN",
|
|
"symbol" => securities(:aapl).ticker,
|
|
"position" => "10",
|
|
"mark_price" => "150.00",
|
|
"currency" => "USD",
|
|
"fx_rate_to_base" => "0.90",
|
|
"cost_basis_price" => "125.50",
|
|
"report_date" => Date.current.to_s,
|
|
"side" => "Long"
|
|
},
|
|
{
|
|
"asset_category" => "STK",
|
|
"conid" => "265598",
|
|
"security_id" => "US0378331005",
|
|
"security_id_type" => "ISIN",
|
|
"symbol" => securities(:aapl).ticker,
|
|
"position" => "20",
|
|
"mark_price" => "150.00",
|
|
"currency" => "USD",
|
|
"fx_rate_to_base" => "0.90",
|
|
"cost_basis_price" => "122.00",
|
|
"report_date" => Date.current.to_s,
|
|
"side" => "Long"
|
|
}
|
|
]
|
|
)
|
|
|
|
IbkrAccount::Processor.new(@ibkr_account).process
|
|
|
|
holding = @account.holdings.find_by(security: securities(:aapl), date: Date.current)
|
|
|
|
assert_not_nil holding
|
|
assert_equal BigDecimal("30"), holding.qty
|
|
assert_in_delta BigDecimal("123.1667"), holding.cost_basis, BigDecimal("0.0001")
|
|
end
|
|
|
|
test "processor repairs default opening anchor after importing activity entries" do
|
|
result = Account::OpeningBalanceManager.new(@account).set_opening_balance(
|
|
balance: @ibkr_account.current_balance,
|
|
date: 2.years.ago.to_date
|
|
)
|
|
|
|
assert result.success?
|
|
|
|
opening_anchor = @account.valuations.opening_anchor.includes(:entry).first
|
|
assert_not_nil opening_anchor
|
|
assert_equal @ibkr_account.current_balance.to_d, opening_anchor.entry.amount.to_d
|
|
|
|
IbkrAccount::Processor.new(@ibkr_account).process
|
|
|
|
opening_anchor.reload
|
|
assert_equal BigDecimal("0"), opening_anchor.entry.amount.to_d
|
|
end
|
|
|
|
test "processor imports commission-free trades without creating fee entries" do
|
|
@ibkr_account.update!(
|
|
raw_activities_payload: {
|
|
trades: [
|
|
{
|
|
"asset_category" => "STK",
|
|
"trade_id" => "1003",
|
|
"transaction_id" => "1003a",
|
|
"conid" => "265598",
|
|
"symbol" => securities(:aapl).ticker,
|
|
"quantity" => "3",
|
|
"trade_price" => "145.00",
|
|
"currency" => "USD",
|
|
"fx_rate_to_base" => "0.91",
|
|
"buy_sell" => "BUY",
|
|
"trade_date" => Date.current.to_s
|
|
}
|
|
],
|
|
cash_transactions: []
|
|
}
|
|
)
|
|
|
|
IbkrAccount::Processor.new(@ibkr_account).process
|
|
|
|
trade = @account.entries.find_by(external_id: "ibkr_trade_1003")
|
|
fee = @account.entries.find_by(external_id: "ibkr_trade_fee_1003")
|
|
|
|
assert_not_nil trade
|
|
assert_equal BigDecimal("3"), trade.entryable.qty
|
|
assert_equal BigDecimal("435.0"), trade.amount
|
|
assert_equal "USD", trade.currency
|
|
assert_nil fee
|
|
end
|
|
|
|
test "processor logs and falls back to current date for invalid trade_date" do
|
|
@ibkr_account.update!(
|
|
raw_activities_payload: {
|
|
trades: [
|
|
{
|
|
"asset_category" => "STK",
|
|
"trade_id" => "1004",
|
|
"transaction_id" => "1004a",
|
|
"conid" => "265598",
|
|
"symbol" => securities(:aapl).ticker,
|
|
"quantity" => "1",
|
|
"trade_price" => "146.00",
|
|
"currency" => "USD",
|
|
"fx_rate_to_base" => "0.91",
|
|
"buy_sell" => "BUY",
|
|
"trade_date" => "not-a-date"
|
|
}
|
|
],
|
|
cash_transactions: []
|
|
}
|
|
)
|
|
|
|
Rails.logger.expects(:warn).with do |message|
|
|
message.include?("IbkrAccount::DataHelpers - Missing or invalid trade_date") &&
|
|
message.include?("1004")
|
|
end
|
|
|
|
IbkrAccount::Processor.new(@ibkr_account).process
|
|
|
|
trade = @account.entries.find_by(external_id: "ibkr_trade_1004")
|
|
|
|
assert_not_nil trade
|
|
assert_equal Date.current, trade.date
|
|
end
|
|
end
|