From 6262b0a49319895b693c85b0ffbfb78c1b20533c Mon Sep 17 00:00:00 2001 From: CrossDrain <32982516+CrossDrain@users.noreply.github.com> Date: Mon, 18 May 2026 19:03:04 +0000 Subject: [PATCH 1/2] fix(ibkr): correct historical cash/non-cash split for linked accounts (#1813) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(ibkr): resolve weekend balance oscillations and improve data processing Address issues where IBKR weekend/holiday data caused incorrect balance calculations and improve the robustness of IBKR account processing. - Fix historical balance oscillations by ignoring anomalous weekend rows and filling gaps by carrying forward the last known trading day value. - Normalize report dates to the last trading day to ensure consistency. - Improve `HoldingsProcessor` to skip individual bad lots instead of failing the entire group. - Refactor `ActivitiesProcessor` to accumulate fee counts locally via return values instead of using instance variables. - Add support for accounting parentheses notation in `DataHelpers`. - Memoize the account object in `IbkrAccount::Processor` to reduce database queries. - Update tests to reflect date normalization and improved precision assertions. * fix(ibkr): derive historical cash from materializer balances, not equity summary Real IBKR Flex exports do not include a reliable cash/stock breakdown in EquitySummaryByReportDateInBase — only the total is consistently present. The previous implementation parsed the missing cash field as zero and wrote cash_balance=0 for every historical date, causing negative and wildly incorrect cash values throughout the account history. Instead, read the materializer's already-computed cash_balance for each date (derived from holdings via the reverse calculator) and use only IBKR's total as an authoritative balance anchor. This is consistent with how present-day balances are handled and requires no weekend/holiday filtering since IBKR does not emit weekend rows and holiday totals are legitimate data points. Also accept equity summary rows without an explicit currency field (some Flex configurations omit it) and explicitly reject BASE_SUMMARY aggregate rows. Co-Authored-By: Claude Sonnet 4.6 * style: simplify boolean coercion in import_commission_transaction Co-Authored-By: Claude Sonnet 4.6 * fix(ibkr): cover trailing weekend gap and align qty with valid lots HistoricalBalancesSync: extend fill_gaps to account.current_anchor_date so days after the last equity summary row (e.g. Saturday/Sunday when a sync runs over the weekend) are also overridden rather than left with the materializer's stale total=cash value. HoldingsProcessor: replace separate quantity sum + weighted_cost_basis_for with a single valid_lots method that computes both from the same set of parseable lots. Previously a lot with a valid position but unparseable cost_basis_price was excluded from the cost basis calculation but still counted in quantity, producing inconsistent qty/cost_basis values. Co-Authored-By: Claude Sonnet 4.6 * review: address PR feedback on ibkr fix branch - Remove all "Fix N:" review-artifact comment labels - Add Sentry.capture_message for silenced anchor repair failure so it surfaces in production monitoring - Add Rails.logger.warn for zero/nil total rows skipped in HistoricalBalancesSync - Document normalize_to_last_trading_day holiday limitation and why gap-fill covers it - Rewrite two non-obvious comments to stand alone without the label prefix Co-Authored-By: Claude Sonnet 4.6 * style: remove alignment padding in balance_rows hash Co-Authored-By: Claude Sonnet 4.6 * fix(ibkr): address two P1 review findings - Allow zero and negative equity summary totals through HistoricalBalancesSync so fully-liquidated and margin accounts are not silently skipped (which would cause fill_gaps to propagate a stale non-zero total forward). - Remove normalize_to_last_trading_day from HoldingsProcessor: shifting weekend report_dates to Friday caused Balance::SyncCache#get_holdings_value to find no holdings on Saturday/Sunday (exact-date lookup), collapsing non_cash to zero — reintroducing the very oscillation the fix was meant to prevent. Co-Authored-By: Claude Sonnet 4.6 * test(ibkr): add tests for historical balances sync and data helpers - Add test case to verify non-cash balance calculation in historical balances sync - Add test case to ensure rows with unparseable or nil totals are skipped - Add new test file for IBKR data helpers * fix(ibkr): prevent date range overflow during historical sync Adjust the calculation of `last_date` in `HistoricalBalancesSync` to ensure it does not exceed the current anchor date or today's date. This prevents the sync process from attempting to fetch or process future dates, which was causing oscillations in weekend data. Also remove the conditional check for Sentry before capturing error messages in the account processor. --------- Co-authored-by: Claude Sonnet 4.6 --- .../ibkr_account/activities_processor.rb | 41 ++-- app/models/ibkr_account/data_helpers.rb | 3 + .../ibkr_account/historical_balances_sync.rb | 98 ++++++-- app/models/ibkr_account/holdings_processor.rb | 51 +++-- app/models/ibkr_account/processor.rb | 19 +- test/models/ibkr_account/data_helpers_test.rb | 57 +++++ .../historical_balances_sync_test.rb | 209 ++++++++++++++---- test/models/ibkr_account_processor_test.rb | 2 +- 8 files changed, 365 insertions(+), 115 deletions(-) create mode 100644 test/models/ibkr_account/data_helpers_test.rb diff --git a/app/models/ibkr_account/activities_processor.rb b/app/models/ibkr_account/activities_processor.rb index 3b33c224d..e2f906d07 100644 --- a/app/models/ibkr_account/activities_processor.rb +++ b/app/models/ibkr_account/activities_processor.rb @@ -13,15 +13,14 @@ class IbkrAccount::ActivitiesProcessor activities = (@ibkr_account.raw_activities_payload || {}).with_indifferent_access trades = Array(activities[:trades]) cash_transactions = Array(activities[:cash_transactions]) - @fee_transactions_count = 0 - trades_count = trades.sum { |trade| process_trade(trade.with_indifferent_access) ? 1 : 0 } - cash_transactions_count = cash_transactions.sum { |cash_transaction| process_cash_transaction(cash_transaction.with_indifferent_access) ? 1 : 0 } + trade_results = trades.map { |trade| process_trade(trade.with_indifferent_access) } + trades_count = trade_results.count { |r| r[:imported] } + fee_count = trade_results.sum { |r| r[:fees] } - { - trades: trades_count, - transactions: cash_transactions_count + @fee_transactions_count - } + cash_count = cash_transactions.sum { |t| process_cash_transaction(t.with_indifferent_access) ? 1 : 0 } + + { trades: trades_count, transactions: cash_count + fee_count } end private @@ -35,14 +34,14 @@ class IbkrAccount::ActivitiesProcessor end def process_trade(row) - return false unless supported_trade?(row) + return { imported: false, fees: 0 } unless supported_trade?(row) security = resolve_security(row) - return false unless security + return { imported: false, fees: 0 } unless security quantity = parse_decimal(row[:quantity]) native_price = parse_decimal(row[:trade_price]) - return false if quantity.nil? || native_price.nil? + return { imported: false, fees: 0 } if quantity.nil? || native_price.nil? buy_sell = row[:buy_sell].to_s.upcase signed_quantity = buy_sell == "SELL" ? -quantity.abs : quantity.abs @@ -65,11 +64,11 @@ class IbkrAccount::ActivitiesProcessor exchange_rate: parse_decimal(row[:fx_rate_to_base])&.to_f ) - import_commission_transaction(row, security, date) - true + fees = import_commission_transaction(row, security, date) ? 1 : 0 + { imported: true, fees: fees } rescue => e Rails.logger.error("IbkrAccount::ActivitiesProcessor - Failed to process trade #{row[:trade_id]}: #{e.message}") - false + { imported: false, fees: 0 } end def process_cash_transaction(row) @@ -114,9 +113,10 @@ class IbkrAccount::ActivitiesProcessor def import_commission_transaction(row, security, date) commission = parse_decimal(row[:ib_commission]) - return if commission.nil? || commission.zero? - currency = row.with_indifferent_access[:ib_commission_currency].to_s.upcase.presence || @ibkr_account.currency - ticker = security&.ticker || row.with_indifferent_access[:symbol] + return false if commission.nil? || commission.zero? + + currency = row[:ib_commission_currency].to_s.upcase.presence || @ibkr_account.currency + ticker = security&.ticker || row[:symbol] result = import_adapter.import_transaction( external_id: "ibkr_trade_fee_#{row[:trade_id]}", @@ -139,7 +139,7 @@ class IbkrAccount::ActivitiesProcessor } ) - @fee_transactions_count += 1 if result + !!result end def build_trade_name(ticker, signed_quantity) @@ -170,6 +170,7 @@ class IbkrAccount::ActivitiesProcessor type != "DIVIDENDS" || row[:conid].present? end + # supported_cash_transaction? ensures only known types reach here; no else branch needed def classify_cash_transaction(row, amount) type = row[:type].to_s.upcase.strip @@ -178,8 +179,6 @@ class IbkrAccount::ActivitiesProcessor amount.positive? ? [ "Contribution", -amount.abs ] : [ "Withdrawal", amount.abs ] when "DIVIDENDS" [ "Dividend", -amount.abs ] - else - [ nil, nil ] end end @@ -205,12 +204,12 @@ class IbkrAccount::ActivitiesProcessor end&.with_indifferent_access&.dig(:symbol) return holding_symbol if holding_symbol.present? - Array(@ibkr_account.raw_activities_payload&.dig("trades") || @ibkr_account.raw_activities_payload&.dig(:trades)).find do |trade| + activities = (@ibkr_account.raw_activities_payload || {}).with_indifferent_access + Array(activities[:trades]).find do |trade| trade.with_indifferent_access[:conid].to_s == conid.to_s end&.with_indifferent_access&.dig(:symbol) end - def fx_rate_available?(row) source_currency = extract_currency(row, fallback: nil) return false if source_currency.blank? diff --git a/app/models/ibkr_account/data_helpers.rb b/app/models/ibkr_account/data_helpers.rb index c3416d74b..6135557a2 100644 --- a/app/models/ibkr_account/data_helpers.rb +++ b/app/models/ibkr_account/data_helpers.rb @@ -9,6 +9,9 @@ module IbkrAccount::DataHelpers normalized = value.is_a?(String) ? value.delete(",").strip : value.to_s return nil if normalized.blank? || normalized == "-" + # Convert accounting parentheses notation: "(1234.56)" → "-1234.56" + normalized = "-#{normalized[1..-2]}" if normalized.start_with?("(") && normalized.end_with?(")") + BigDecimal(normalized) rescue ArgumentError nil diff --git a/app/models/ibkr_account/historical_balances_sync.rb b/app/models/ibkr_account/historical_balances_sync.rb index a7a0bc363..4d302d7ba 100644 --- a/app/models/ibkr_account/historical_balances_sync.rb +++ b/app/models/ibkr_account/historical_balances_sync.rb @@ -18,33 +18,93 @@ class IbkrAccount::HistoricalBalancesSync end private + def account ibkr_account.current_account end + def account_currency + ibkr_account.currency.to_s.upcase + end + def normalized_rows - @normalized_rows ||= Array(ibkr_account.raw_equity_summary_payload) - .filter_map do |row| - next unless row.is_a?(Hash) + @normalized_rows ||= begin + # Batch-load the materializer's already-computed balances so we can + # preserve its cash split rather than reading cash from the equity summary. + # Real IBKR Flex exports do not reliably include a cash/stock breakdown in + # EquitySummaryByReportDateInBase — only the total is consistently present. + existing_balances = account.balances + .where(currency: account.currency) + .index_by(&:date) - data = row.with_indifferent_access - currency = data[:currency].presence&.upcase - account_currency = ibkr_account.currency.to_s.upcase - next if currency.present? && currency != account_currency + trading_day_rows = Array(ibkr_account.raw_equity_summary_payload) + .filter_map do |row| + next unless row.is_a?(Hash) - date = parse_date(data[:report_date]) - total = parse_decimal(data[:total]) - cash = parse_decimal(data[:cash]) || BigDecimal("0") - next unless date && total + data = row.with_indifferent_access + currency = data[:currency].presence&.upcase - { - date: date, - total: total, - cash: cash, - non_cash: total - cash - } + # BASE_SUMMARY rows aggregate across all currencies — not a per-date balance + next if currency == "BASE_SUMMARY" + # Reject rows with an explicit wrong currency; absent currency is accepted + # (some Flex configurations omit it and the row is implicitly in base currency) + next if currency.present? && currency != account_currency + + date = parse_date(data[:report_date]) + next unless date + + total = parse_decimal(data[:total]) + if total.nil? + Rails.logger.warn( + "IbkrAccount::HistoricalBalancesSync - Skipping equity summary row with missing or unparseable total " \ + "for date=#{data[:report_date].inspect} account=#{account.id}" + ) + next + end + + # Use the materializer's cash_balance as ground truth for the cash split. + # This is consistent with how the reverse calculator handles present-day + # weekends and holidays — derive cash from holdings, not from IBKR's field. + cash = existing_balances[date]&.cash_balance || BigDecimal("0") + + { date: date, total: total, cash: cash, non_cash: total - cash } + end + .sort_by { |r| r[:date] } + + fill_gaps(trading_day_rows, existing_balances) + end + end + + # IBKR does not emit rows for weekends and some holidays. The reverse + # calculator fills those dates using only imported holdings — which only + # cover the current snapshot — so it cannot reconstruct the correct + # non-cash value for historical gap dates. We carry the most recent + # IBKR total forward to every missing calendar day and pair it with the + # materializer's already-correct cash for that date. + # + # The range is extended to the account's current anchor date so that days + # after the last equity summary row (e.g. a Saturday sync where the payload + # ends on Friday) are also covered and not left with the materializer's + # stale total=cash value. + def fill_gaps(rows, existing_balances) + return [] if rows.empty? + + by_date = rows.index_by { |r| r[:date] } + first_date = rows.first[:date] + anchor_date = [ account.current_anchor_date || Date.current, Date.current ].min + last_date = [ rows.last[:date], anchor_date ].max + + last_total = nil + (first_date..last_date).filter_map do |date| + if by_date[date] + last_total = by_date[date][:total] + by_date[date] + else + next unless last_total + cash = existing_balances[date]&.cash_balance || BigDecimal("0") + { date: date, total: last_total, cash: cash, non_cash: last_total - cash } end - .sort_by { |row| row[:date] } + end end def balance_rows @@ -52,7 +112,7 @@ class IbkrAccount::HistoricalBalancesSync normalized_rows.each_with_index.map do |row, index| previous_row = index.zero? ? nil : normalized_rows[index - 1] - start_cash_balance = previous_row ? previous_row[:cash] : row[:cash] + start_cash_balance = previous_row ? previous_row[:cash] : row[:cash] start_non_cash_balance = previous_row ? previous_row[:non_cash] : row[:non_cash] { diff --git a/app/models/ibkr_account/holdings_processor.rb b/app/models/ibkr_account/holdings_processor.rb index 9ff657f3f..8d5ebc704 100644 --- a/app/models/ibkr_account/holdings_processor.rb +++ b/app/models/ibkr_account/holdings_processor.rb @@ -8,8 +8,8 @@ class IbkrAccount::HoldingsProcessor def process return unless account.present? - grouped_positions.each_value do |group| - process_group(group) + grouped_positions.each do |(_, _, report_date), group| + process_group(group, report_date) end end @@ -28,32 +28,32 @@ class IbkrAccount::HoldingsProcessor data = position.with_indifferent_access next unless supported_position?(data) - symbol_key = data[:conid].presence || data[:symbol].presence || data[:security_id].presence + # conid is guaranteed present by supported_position?, so no fallbacks needed currency = extract_currency(data, fallback: @ibkr_account.currency) report_date = parse_date(data[:report_date]) || @ibkr_account.report_date || Date.current - key = [ symbol_key, currency, report_date ] + key = [ data[:conid], currency, report_date ] groups[key] ||= [] groups[key] << data end end - def process_group(rows) + def process_group(rows, report_date) sample = rows.first security = resolve_security(sample) return unless security - quantity = rows.sum { |row| parse_decimal(row[:position]) || BigDecimal("0") } - return if quantity.zero? - price = parse_decimal(sample[:mark_price]) - cost_basis = weighted_cost_basis_for(rows) - return unless price && cost_basis - - amount = quantity.abs * price + # quantity and cost_basis are derived from the same set of valid lots so + # they are always consistent — a lot with an unparseable cost_basis_price + # is excluded from both counts rather than inflating qty while shrinking basis. + aggregate = valid_lots(rows) + return unless price && aggregate + quantity = aggregate[:quantity] + cost_basis = aggregate[:cost_basis] + amount = quantity * price currency = extract_currency(sample, fallback: @ibkr_account.currency) - report_date = parse_date(sample[:report_date]) || @ibkr_account.report_date || Date.current - external_id = [ "ibkr", @ibkr_account.ibkr_account_id, sample[:conid].presence || security.ticker, report_date, currency ].join("_") + external_id = [ "ibkr", @ibkr_account.ibkr_account_id, sample[:conid], report_date, currency ].join("_") import_adapter.import_holding( security: security, @@ -61,7 +61,7 @@ class IbkrAccount::HoldingsProcessor amount: amount, currency: currency, date: report_date, - price: price || BigDecimal("0"), + price: price, cost_basis: cost_basis, external_id: external_id, source: "ibkr", @@ -70,22 +70,33 @@ class IbkrAccount::HoldingsProcessor ) end - def weighted_cost_basis_for(rows) + # Aggregates only the lots that have both a parseable position and cost_basis_price. + # Returns { quantity:, cost_basis: } so the caller uses a consistent lot set for + # both values — a lot skipped here is excluded from quantity too, preventing the + # case where qty covers more shares than the cost basis was computed from. + def valid_lots(rows) total_quantity = BigDecimal("0") total_cost = BigDecimal("0") rows.each do |row| - row_quantity = parse_decimal(row[:position]) + row_quantity = parse_decimal(row[:position]) row_cost_basis = parse_decimal(row[:cost_basis_price]) - return nil unless row_quantity && row_cost_basis + + unless row_quantity && row_cost_basis + Rails.logger.warn( + "IbkrAccount::HoldingsProcessor - Skipping lot with missing position or cost_basis_price " \ + "for conid=#{row[:conid].inspect}" + ) + next + end total_quantity += row_quantity.abs - total_cost += row_quantity.abs * row_cost_basis + total_cost += row_quantity.abs * row_cost_basis end return nil if total_quantity.zero? - total_cost / total_quantity + { quantity: total_quantity, cost_basis: total_cost / total_quantity } end def supported_position?(row) diff --git a/app/models/ibkr_account/processor.rb b/app/models/ibkr_account/processor.rb index 2fa30e4e2..0a924d4a7 100644 --- a/app/models/ibkr_account/processor.rb +++ b/app/models/ibkr_account/processor.rb @@ -6,21 +6,23 @@ class IbkrAccount::Processor end def process - return unless ibkr_account.current_account.present? + return unless account.present? update_account_balance! IbkrAccount::HoldingsProcessor.new(ibkr_account).process IbkrAccount::ActivitiesProcessor.new(ibkr_account).process repair_default_opening_anchor! - ibkr_account.current_account.broadcast_sync_complete + account.broadcast_sync_complete end private - def update_account_balance! - account = ibkr_account.current_account + def account + @account ||= ibkr_account.current_account + end + def update_account_balance! total_balance = ibkr_account.current_balance || ibkr_account.cash_balance || 0 cash_balance = ibkr_account.cash_balance || 0 @@ -34,7 +36,6 @@ class IbkrAccount::Processor end def repair_default_opening_anchor! - account = ibkr_account.current_account return unless account&.linked_to?("IbkrAccount") return unless account.has_opening_anchor? @@ -51,6 +52,12 @@ class IbkrAccount::Processor date: opening_anchor_entry.date ) - raise result.error if result.error + # Don't raise — broadcast_sync_complete must still run after a repair failure. + if result.error + Rails.logger.error( + "IbkrAccount::Processor - Failed to repair opening anchor for account #{account.id}: #{result.error}" + ) + Sentry.capture_message(result.error) + end end end diff --git a/test/models/ibkr_account/data_helpers_test.rb b/test/models/ibkr_account/data_helpers_test.rb new file mode 100644 index 000000000..bc8089ac1 --- /dev/null +++ b/test/models/ibkr_account/data_helpers_test.rb @@ -0,0 +1,57 @@ +# frozen_string_literal: true + +require "test_helper" + +class IbkrAccount::DataHelpersTest < ActiveSupport::TestCase + class TestHelper + include IbkrAccount::DataHelpers + + public :parse_decimal + end + + setup do + @helper = TestHelper.new + end + + test "parse_decimal returns nil for nil input" do + assert_nil @helper.parse_decimal(nil) + end + + test "parse_decimal returns nil for blank string" do + assert_nil @helper.parse_decimal("") + assert_nil @helper.parse_decimal(" ") + end + + test "parse_decimal returns nil for dash placeholder" do + assert_nil @helper.parse_decimal("-") + end + + test "parse_decimal converts parentheses notation to negative" do + assert_equal BigDecimal("-1234.56"), @helper.parse_decimal("(1234.56)") + end + + test "parse_decimal converts parentheses notation with comma-separated number" do + assert_equal BigDecimal("-1234.56"), @helper.parse_decimal("(1,234.56)") + end + + test "parse_decimal strips commas from positive numbers" do + assert_equal BigDecimal("1234.56"), @helper.parse_decimal("1,234.56") + end + + test "parse_decimal parses plain decimal string" do + assert_equal BigDecimal("3351.00"), @helper.parse_decimal("3351.00") + end + + test "parse_decimal returns nil for empty parentheses" do + assert_nil @helper.parse_decimal("()") + end + + test "parse_decimal returns nil for unclosed parenthesis" do + assert_nil @helper.parse_decimal("(123") + end + + test "parse_decimal returns nil for non-numeric string" do + assert_nil @helper.parse_decimal("N/A") + assert_nil @helper.parse_decimal("not_a_number") + end +end diff --git a/test/models/ibkr_account/historical_balances_sync_test.rb b/test/models/ibkr_account/historical_balances_sync_test.rb index 22d082284..66b5ee89b 100644 --- a/test/models/ibkr_account/historical_balances_sync_test.rb +++ b/test/models/ibkr_account/historical_balances_sync_test.rb @@ -21,33 +21,23 @@ class IbkrAccount::HistoricalBalancesSyncTest < ActiveSupport::TestCase current_balance: 3351, cash_balance: 1000.5, raw_equity_summary_payload: [ - { - currency: "CHF", - report_date: "2026-05-07", - cash: "900.50", - stock: "2300.50", - total: "3201.00" - }, - { - currency: "CHF", - report_date: "2026-05-08", - cash: "1000.50", - stock: "2350.50", - total: "3351.00" - } + { report_date: "2026-05-07", total: "3201.00" }, + { report_date: "2026-05-08", total: "3351.00" } ] ) @ibkr_account.ensure_account_provider!(@account) end - test "upserts historical balances without creating activity entries" do + # Seed an existing balance row as if the materializer already ran. + def seed_balance(date:, balance:, cash_balance:) + non_cash = balance - cash_balance @account.balances.create!( - date: Date.new(2026, 5, 7), - balance: 0, - cash_balance: 0, + date: date, + balance: balance, + cash_balance: cash_balance, currency: "CHF", - start_cash_balance: 0, - start_non_cash_balance: 0, + start_cash_balance: cash_balance, + start_non_cash_balance: non_cash, cash_inflows: 0, cash_outflows: 0, non_cash_inflows: 0, @@ -57,60 +47,183 @@ class IbkrAccount::HistoricalBalancesSyncTest < ActiveSupport::TestCase non_cash_adjustments: 0, flows_factor: 1 ) + end + + test "overrides total from IBKR equity summary while preserving materializer cash split" do + seed_balance(date: Date.new(2026, 5, 7), balance: 3000.00, cash_balance: 900.50) + seed_balance(date: Date.new(2026, 5, 8), balance: 3100.00, cash_balance: 1000.50) assert_no_difference "@account.entries.count" do IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync! end - first_balance = @account.balances.find_by!(date: Date.new(2026, 5, 7), currency: "CHF") - second_balance = @account.balances.find_by!(date: Date.new(2026, 5, 8), currency: "CHF") + first = @account.balances.find_by!(date: Date.new(2026, 5, 7), currency: "CHF") + second = @account.balances.find_by!(date: Date.new(2026, 5, 8), currency: "CHF") - assert_equal BigDecimal("3201.0"), first_balance.end_balance - assert_equal BigDecimal("900.5"), first_balance.end_cash_balance - assert_equal BigDecimal("2300.5"), first_balance.end_non_cash_balance + # Total overridden with IBKR's reported figure + assert_equal BigDecimal("3201.00"), first.end_balance + assert_equal BigDecimal("3351.00"), second.end_balance - assert_equal BigDecimal("3351.0"), second_balance.end_balance - assert_equal BigDecimal("1000.5"), second_balance.end_cash_balance - assert_equal BigDecimal("2350.5"), second_balance.end_non_cash_balance - assert_equal BigDecimal("900.5"), second_balance.start_cash_balance - assert_equal BigDecimal("2300.5"), second_balance.start_non_cash_balance + # Cash preserved from the materializer, not read from equity summary + assert_equal BigDecimal("900.50"), first.end_cash_balance + assert_equal BigDecimal("1000.50"), second.end_cash_balance + + # Non-cash = IBKR total - materializer cash + assert_equal BigDecimal("2300.50"), first.end_non_cash_balance + assert_equal BigDecimal("2350.50"), second.end_non_cash_balance end - test "accepts equity summary rows when stored account currency casing differs" do - @ibkr_account.update!(currency: "chf") + test "uses zero cash when no prior materializer balance exists for a date" do + # No existing balance rows — first-ever sync + IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync! + + balance = @account.balances.find_by!(date: Date.new(2026, 5, 7), currency: "CHF") + assert_equal BigDecimal("3201.00"), balance.end_balance + assert_equal BigDecimal("0"), balance.end_cash_balance + assert_equal BigDecimal("3201.00"), balance.end_non_cash_balance + end + + test "accepts rows without a currency field (Flex configs that omit the attribute)" do + @ibkr_account.update!( + raw_equity_summary_payload: [ + { report_date: "2026-05-07", total: "3201.00" } # no currency key + ] + ) + seed_balance(date: Date.new(2026, 5, 7), balance: 3000.00, cash_balance: 900.50) IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync! - first_balance = @account.balances.find_by!(date: Date.new(2026, 5, 7), currency: "CHF") - second_balance = @account.balances.find_by!(date: Date.new(2026, 5, 8), currency: "CHF") - - assert_equal BigDecimal("3201.0"), first_balance.end_balance - assert_equal BigDecimal("3351.0"), second_balance.end_balance + balance = @account.balances.find_by!(date: Date.new(2026, 5, 7), currency: "CHF") + assert_equal BigDecimal("3201.00"), balance.end_balance + assert_equal BigDecimal("900.50"), balance.end_cash_balance end - test "skips malformed equity summary rows and still imports valid rows" do + test "accepts rows when account currency casing differs from payload" do + @ibkr_account.update!(currency: "chf") + seed_balance(date: Date.new(2026, 5, 7), balance: 3000.00, cash_balance: 900.50) + + @ibkr_account.update!( + raw_equity_summary_payload: [ + { currency: "CHF", report_date: "2026-05-07", total: "3201.00" } + ] + ) + + IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync! + + balance = @account.balances.find_by!(date: Date.new(2026, 5, 7), currency: "CHF") + assert_equal BigDecimal("3201.00"), balance.end_balance + end + + test "skips BASE_SUMMARY rows" do + @ibkr_account.update!( + raw_equity_summary_payload: [ + { currency: "BASE_SUMMARY", report_date: "2026-05-07", total: "9999.00" }, + { currency: "CHF", report_date: "2026-05-07", total: "3201.00" } + ] + ) + + IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync! + + balance = @account.balances.find_by!(date: Date.new(2026, 5, 7), currency: "CHF") + assert_equal BigDecimal("3201.00"), balance.end_balance + end + + test "skips rows with a mismatched explicit currency" do + @ibkr_account.update!( + raw_equity_summary_payload: [ + { currency: "USD", report_date: "2026-05-07", total: "9999.00" }, + { currency: "CHF", report_date: "2026-05-07", total: "3201.00" } + ] + ) + + IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync! + + balance = @account.balances.find_by!(date: Date.new(2026, 5, 7), currency: "CHF") + assert_equal BigDecimal("3201.00"), balance.end_balance + end + + test "skips malformed rows and still imports valid ones" do @ibkr_account.update!( raw_equity_summary_payload: [ nil, "bad-row", [], - { - currency: "CHF", - report_date: "2026-05-09", - cash: "1100.50", - total: "3400.00" - } + { report_date: "2026-05-11", total: "3400.00" } ] ) + seed_balance(date: Date.new(2026, 5, 11), balance: 3300.00, cash_balance: 1100.50) assert_nothing_raised do IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync! end - balance = @account.balances.find_by!(date: Date.new(2026, 5, 9), currency: "CHF") + balance = @account.balances.find_by!(date: Date.new(2026, 5, 11), currency: "CHF") + assert_equal BigDecimal("3400.00"), balance.end_balance + assert_equal BigDecimal("1100.50"), balance.end_cash_balance + end - assert_equal BigDecimal("3400.0"), balance.end_balance - assert_equal BigDecimal("1100.5"), balance.end_cash_balance - assert_equal BigDecimal("2299.5"), balance.end_non_cash_balance + test "fills weekend and holiday gaps by carrying forward the last IBKR total with materializer cash" do + # Simulate the real-world situation: IBKR has no weekend rows, and historical + # holdings only cover the current snapshot so the materializer writes total=cash + # for gap dates. HistoricalBalancesSync must write the correct total for those days. + @ibkr_account.update!( + raw_equity_summary_payload: [ + { report_date: "2026-05-08", total: "3351.00" }, # Friday + # Saturday May 9 and Sunday May 10 absent — IBKR never sends them + { report_date: "2026-05-11", total: "3400.00" } # Monday + ] + ) + + # Materializer computed correct cash for all dates; wrong total for the weekend + seed_balance(date: Date.new(2026, 5, 8), balance: 3351.00, cash_balance: 900.50) + seed_balance(date: Date.new(2026, 5, 9), balance: 900.50, cash_balance: 900.50) # wrong total + seed_balance(date: Date.new(2026, 5, 10), balance: 900.50, cash_balance: 900.50) # wrong total + seed_balance(date: Date.new(2026, 5, 11), balance: 3400.00, cash_balance: 910.00) + + IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync! + + saturday = @account.balances.find_by!(date: Date.new(2026, 5, 9), currency: "CHF") + sunday = @account.balances.find_by!(date: Date.new(2026, 5, 10), currency: "CHF") + + # Total corrected to Friday's IBKR total; cash preserved from materializer + assert_equal BigDecimal("3351.00"), saturday.end_balance + assert_equal BigDecimal("900.50"), saturday.end_cash_balance + assert_equal BigDecimal("2450.50"), saturday.end_non_cash_balance + + assert_equal BigDecimal("3351.00"), sunday.end_balance + assert_equal BigDecimal("900.50"), sunday.end_cash_balance + assert_equal BigDecimal("2450.50"), sunday.end_non_cash_balance + end + + test "skips rows with missing or unparseable total" do + @ibkr_account.update!( + raw_equity_summary_payload: [ + { report_date: "2026-05-06", total: "N/A" }, # unparseable string — before first valid date + { report_date: "2026-05-07", total: nil }, # nil total + { report_date: "2026-05-08", total: "3351.00" } # valid + ] + ) + + IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync! + + # Gap-fill starts at the first valid trading day (May 8), so pre-range + # dates with bad totals must not produce any balance row. + assert_nil @account.balances.find_by(date: Date.new(2026, 5, 6), currency: "CHF") + assert_nil @account.balances.find_by(date: Date.new(2026, 5, 7), currency: "CHF") + assert_not_nil @account.balances.find_by(date: Date.new(2026, 5, 8), currency: "CHF") + end + + test "writes balance row with zero total for fully liquidated dates" do + @ibkr_account.update!( + raw_equity_summary_payload: [ + { report_date: "2026-05-07", total: "0" }, + { report_date: "2026-05-08", total: "3351.00" } + ] + ) + + IbkrAccount::HistoricalBalancesSync.new(@ibkr_account).sync! + + balance = @account.balances.find_by!(date: Date.new(2026, 5, 7), currency: "CHF") + assert_equal BigDecimal("0"), balance.end_balance end end diff --git a/test/models/ibkr_account_processor_test.rb b/test/models/ibkr_account_processor_test.rb index 94555bb61..7e2254f61 100644 --- a/test/models/ibkr_account_processor_test.rb +++ b/test/models/ibkr_account_processor_test.rb @@ -189,7 +189,7 @@ class IbkrAccountProcessorTest < ActiveSupport::TestCase assert_not_nil holding assert_equal BigDecimal("30"), holding.qty - assert_equal BigDecimal("123.1667"), holding.cost_basis + assert_in_delta BigDecimal("123.1667"), holding.cost_basis, BigDecimal("0.0001") end test "processor repairs default opening anchor after importing activity entries" do From bc9f13059ad6e8b7f609db495cc3c599edb31196 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Juan=20Jos=C3=A9=20Mata?= Date: Mon, 18 May 2026 21:46:28 +0200 Subject: [PATCH 2/2] Bump version by hand --- .sure-version | 2 +- charts/sure/Chart.yaml | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/.sure-version b/.sure-version index 6d64966b7..38d939e70 100644 --- a/.sure-version +++ b/.sure-version @@ -1 +1 @@ -0.7.1-alpha.9 +0.7.1-alpha.10 diff --git a/charts/sure/Chart.yaml b/charts/sure/Chart.yaml index 650c52a72..412a3c29e 100644 --- a/charts/sure/Chart.yaml +++ b/charts/sure/Chart.yaml @@ -2,8 +2,8 @@ apiVersion: v2 name: sure description: Official Helm chart for deploying the Sure Rails app (web + Sidekiq) on Kubernetes with optional HA PostgreSQL (CloudNativePG) and Redis. type: application -version: 0.7.1-alpha.9 -appVersion: "0.7.1-alpha.9" +version: 0.7.1-alpha.10 +appVersion: "0.7.1-alpha.10" kubeVersion: ">=1.25.0-0"