mirror of
https://github.com/we-promise/sure.git
synced 2026-09-09 00:24:15 +00:00
210de793c4eacd7f9d8d33664a191ed330c2eb7f
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 buzzromainandClaude Opus 5
|
07524d0d84 |
fix(holdings): a transfer must not set a cost basis (#3154)
* fix(holdings): a transfer must not set a cost basis calculate_avg_cost sums every trade with a positive quantity, so an asset moved in from elsewhere is counted as bought on the day it arrived. A coin acquired at 30k and transferred in at 60k reports a cost of 60k and no gain at all — a number that looks authoritative and is wrong. Nothing here can know what a transferred asset cost: the purchase happened somewhere this app never saw. Leaving the cost unknown is what the method already does when it has nothing to work from, and for the same stated reason the fallback to market price was removed from it: "Previously this fell back to current market price, which was misleading." Two things it would be easy to get wrong, and both are covered: - **One transfer makes the whole position unknown**, not just its own row. Averaging the purchases alone and applying that to every unit is the same fabrication in a quieter form: buy one at 30k, receive one, and the position reports 30k a unit for two units that did not cost that. - **Unlabelled purchases are preserved.** `!=` is NULL for a row with no label, so a naive exclusion would drop the ordinary trades that carry none — which is most of them. Hence IS DISTINCT FROM. Balances and value are unaffected: they come from holdings, which providers import from the position itself rather than from trade history. This reaches every integration that labels a movement as a transfer. Questrade journals already did; the self-custody wallets do as of #3153. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(holdings): stop a stored figure outranking the transfer guard Review on #3154, and the reviewers were right that the first pass only covered half the path. `Holding#avg_cost` returns a stored `cost_basis` before it ever calls `calculate_avg_cost`, so the transfer guard was protecting only holdings that had nothing stored. Worse, the stored value was itself wrong: both calculators counted every positive-quantity trade toward the running average, transfers included, and the materializer persisted that as a `calculated` basis. A coin bought elsewhere at 30k and moved in at 60k reported no gain at all, and said so with a figure that looks derived. Fixed in the write path rather than the read one. Adding an `exists?` per holding to `avg_cost` would have reintroduced exactly the N+1 the stored value exists to avoid; clearing the stored value instead lets the read path fall through to the guard that was already there. Both calculators now exclude transfers from the average and mark the security's basis unknown — the forward one for good, the reverse one from the transfer's date onward, since the purchases before it still stand on their own. `cost_basis_unknown` is carried separately from a nil `cost_basis` because the materializer treats them differently: nil means "nothing computed, leave what is there", unknown means "this cannot be known, clear what is there". A `manual` or `provider` basis survives. That is somebody asserting what the position cost them, which is precisely the thing the app cannot derive for a transfer. The migration clears figures already stored. Positions heal on the next materialization anyway, but a manual or disconnected account may not materialize again for a long time, and the wrong number is not visibly wrong. The regression test materializes first and relabels after, because that is the case that matters: a position already carrying a figure worked out before anyone knew the movement was a transfer. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016GTNba5qE5NwzaHzbp27ye * fix(holdings): clear a transferred basis that recorded no source Follow-up on the same review. `load_existing_holdings_map` loaded holdings that were locked, sourced, or provider-owned — so a row carrying a `cost_basis` with no `cost_basis_source` was invisible to it. The clearing then saw no existing holding and left the figure standing, which meant the rows least able to justify the number they hold were the ones that kept it. The migration takes `[7.2]` to match `schema.rb` and the other 398 migrations, rather than the `[8.0]` I had written. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016GTNba5qE5NwzaHzbp27ye * fix(holdings): renumber the migration off a colliding version `20260825120000` is already taken by `add_consumed_amount_to_goals` on the goals stack. Two migrations sharing a version is not a merge conflict — `schema_migrations` is keyed by it, so whichever landed second would be recorded as already run and skipped in silence. For a data migration that means transferred positions quietly keeping the cost basis this branch exists to clear. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016GTNba5qE5NwzaHzbp27ye * docs(holdings): say why the basis guard keys on one label, not the set #3192 landed Trade::INTERNAL_MOVEMENT_LABELS in this file, next to the TRANSFER_LABEL this branch adds. Both rest on ownership being preserved, so two constants sitting together invite the question of why the basis guard does not simply use the broader one. It could, and that would be a behaviour change: the sweep labels would start clearing a cost basis too. Nothing has shown a sweep landing on a security, and widening a guard that erases figures on the strength of a guess is the wrong direction, so it stays narrow and now says so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016GTNba5qE5NwzaHzbp27ye --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com> |
||
|
Abhinav Dhiman
|
e3d46021c2 |
fix: memory leak in sidekiq (#1940)
* perf(sync): reduce per-job memory peak in Balance/Holding materialization
Profiling of SyncJob (StackProf object mode + Sidekiq memory middleware)
showed peaks of 600k-1.1M live heap slots per job and ~196k retained
ActiveModel::Attribute::FromUser objects post-GC, driven by full-history
in-memory accumulation in the balance/holding sync pipeline.
Changes:
- Replace Holding.new / Balance.new in calculators with lightweight
Struct-based HoldingData / BalanceData. Skips AR attribute sets,
belongs_to proxies, dirty tracking, type casting, and callbacks
that were never used (upsert_all bypasses validations/callbacks
anyway). Eliminates ~30% of allocations and the bulk of retained
ActiveModel::Attribute::* instances.
- Build upsert payloads directly from struct fields instead of
Holding/Balance#attributes.slice(...).
- Batch upsert_all in PERSIST_BATCH_SIZE (2,000) slices in both
Balance::Materializer and Holding::Materializer so the intermediate
attribute-hash array is bounded instead of holding the full
multi-year history alongside the calculator output.
- Replace account.holdings.reload with account.holdings.reset in
Holding::Materializer. Same cache invalidation, no eager re-query;
the next consumer (Balance::SyncCache) loads on demand.
- Mutate entries in place in Balance::SyncCache#converted_entries
instead of Entry#dup. The instances are scoped to the throwaway
sync-cache and never persisted, so dup'ing was producing tens of
thousands of unused FromUser/FromDatabase attribute wrappers per
sync.
All persist paths run inside the existing Balance.transaction wrapper,
so batched upserts retain transactional atomicity. No production caller
of Balance::SyncCache or Holding::Materializer reuses the affected
instances outside the materializer's lifetime.
Test coverage: balance/{sync_cache,materializer,forward_calculator,
reverse_calculator} and holding/{materializer,forward_calculator,
reverse_calculator} plus account/syncer and sync (82 runs, 2,548
assertions, 0 failures).
* refactor(holding): stream materializer upserts to bound peak memory
Replace full-array accumulation + each_slice in Materializer#persist_holdings
with two flush-on-fill buffers (holdings_buffer_to_upsert_with_cost /
holdings_buffer_to_upsert_without_cost) that upsert and clear at
PERSIST_BATCH_SIZE, keeping peak RSS bounded to ~2x batch size.
Also add assert_not_nil guards in ReverseCalculatorTest before
dereferencing calculated.find results to surface clear failures
instead of NoMethodError.
* refactor(balance): promote BalanceData to Balance namespace and document sync mutation safety
- Extract Balance::BalanceData struct into its own file (app/models/balance/balance_data.rb)
so it is discoverable without knowing it lived inside BaseCalculator
- Remove inline Struct definition from Balance::BaseCalculator; update build_balance
to reference Balance::BalanceData explicitly (required because class Foo::Bar syntax
does not nest Foo in constant lookup)
- Add comment to SyncCache#converted_entries clarifying that to_a materialises
independent AR instances with no identity map active, making in-place mutation safe
- Update all Balance::BaseCalculator::BalanceData references in materializer_test
* test(balance): use to_h instead of attributes on BalanceData struct in waypoint test
* perf(balance,market_data): replace sort_by + first/last with minmax_by and push account-entry join to SQL
- Balance::Materializer#purge_stale_balances: replace sort_by(&:date) + first/last with minmax_by(&:date) to avoid full sort when only min/max are needed
- MarketDataImporter: replace Entry.group(:account_id).minimum(:date) (which loads all account IDs into a Hash) with a LEFT JOIN subquery that computes MIN(date) per account in SQL and exposes it as first_entry_date on the Account relation
* test(balance): use BalanceData struct in materializer purge test
* fix(holding): carry cost_basis forward onto gap-filled dates
|
||
|
CrossDrainandClaude Sonnet 4.6
|
b8ebb24e8b |
fix(holdings): carry provider cost_basis forward to calculated rows past snapshot date (#1818)
* fix(holdings): carry provider cost_basis forward to calculated rows Providers like IBKR Flex emit holdings on report_date and only include trades within the query window. The reverse calculator + gapfill therefore produces rows past report_date with nil cost_basis, even though the provider supplied a basis on the snapshot. That nil basis silently blanks `Trend`, the Reports "Total Return" card, the Top Holdings return column, and Gains by Tax Treatment, because every one of them gates on `holding.avg_cost`. When a calculated row would otherwise have no usable cost_basis, backfill it with the most recent provider-supplied cost_basis for the same (security, currency) on or before the holding date. Existing calculated/manual values are preserved (they outrank a provider carry-forward), and existing provider carry-forwards are refreshed when a newer snapshot supersedes them. * - Fix currency mismatch: provider snapshots were keyed by (security_id, currency) but calculated rows use account currency while IBKR provider rows use the security's native currency (e.g., USD vs EUR). Now keyed by security_id only; carry_forward_provider_cost_basis converts via Money#exchange_to at the snapshot date (same convention as ReverseCalculator for trade prices), with a ConversionError fallback. - Trim long inline comment to three lines - Fix safe-nav inconsistency: existing.cost_basis.positive? -> existing&.cost_basis&.positive? - Add test: refreshes stale carry-forward when a newer provider snapshot arrives - Add test: carry-forward is a no-op for forward-strategy accounts with no provider holdings * fix(holdings): prevent overwriting zero-valued manual cost basis Ensure that manual cost basis entries with a value of zero (e.g., for free shares) are not overwritten by provider carry-forward values during materialization. Additionally, updated the logic to allow zero-valued manual or calculated cost bases to be preserved, and added tests to verify currency conversion and error handling during cost basis carry-forward. * refactor(holdings): allow zero-valued cost basis in provider snapshots Remove the filter that restricted provider cost basis snapshots to values greater than zero. This ensures that manual cost basis entries with a value of zero (e.g., for free shares) are correctly captured and available for carry-forward logic. * perf(holdings): optimize provider cost basis snapshot lookup Filter provider cost basis snapshots by the security IDs present in the current holdings set to reduce the amount of data loaded into memory. * refactor(holdings): move PortfolioCache FX fix to dedicated branch Remove date-accurate exchange rate fix from this branch — it has been split into fix/portfolio-cache-historical-fx-rate to keep concerns separate. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * revert(portfolio_cache): restore date-accurate FX in get_price |
||
|
Gian-Reto Tarnutzer
|
ce5d7dd736 |
Add Interactive Brokers Provider (#1722)
* Display multi-currency holdings correctly * Implement IBKR provider * Fix: Use historical exchange rate for historical prices * Add brokerage exchange rate for trades * Sync historical balances from IBKR * Add logos in activity history * Fix privacy mode blur in account view * Improve IBKR XML Flex report parser errors |
||
|
wps260
|
c294cbf54b |
Performance improvements in holding calculation pipeline (#1579)
* Performance improvements in holding calculation pipeline
Investment accounts with large histories were pegging CPU at 100% during
sync. Root cause was a cluster of quadratic and superlinear algorithms in
the inner holding calculation loop. All are replaced with O(1) hash lookups
built from single-pass indexes over the already-loaded data.
Holding::PortfolioCache - load_prices:
Three O(SxN) patterns inside the per-security loop:
1. DB prices: `security.prices.where(...)` fired one SQL query per
security (N+1). Replaced with a single bulk query before the loop:
Security::Price.where(security_id: ..., date: ...).group_by(&:security_id)
70 securities -> 70 queries becomes 1.
2. Trade prices: `trades.select { |t| t.entryable.security_id == id }`
scanned the full trades array for every security - O(SxT). Replaced
with trades_by_security_id, pre-indexed once from the loaded array.
3. Holding prices: `holdings.select { |h| h.security_id == id }` - same
O(SxH) pattern. Replaced with holdings_by_security_id.
Prices are now indexed into prices_by_date and prices_by_date_and_source
hashes during load_prices, making get_price O(1) instead of scanning the
flat prices array on every lookup.
Holding::PortfolioCache - get_trades / get_price:
- get_trades(date:): `trades.select { |t| t.date == date }` (O(T) scan)
replaced with trades_by_date hash (O(1)).
- get_price: two `prices.select { p.date == date ... }.min_by` linear
scans replaced with direct hash lookups into prices_by_date and
prices_by_date_and_source.
Holding::PortfolioCache - collect_unique_securities:
`holdings.map(&:security)` traversed the security association on every
holding record (N+1 if not preloaded). Replaced with a pluck of
security_ids followed by a single Security.where(id: ...) batch load.
Holding::ForwardCalculator / ReverseCalculator:
`holdings += build_holdings(...)` allocated a new array copy on every
iteration - O(N) per day x thousands of days = O(D^2) total allocations.
Replaced with holdings.concat(...) which appends in place, O(1).
Holding::ReverseCalculator - precompute_cost_basis:
Old: walked every date from account.start_date to Date.current (O(D)),
writing a cost_basis entry for every security on every date. For an
account with 2 trades over 9,250 days this wrote ~18,500 hash entries
and consumed the full date range in the outer loop regardless of trade
density.
New: walks only buy trades (O(T)), appending one [date, avg_cost]
snapshot per trade. cost_basis_for binary-searches the sparse snapshot
array - O(log T) per lookup. Memory drops from O(DxS) to O(T).
Holding::Gapfillable:
`security_holdings.find { |h| h.date == date }` was called on every
date in the gapfill range - O(H) per date, O(HxD) total. Replaced with
security_holdings.index_by(:date) built once before the loop, making
each date lookup O(1).
Holding::Materializer - purge_stale_holdings:
`account.entries.trades.map { |entry| entry.entryable.security_id }.uniq`
loaded all trade entry records into Ruby then traversed the entryable
association on each (N+1). Replaced with account.trades.pluck(:security_id).uniq
(single SQL query returning only the IDs).
In testing, these changes were able to reduce sync time of an account with
25 years of history and 70 securities from about 90 minutes down to under
3 minutes.
* Lint fix
* Lint fix
* addressing the open review nits I agreed with:
* return dup'd arrays from PortfolioCache#get_trades so callers can't mutate memoized cache state
* use the precomputed security-id indexes in collect_unique_securities
* keep security-id dedupe in SQL via distinct.pluck(:security_id)
* tighten the DB price preload to select only needed columns
* harden cost-basis assertions with assert_in_delta
* Back out unnecessary AI slop
* Add back dup to trades array returned from memoized hash
trades_by_date[date] returns a live reference into the memoized hash.
Any caller that mutates the result would silently corrupt the cache for
subsequent calls on the same date within the same sync run. Add .dup to
return a shallow copy, matching the safety of the original select path.
|
||
|
Anas LimouriandJuan José Mata
|
a90f9b7317 |
Add CoinStats exchange portfolio sync and normalize linked investment charts (#1308)
* [FEATURE] Add CoinStats exchange portfolios and normalize linked investment charts * [BUGFIX] Fix CoinStats PR regressions * [BUGFIX] Fix CoinStats PR review findings * [BUGFIX] Address follow-up CoinStats PR feedback * [REFACTO] Extract CoinStats exchange account helpers * [BUGFIX] Batch linked CoinStats chart normalization * [BUGFIX] Fix CoinStats processor lint --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
Serge LandClaude Opus 4.6
|
a92fd3b3e8 |
feat: Enhance holding detail drawer with live price sync and enriched overview (#1086)
* Feat: Implement manual sync prices functionality and enhance holdings display * Feat: Enhance sync prices functionality with error handling and update UI components * Feat: Update sync prices error handling and enhance Spanish locale messages * Fix: Address CodeRabbit review feedback - Set fallback @provider_error when prices_updated == 0 so turbo stream never fails silently without a visible error message - Move attr_reader :provider_error to class header in Price::Importer for conventional placement alongside other attribute declarations - Precompute @last_price_updated in controller (show + sync_prices) instead of running a DB query directly inside ERB templates Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix: Replace bare rescue with explicit exception handling in turbo stream view Bare `rescue` silently swallows all exceptions, making debugging impossible. Match the pattern already used in show.html.erb: rescue ActiveRecord::RecordInvalid explicitly, then catch StandardError with logging (message + backtrace) before falling back to the unknown label. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix: Update test assertion to expect actual provider error message The stub returns "Yahoo Finance rate limit exceeded" as the provider error. After the @provider_error fallback fix, the controller now correctly surfaces the real provider error when present (using .presence || fallback), so the flash[:alert] is the actual error string, not the generic fallback. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix: Assert scoped security_ids in sync_prices materializer test Replace loose stub with constructor expectation to verify that Balance::Materializer is instantiated with the single-security scope. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix: Assert holding remap in remap_security test Add assertion that @holding.security_id is updated to the target security after remap, covering the core command outcome. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Fix: CI test failure - Update disconnect external assistant test to use env overrides --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
|
dd991fa339 |
Add Coinbase exchange integration with CDP API support (#704)
* **Add Coinbase integration with item and account management** - Creates migrations for `coinbase_items` and `coinbase_accounts`. - Adds models, controllers, views, and background tasks to support account linking, syncing, and transaction handling. - Implements Coinbase API client and adapter for seamless integration. - Supports ActiveRecord encryption for secure credential storage. - Adds UI components for provider setup, account management, and synchronization. * Localize Coinbase-related UI strings, refine account linking for security, and add timeouts to Coinbase API requests. * Localize Coinbase account handling to support native currencies (USD, EUR, GBP, etc.) across balances, trades, holdings, and transactions. * Improve Coinbase processing with timezone-safe parsing, native currency support, and immediate holdings updates. * Improve trend percentage formatting and enhance race condition handling for Coinbase account linking. * Fix log message wording for orphan cleanup * Ensure `selected_accounts` parameter is sanitized by rejecting blank entries. * Add tests for Coinbase integration: account, item, and controller coverage - Adds unit tests for `CoinbaseAccount` and `CoinbaseItem` models. - Adds integration tests for `CoinbaseItemsController`. - Introduces Stimulus `select-all` controller for UI checkbox handling. - Localizes UI strings and logging for Coinbase integration. * Update test fixtures to use consistent placeholder API keys and secrets * Refine `coinbase_item` tests to ensure deterministic ordering and improve scope assertions. * Integrate `SyncStats::Collector` into Coinbase syncer to streamline statistics collection and enhance consistency. * Localize Coinbase sync status messages and improve sync summary test coverage. * Update `CoinbaseItem` encryption: use deterministic encryption for `api_key` and standard for `api_secret`. * fix schema drift * Beta labels to lower expectations --------- Co-authored-by: luckyPipewrench <luckypipewrench@proton.me> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
LPWandluckyPipewrench
|
8b8ac705f6 |
Correct brokerage cash calculation for SimpleFIN investment accounts (#710)
* Refactor: Enhance cash balance calculation and holdings management with money market classification and provider-sourced data handling * Fix: Clear fixture holdings in test to ensure clean creation and update raw_holdings_payload format --------- Co-authored-by: luckyPipewrench <luckypipewrench@proton.me> |
||
|
LPWandJosh Waldrep
|
bbaf7a06cc |
Add cost basis source tracking with manual override and lock protection (#623)
* Add cost basis tracking and management to holdings - Added migration to introduce `cost_basis_source` and `cost_basis_locked` fields to `holdings`. - Implemented backfill for existing holdings to set `cost_basis_source` based on heuristics. - Introduced `Holding::CostBasisReconciler` to manage cost basis resolution logic. - Added user interface components for editing and locking cost basis in holdings. - Updated `materializer` to integrate reconciliation logic and respect locked holdings. - Extended tests for cost basis-related workflows to ensure accuracy and reliability. * Fix cost basis calculation in holdings controller - Ensure `cost_basis` is converted to decimal for accurate arithmetic. - Fix conditional check to properly validate positive `cost_basis`. * Improve cost basis validation and error handling in holdings controller - Allow zero as a valid cost basis for gifted/inherited shares. - Add error handling with user feedback for invalid cost basis values. --------- Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com> |
||
|
LPWandJosh Waldrep
|
fa78e1d292 |
Improve handling of cost_basis during holding materialization and display (#619)
- Refactored `persist_holdings` to separate and conditionally upsert holdings with and without cost_basis. - Updated `avg_cost` logic to treat 0 cost_basis as unknown and return nil when cost_basis cannot be determined. - Modified trend and investment calculation to exclude holdings with unknown cost_basis. - Adjusted `average_cost` formatting to handle nil values in API responses and views. - Added comprehensive tests to ensure cost_basis preservation and fallback behavior. - Localized `unknown` label for display when cost_basis is unavailable. Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com> |
||
|
soky srm
|
6ebe8da928 |
Add investment tracking to expenses (#381)
* Add investment tracking to expenses Add new sections to dashboard and reporting around investments. * Create investment-integration-assessment.md * Delete .claude/settings.local.json Signed-off-by: soky srm <sokysrm@gmail.com> * Category trades * Simplify * Simplification and test fixes * FIX merge * Update views * Update 20251125141213_add_category_to_trades.rb * FIX tests * FIX statements and account status * cleanup * Add default cat for csv imports * Delete docs/roadmap/investment-integration-assessment.md Signed-off-by: soky srm <sokysrm@gmail.com> * Update trend calculation Use already existing column cost basis for trend calculation - Current value: qty * price (already stored as amount) - Cost basis total: qty * cost_basis - Unrealized gain: current value - cost basis total Fixes N+1 query also --------- Signed-off-by: soky srm <sokysrm@gmail.com> |
||
|
Zach Gollwitzer
|
8db95623cf |
Handle holding quantity generation for reverse syncs correctly when not all holdings are generated for current day (#2417)
* Handle reverse calculator starting portfolio generation correctly * Fix current_holdings to handle different dates and hide zero quantities - Use DISTINCT ON to get most recent holding per security instead of assuming same date - Filter out zero quantity holdings from UI display - Maintain cash display regardless of zero balance - Use single efficient query with proper Rails syntax * Continue to process holdings even if one is not resolvable * Lint fixes |
||
|
Zach Gollwitzer
|
10f255a9a9 |
Clarify backend data pipeline naming concepts (importers, processors, materializers, calculators, and syncers) (#2255)
* Rename MarketDataSyncer to MarketDataImporter * Materializers * Importers * More reference replacements |