Files
sure/app/models/questrade_account/activities_processor.rb
Jestin Palamuttam 5803023fa7 feat(provider): add native Questrade brokerage provider integration (#2534)
* Add native Questrade brokerage provider integration

Adds a per-family Questrade provider so users can sync their Questrade
investment accounts (TFSA, FHSA, RRSP, margin, etc.) directly via
Questrade's free personal API, with no paid aggregator.

- OAuth2 refresh-token flow with single-use token rotation, persisted
  under a row lock. Tokens self-renew on each sync; the connected panel
  lets users paste a fresh token if a connection goes stale (no need to
  disconnect and re-link).
- Imports accounts, balances, positions and activities; multi-currency
  holdings with per-currency cash holdings; Norbert's Gambit journals.
- New-account and link-existing-account flows, settings card with
  desktop-only setup steps, and connect/update/disconnect.
- Restricted to Investment account types. Registered in the provider
  connection-status registry with a syncable scope so it participates in
  nightly family sync.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: linting error

* fix: refresh token encrypted

The OAuth token exchange passed the single-use refresh token as a GET
query parameter, so it could leak into URL-based logs (Sentry
breadcrumbs, APM spans, debug output). Switch to POST with a
form-encoded body (RFC 6749 3.2) so the credential stays out of URLs.
Verified Questrade's token endpoint accepts POST (returns 400 for a bad
token, not 405). Adds a test asserting the token travels in the body.

* Address PR review: authz, data integrity, retries, logging

Batch of fixes from the automated PR review:

- Require admin for all mutating/linking Questrade actions, and gate
  existing-account linking through accessible_accounts + write permission
  (was only Current.family scoped).
- Clear requires_update when a fresh token is accepted; use a real 302
  redirect (not 422) on full-page failures.
- Require refresh_token on all saves (not just create) unless the item is
  scheduled for deletion.
- Migrations target Rails 7.2; questrade_items state columns are NOT NULL.
- Background activity dedup keys on Questrade fields (matches the importer)
  so multiple activities no longer collapse to one.
- Persist the normalized account payload; date-scope synthetic cash
  holdings so daily history is not overwritten.
- Retry 429/5xx via a RetryableResponseError instead of hard-failing.
- Route provider error bodies to DebugLogEntry instead of Rails.logger /
  exception messages, so payloads do not leak into application logs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Address PR review: atomic linking, sync health, retry loop, USD cash

- Wrap account creation + provider linking (+ sync_start_date) in a
  transaction in both link paths so a link failure rolls back the orphan
  account.
- Surface per-account process/schedule failures in the item sync health
  instead of always reporting healthy.
- Always stamp last_activities_sync once the background fetch completes,
  so legitimately empty accounts stop being re-queued every sync.
- Treat only the account-currency (CAD) balance as primary cash; other
  currencies (e.g. USD) now surface as separate cash holdings instead of
  being hidden as primary.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Address PR review: serialize token exchange, real processor tests

- Single-use token race: the SDK now wraps every token exchange (initial
  and 401 re-auth) in a model-supplied lock that reloads and spends the
  freshest persisted token (provided.rb#synchronize_exchange). Two
  concurrent syncs/jobs can no longer double-spend the same refresh token.
  Adds a test asserting the exchange runs inside the lock with the fresh
  token.
- Replace the all-skipped QuestradeAccount processor test stubs with real
  fixture-backed tests covering balance anchoring, holdings import, and
  Buy-trade import (plus blank-symbol / blank-type guards).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Fix indentation of spliced Questrade schema blocks

The manually added questrade_accounts/questrade_items create_table blocks
sat at column 0 instead of the file 2-space indent, so rubocop flagged
them as inconsistent. Re-indent to match the rest of the schema.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Include currency and type in the Questrade activity merge key

Two activities that differ only by currency or type could collapse to a
single row in merge_activities. Add both fields to activity_key in the
importer and the background fetch job so multi-currency imports dedup
correctly.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Address review: infer account currency, Encryptable, safer flag clear

From @jjmata's review:

- Currency: QuestradeAccount#upsert_from_questrade! no longer hardcodes CAD
  for every account. upsert_balances! now infers the home currency from the
  per-currency balances (the currency holding the cash wins, ties broken by
  total equity, default CAD) so USD-denominated accounts are labelled USD and
  match the right combinedBalances anchor. Adds tests for USD and CAD cases.
- QuestradeItem now includes the shared Encryptable concern instead of
  reimplementing encryption_ready? inline.
- QuestradeActivitiesFetchJob#clear_pending_flag is now best-effort so it can
  never mask (and swallow) the original error in perform's rescue.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Fix review issues: safe_return_to_path, DebugLogEntry, financial reset, turbo_prefetch, N+1 counts

- Add safe_return_to_path to QuestradeItemsController (blocks //evil.com
  protocol-relative open redirect; same 3-check guard as PR #2591 Wise provider)
- Pass return_to through select_accounts and complete_account_setup so
  users land back on the account they were linking from
- Replace Rails.logger.error/warn with DebugLogEntry.capture in controller
  and unlinking concern (surface errors in the app debug log UI)
- Add questrade_items to Family::FinancialDataReset::PROVIDER_ITEM_ASSOCIATIONS
  so Reset Financial Data actually removes Questrade data
- Add when "questrade" case to load_provider_items in providers_controller
  so the settings panel lazy-load refresh works
- Fix turbo_prefetch: false on non-lunchflow provider links in
  _method_selector.html.erb and select_provider.html.erb (prevents
  prefetch-cache blank-modal bug for all generic sync providers)
- Preload questrade_accounts: :account_provider and build
  @questrade_account_counts_map in AccountsController; read from map in
  partial instead of calling .count on associations (eliminates N+1)
- Localize default connection name via I18n.t(questrade_items.default_name)
- Add default_name key to questrade_items locale

Patterns and bugs surfaced during review of PR #2591 (Wise provider).

* Cross-apply Wise learnings to Questrade provider

Encryption (matched convention from Wise/jjmata review):
- Add deterministic: true to QuestradeItem#refresh_token
- Add encrypts :raw_payload + :raw_institution_payload to QuestradeItem
- Add Encryptable + encrypts :raw_payload, :raw_holdings_payload,
  :raw_activities_payload, :raw_balances_payload to QuestradeAccount
  (brokerage-specific columns; matches MercuryAccount/UpAccount pattern)

Bug fix:
- Add missing RetryableResponseError class to Provider::Questrade
  (used in with_retries rescue clause but never defined — would cause
  NameError on any rate-limited or 5xx response)

Logging:
- Replace Rails.logger.error with DebugLogEntry.capture in
  QuestradeItem#import_latest_questrade_data, #process_accounts,
  and #schedule_account_syncs to surface errors in the support UI

Consistency:
- Extract update_sync_status(sync, key, **i18n_options) helper in
  QuestradeItem::Syncer, replacing 5 inline sync.update! guard calls
- Use blank? instead of ||= for default name fallback in create action

Tests:
- Add QuestradeItemsControllerTest (18 tests: CRUD, sync, account
  linking/setup flows, admin guard enforcement)
- Add questrade fixtures: questrade_items.yml, questrade_accounts.yml
- Add retry/backoff tests to Provider::QuestradeTest (network error,
  429, 5xx — all verify MAX_RETRIES exhaustion raises Error)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Signed-off-by: Jestin Palamuttam <34907800+jestinjoshi@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-08 09:00:49 +02:00

213 lines
7.4 KiB
Ruby

# frozen_string_literal: true
class QuestradeAccount::ActivitiesProcessor
include QuestradeAccount::DataHelpers
# Questrade groups activities by `type` (category) and `action` (sub-action,
# e.g. Buy/Sell/CON/FCH). We route on `type`, then use `action` for direction.
#
# Sign convention: Questrade `netAmount` is +money-in / -money-out, but Sure
# stores transactions as -inflow / +outflow, so cash signed_amount = -netAmount.
TRADE_TYPE = "Trades"
# Questrade `type` -> Sure investment activity label (cash transactions)
CASH_TYPE_TO_LABEL = {
"Deposits" => "Contribution",
"Withdrawals" => "Withdrawal",
"Dividends" => "Dividend",
"Interest" => "Interest",
"Fees and rebates" => "Fee"
}.freeze
# Still unmapped: FX conversions (cash currency exchanges) and corporate
# actions. Skipped with a log rather than imported wrong.
UNSUPPORTED_TYPES = [ "FX conversion", "Corporate actions" ].freeze
# Norbert's Gambit / in-kind transfers: shares journaled between symbols or
# currencies with no cash impact. Recorded as zero-cost "Transfer" trades.
JOURNAL_TYPES = [ "Other", "Transfers" ].freeze
def initialize(questrade_account)
@questrade_account = questrade_account
end
def process
return { trades: 0, transactions: 0 } unless account.present?
@trades_count = 0
@transactions_count = 0
activities.each do |raw|
process_activity(raw.with_indifferent_access)
rescue => e
Rails.logger.error "QuestradeAccount::ActivitiesProcessor - Failed to process activity: #{e.message}"
Rails.logger.error e.backtrace.first(5).join("\n") if e.backtrace
end
{ trades: @trades_count, transactions: @transactions_count }
end
private
def account
@questrade_account.current_account
end
def import_adapter
@import_adapter ||= Account::ProviderImportAdapter.new(account)
end
# raw_activities_payload may be the array itself or the { activities: [...] }
# hash returned by Provider::Questrade#get_activities.
def activities
payload = @questrade_account.raw_activities_payload
payload = payload.with_indifferent_access[:activities] if payload.is_a?(Hash)
Array(payload)
end
def process_activity(data)
type = data[:type].to_s.strip
return if type.blank?
if type == TRADE_TYPE
process_trade(data)
elsif CASH_TYPE_TO_LABEL.key?(type)
process_cash_activity(data, CASH_TYPE_TO_LABEL[type])
elsif JOURNAL_TYPES.include?(type)
if journal?(data)
process_journal(data)
else
Rails.logger.info "QuestradeAccount::ActivitiesProcessor - Skipping non-journal '#{type}'"
end
elsif UNSUPPORTED_TYPES.include?(type)
Rails.logger.info "QuestradeAccount::ActivitiesProcessor - Skipping unsupported type '#{type}'"
else
Rails.logger.warn "QuestradeAccount::ActivitiesProcessor - Unmapped activity type '#{type}'"
end
end
# Questrade activities carry no unique id, so synthesize a stable one from
# the immutable fields to keep re-syncs idempotent.
def external_id(data, prefix)
digest = Digest::SHA256.hexdigest(
[ data[:transactionDate], data[:action], data[:symbolId], data[:quantity], data[:netAmount], data[:description] ].join("|")
)
"questrade_#{prefix}_#{digest.first(24)}"
end
def process_trade(data)
ticker = data[:symbol].to_s.strip
return if ticker.blank?
security = resolve_security(ticker, { name: data[:description], currency: data[:currency] })
return unless security
quantity = parse_decimal(data[:quantity])
price = parse_decimal(data[:price])
return if quantity.nil? || quantity.zero?
sell = data[:action].to_s.casecmp("Sell").zero?
signed_quantity = sell ? -quantity.abs : quantity.abs
# Buy => positive cost, Sell => negative (matches Sure's trade convention).
amount = price ? signed_quantity * price : parse_decimal(data[:netAmount])&.abs
return if amount.nil?
date = parse_date(data[:tradeDate]) || parse_date(data[:transactionDate]) || Date.current
currency = extract_currency(data, fallback: account.currency)
result = import_adapter.import_trade(
external_id: external_id(data, "trade"),
security: security,
quantity: signed_quantity,
price: price,
amount: amount,
currency: currency,
date: date,
name: data[:description].presence || "#{sell ? 'Sell' : 'Buy'} #{ticker}",
source: "questrade",
activity_label: sell ? "Sell" : "Buy"
)
@trades_count += 1 if result
import_commission(data, ticker, date, currency)
end
def import_commission(data, ticker, date, currency)
commission = parse_decimal(data[:commission])
return if commission.nil? || commission.zero?
result = import_adapter.import_transaction(
external_id: external_id(data, "fee"),
amount: commission.abs, # money out
currency: currency,
date: date,
name: "Commission for #{ticker}",
source: "questrade",
investment_activity_label: "Fee"
)
@transactions_count += 1 if result
end
def journal?(data)
data[:symbol].to_s.strip.present? && !(parse_decimal(data[:quantity]) || 0).zero?
end
# A journal (Norbert's Gambit / transfer) moves shares with no cash impact.
# Recorded as a zero-cost Transfer trade; the holding's cost basis still
# comes from the positions snapshot.
def process_journal(data)
ticker = data[:symbol].to_s.strip
security = resolve_security(ticker, { name: data[:description], currency: data[:currency] })
return unless security
quantity = parse_decimal(data[:quantity])
return if quantity.nil? || quantity.zero?
date = parse_date(data[:tradeDate]) || parse_date(data[:transactionDate]) || Date.current
currency = extract_currency(data, fallback: account.currency)
result = import_adapter.import_trade(
external_id: external_id(data, "journal"),
security: security,
quantity: quantity,
price: 0,
amount: 0,
currency: currency,
date: date,
name: data[:description].presence || "Journal #{ticker}",
source: "questrade",
activity_label: "Transfer"
)
@trades_count += 1 if result
end
def process_cash_activity(data, label)
net = parse_decimal(data[:netAmount])
return if net.nil?
signed_amount = -net # Questrade +in / -out -> Sure -in / +out
date = parse_date(data[:settlementDate]) ||
parse_date(data[:transactionDate]) ||
parse_date(data[:tradeDate]) ||
Date.current
currency = extract_currency(data, fallback: account.currency)
symbol = data[:symbol].to_s.strip
security = symbol.present? ? resolve_security(symbol, { name: data[:description], currency: data[:currency] }) : nil
name = data[:description].presence || (symbol.present? ? "#{label} - #{symbol}" : label)
result = import_adapter.import_transaction(
external_id: external_id(data, "cash"),
amount: signed_amount,
currency: currency,
date: date,
name: name,
source: "questrade",
investment_activity_label: label,
extra: { security_id: security&.id }.compact
)
@transactions_count += 1 if result
end
end