mirror of
https://github.com/we-promise/sure.git
synced 2026-07-20 00:35:22 +00:00
* Add native Questrade brokerage provider integration Adds a per-family Questrade provider so users can sync their Questrade investment accounts (TFSA, FHSA, RRSP, margin, etc.) directly via Questrade's free personal API, with no paid aggregator. - OAuth2 refresh-token flow with single-use token rotation, persisted under a row lock. Tokens self-renew on each sync; the connected panel lets users paste a fresh token if a connection goes stale (no need to disconnect and re-link). - Imports accounts, balances, positions and activities; multi-currency holdings with per-currency cash holdings; Norbert's Gambit journals. - New-account and link-existing-account flows, settings card with desktop-only setup steps, and connect/update/disconnect. - Restricted to Investment account types. Registered in the provider connection-status registry with a syncable scope so it participates in nightly family sync. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: linting error * fix: refresh token encrypted The OAuth token exchange passed the single-use refresh token as a GET query parameter, so it could leak into URL-based logs (Sentry breadcrumbs, APM spans, debug output). Switch to POST with a form-encoded body (RFC 6749 3.2) so the credential stays out of URLs. Verified Questrade's token endpoint accepts POST (returns 400 for a bad token, not 405). Adds a test asserting the token travels in the body. * Address PR review: authz, data integrity, retries, logging Batch of fixes from the automated PR review: - Require admin for all mutating/linking Questrade actions, and gate existing-account linking through accessible_accounts + write permission (was only Current.family scoped). - Clear requires_update when a fresh token is accepted; use a real 302 redirect (not 422) on full-page failures. - Require refresh_token on all saves (not just create) unless the item is scheduled for deletion. - Migrations target Rails 7.2; questrade_items state columns are NOT NULL. - Background activity dedup keys on Questrade fields (matches the importer) so multiple activities no longer collapse to one. - Persist the normalized account payload; date-scope synthetic cash holdings so daily history is not overwritten. - Retry 429/5xx via a RetryableResponseError instead of hard-failing. - Route provider error bodies to DebugLogEntry instead of Rails.logger / exception messages, so payloads do not leak into application logs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address PR review: atomic linking, sync health, retry loop, USD cash - Wrap account creation + provider linking (+ sync_start_date) in a transaction in both link paths so a link failure rolls back the orphan account. - Surface per-account process/schedule failures in the item sync health instead of always reporting healthy. - Always stamp last_activities_sync once the background fetch completes, so legitimately empty accounts stop being re-queued every sync. - Treat only the account-currency (CAD) balance as primary cash; other currencies (e.g. USD) now surface as separate cash holdings instead of being hidden as primary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address PR review: serialize token exchange, real processor tests - Single-use token race: the SDK now wraps every token exchange (initial and 401 re-auth) in a model-supplied lock that reloads and spends the freshest persisted token (provided.rb#synchronize_exchange). Two concurrent syncs/jobs can no longer double-spend the same refresh token. Adds a test asserting the exchange runs inside the lock with the fresh token. - Replace the all-skipped QuestradeAccount processor test stubs with real fixture-backed tests covering balance anchoring, holdings import, and Buy-trade import (plus blank-symbol / blank-type guards). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Fix indentation of spliced Questrade schema blocks The manually added questrade_accounts/questrade_items create_table blocks sat at column 0 instead of the file 2-space indent, so rubocop flagged them as inconsistent. Re-indent to match the rest of the schema. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Include currency and type in the Questrade activity merge key Two activities that differ only by currency or type could collapse to a single row in merge_activities. Add both fields to activity_key in the importer and the background fetch job so multi-currency imports dedup correctly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address review: infer account currency, Encryptable, safer flag clear From @jjmata's review: - Currency: QuestradeAccount#upsert_from_questrade! no longer hardcodes CAD for every account. upsert_balances! now infers the home currency from the per-currency balances (the currency holding the cash wins, ties broken by total equity, default CAD) so USD-denominated accounts are labelled USD and match the right combinedBalances anchor. Adds tests for USD and CAD cases. - QuestradeItem now includes the shared Encryptable concern instead of reimplementing encryption_ready? inline. - QuestradeActivitiesFetchJob#clear_pending_flag is now best-effort so it can never mask (and swallow) the original error in perform's rescue. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Fix review issues: safe_return_to_path, DebugLogEntry, financial reset, turbo_prefetch, N+1 counts - Add safe_return_to_path to QuestradeItemsController (blocks //evil.com protocol-relative open redirect; same 3-check guard as PR #2591 Wise provider) - Pass return_to through select_accounts and complete_account_setup so users land back on the account they were linking from - Replace Rails.logger.error/warn with DebugLogEntry.capture in controller and unlinking concern (surface errors in the app debug log UI) - Add questrade_items to Family::FinancialDataReset::PROVIDER_ITEM_ASSOCIATIONS so Reset Financial Data actually removes Questrade data - Add when "questrade" case to load_provider_items in providers_controller so the settings panel lazy-load refresh works - Fix turbo_prefetch: false on non-lunchflow provider links in _method_selector.html.erb and select_provider.html.erb (prevents prefetch-cache blank-modal bug for all generic sync providers) - Preload questrade_accounts: :account_provider and build @questrade_account_counts_map in AccountsController; read from map in partial instead of calling .count on associations (eliminates N+1) - Localize default connection name via I18n.t(questrade_items.default_name) - Add default_name key to questrade_items locale Patterns and bugs surfaced during review of PR #2591 (Wise provider). * Cross-apply Wise learnings to Questrade provider Encryption (matched convention from Wise/jjmata review): - Add deterministic: true to QuestradeItem#refresh_token - Add encrypts :raw_payload + :raw_institution_payload to QuestradeItem - Add Encryptable + encrypts :raw_payload, :raw_holdings_payload, :raw_activities_payload, :raw_balances_payload to QuestradeAccount (brokerage-specific columns; matches MercuryAccount/UpAccount pattern) Bug fix: - Add missing RetryableResponseError class to Provider::Questrade (used in with_retries rescue clause but never defined — would cause NameError on any rate-limited or 5xx response) Logging: - Replace Rails.logger.error with DebugLogEntry.capture in QuestradeItem#import_latest_questrade_data, #process_accounts, and #schedule_account_syncs to surface errors in the support UI Consistency: - Extract update_sync_status(sync, key, **i18n_options) helper in QuestradeItem::Syncer, replacing 5 inline sync.update! guard calls - Use blank? instead of ||= for default name fallback in create action Tests: - Add QuestradeItemsControllerTest (18 tests: CRUD, sync, account linking/setup flows, admin guard enforcement) - Add questrade fixtures: questrade_items.yml, questrade_accounts.yml - Add retry/backoff tests to Provider::QuestradeTest (network error, 429, 5xx — all verify MAX_RETRIES exhaustion raises Error) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Signed-off-by: Jestin Palamuttam <34907800+jestinjoshi@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
267 lines
9.1 KiB
Ruby
267 lines
9.1 KiB
Ruby
# frozen_string_literal: true
|
|
|
|
class ProviderConnectionStatus
|
|
PROVIDERS = [
|
|
{ key: "akahu", type: "AkahuItem", association: :akahu_items, accounts: :akahu_accounts },
|
|
{ key: "up", type: "UpItem", association: :up_items, accounts: :up_accounts },
|
|
{ key: "plaid", type: "PlaidItem", association: :plaid_items, accounts: :plaid_accounts },
|
|
{ key: "simplefin", type: "SimplefinItem", association: :simplefin_items, accounts: :simplefin_accounts },
|
|
{ key: "lunchflow", type: "LunchflowItem", association: :lunchflow_items, accounts: :lunchflow_accounts },
|
|
{ key: "enable_banking", type: "EnableBankingItem", association: :enable_banking_items, accounts: :enable_banking_accounts },
|
|
{ key: "coinbase", type: "CoinbaseItem", association: :coinbase_items, accounts: :coinbase_accounts },
|
|
{ key: "binance", type: "BinanceItem", association: :binance_items, accounts: :binance_accounts },
|
|
{ key: "kraken", type: "KrakenItem", association: :kraken_items, accounts: :kraken_accounts },
|
|
{ key: "coinstats", type: "CoinstatsItem", association: :coinstats_items, accounts: :coinstats_accounts },
|
|
{ key: "snaptrade", type: "SnaptradeItem", association: :snaptrade_items, accounts: :snaptrade_accounts, linked_accounts: :linked_accounts },
|
|
{ key: "ibkr", type: "IbkrItem", association: :ibkr_items, accounts: :ibkr_accounts },
|
|
{ key: "mercury", type: "MercuryItem", association: :mercury_items, accounts: :mercury_accounts },
|
|
{ key: "brex", type: "BrexItem", association: :brex_items, accounts: :brex_accounts },
|
|
{ key: "sophtron", type: "SophtronItem", association: :sophtron_items, accounts: :sophtron_accounts },
|
|
{ key: "indexa_capital", type: "IndexaCapitalItem", association: :indexa_capital_items, accounts: :indexa_capital_accounts },
|
|
{ key: "questrade", type: "QuestradeItem", association: :questrade_items, accounts: :questrade_accounts }
|
|
].freeze
|
|
|
|
class << self
|
|
def for_family(family)
|
|
PROVIDERS.flat_map do |provider|
|
|
relation = family.public_send(provider[:association])
|
|
items = relation.includes(association_includes_for(relation, provider)).ordered.to_a
|
|
sync_contexts = sync_contexts_for(provider[:type], items)
|
|
|
|
items.map do |item|
|
|
new(provider, item, sync_contexts.fetch(item.id, {})).to_h
|
|
end
|
|
end
|
|
end
|
|
|
|
private
|
|
|
|
def association_includes_for(relation, provider)
|
|
includes = [ { provider[:accounts] => :account_provider } ]
|
|
includes << provider[:linked_accounts] if provider[:linked_accounts]
|
|
includes << :accounts if relation.klass.reflect_on_association(:accounts)
|
|
includes
|
|
end
|
|
|
|
def sync_contexts_for(provider_type, items)
|
|
item_ids = items.map(&:id)
|
|
return {} if item_ids.empty?
|
|
|
|
latest_syncs = latest_syncs_for(provider_type, item_ids)
|
|
latest_completed_syncs = latest_syncs_for(provider_type, item_ids, scope: Sync.completed)
|
|
syncing_item_ids = Sync.visible
|
|
.where(syncable_type: provider_type, syncable_id: item_ids)
|
|
.distinct
|
|
.pluck(:syncable_id)
|
|
|
|
item_ids.index_with do |item_id|
|
|
{
|
|
latest_sync: latest_syncs[item_id],
|
|
latest_completed_sync: latest_completed_syncs[item_id],
|
|
syncing: syncing_item_ids.include?(item_id)
|
|
}
|
|
end
|
|
end
|
|
|
|
def latest_syncs_for(provider_type, item_ids, scope: Sync.all)
|
|
ranked_syncs = scope.where(syncable_type: provider_type, syncable_id: item_ids)
|
|
.select(
|
|
"syncs.*, " \
|
|
"ROW_NUMBER() OVER (PARTITION BY syncable_id ORDER BY created_at DESC, id DESC) AS sync_rank"
|
|
)
|
|
|
|
Sync.from(ranked_syncs, :syncs).where("sync_rank = 1").index_by(&:syncable_id)
|
|
end
|
|
end
|
|
|
|
def initialize(provider, item, sync_context = {})
|
|
@provider = provider
|
|
@item = item
|
|
@sync_context = sync_context
|
|
end
|
|
|
|
def to_h
|
|
{
|
|
id: item.id,
|
|
provider: provider[:key],
|
|
provider_type: provider[:type],
|
|
name: item_value(:name, provider[:key].humanize),
|
|
status: item_status,
|
|
requires_update: item_requires_update?,
|
|
credentials_configured: credentials_configured?,
|
|
scheduled_for_deletion: item_boolean(:scheduled_for_deletion?),
|
|
pending_account_setup: pending_account_setup?,
|
|
institution: institution_payload,
|
|
accounts: accounts_payload,
|
|
sync: sync_payload,
|
|
created_at: item.created_at,
|
|
updated_at: item.updated_at
|
|
}
|
|
end
|
|
|
|
private
|
|
|
|
attr_reader :provider, :item, :sync_context
|
|
|
|
def credentials_configured?
|
|
item_boolean(:credentials_configured?)
|
|
end
|
|
|
|
def item_status
|
|
return item.effective_status(latest_sync: latest_sync) if item.respond_to?(:setup_token_update_required?)
|
|
|
|
item_value(:status)
|
|
end
|
|
|
|
def item_requires_update?
|
|
return item.setup_token_update_required?(latest_sync: latest_sync) if item.respond_to?(:setup_token_update_required?)
|
|
|
|
item_boolean(:requires_update?)
|
|
end
|
|
|
|
def pending_account_setup?
|
|
item_boolean(:pending_account_setup?)
|
|
end
|
|
|
|
def institution_payload
|
|
{
|
|
name: item_value(:institution_display_name, item_value(:name, provider[:key].humanize)),
|
|
domain: item_value(:institution_domain),
|
|
url: item_value(:institution_url)
|
|
}
|
|
end
|
|
|
|
def accounts_payload
|
|
@accounts_payload ||= begin
|
|
total = provider_account_count
|
|
linked = linked_account_count
|
|
|
|
{
|
|
total_count: total,
|
|
linked_count: linked,
|
|
unlinked_count: [ total - linked, 0 ].max
|
|
}
|
|
end
|
|
end
|
|
|
|
def provider_account_count
|
|
records = provider_account_records
|
|
return records.size if records
|
|
return item.total_accounts_count if item.respond_to?(:total_accounts_count)
|
|
|
|
0
|
|
end
|
|
|
|
def linked_account_count
|
|
records = provider_account_records
|
|
return records.count { |provider_account| linked_provider_account?(provider_account) } if records
|
|
return item.linked_accounts_count if item.respond_to?(:linked_accounts_count)
|
|
|
|
if provider[:linked_accounts] && item.respond_to?(provider[:linked_accounts])
|
|
return item.public_send(provider[:linked_accounts]).size
|
|
end
|
|
|
|
return item.accounts.size if item.respond_to?(:accounts)
|
|
|
|
0
|
|
end
|
|
|
|
def provider_account_records
|
|
return unless item.respond_to?(provider[:accounts])
|
|
|
|
@provider_account_records ||= item.public_send(provider[:accounts]).to_a
|
|
end
|
|
|
|
def linked_provider_account?(provider_account)
|
|
return false unless provider_account.respond_to?(:account_provider)
|
|
|
|
association = provider_account.association(:account_provider)
|
|
association.loaded? ? association.target.present? : provider_account.account_provider.present?
|
|
end
|
|
|
|
def sync_payload
|
|
{
|
|
syncing: syncing?,
|
|
status_summary: sync_status_summary,
|
|
last_synced_at: latest_completed_sync&.completed_at,
|
|
latest: latest_sync_payload(latest_sync)
|
|
}
|
|
end
|
|
|
|
def sync_status_summary
|
|
stats = latest_completed_sync_stats
|
|
counts = accounts_payload
|
|
total = stats.fetch("total_accounts", counts[:total_count]).to_i
|
|
linked = stats.fetch("linked_accounts", counts[:linked_count]).to_i
|
|
unlinked = stats.fetch("unlinked_accounts", [ total - linked, 0 ].max).to_i
|
|
|
|
if total.zero?
|
|
"No accounts found"
|
|
elsif unlinked.zero?
|
|
"#{linked} #{'account'.pluralize(linked)} synced"
|
|
else
|
|
"#{linked} synced, #{unlinked} need setup"
|
|
end
|
|
end
|
|
|
|
def syncing?
|
|
return sync_context[:syncing] if sync_context.key?(:syncing)
|
|
|
|
item_boolean(:syncing?)
|
|
end
|
|
|
|
def latest_sync
|
|
sync_context[:latest_sync]
|
|
end
|
|
|
|
def latest_completed_sync
|
|
sync_context[:latest_completed_sync]
|
|
end
|
|
|
|
def latest_completed_sync_stats
|
|
stats = latest_completed_sync&.sync_stats
|
|
return stats.stringify_keys if stats.is_a?(Hash)
|
|
return {} unless stats.is_a?(String)
|
|
|
|
parsed = JSON.parse(stats)
|
|
parsed.is_a?(Hash) ? parsed.stringify_keys : {}
|
|
rescue JSON::ParserError
|
|
{}
|
|
end
|
|
|
|
def latest_sync_payload(sync)
|
|
return unless sync
|
|
|
|
{
|
|
id: sync.id,
|
|
status: sync.status,
|
|
created_at: sync.created_at,
|
|
syncing_at: sync.syncing_at,
|
|
completed_at: sync.completed_at,
|
|
failed_at: sync.failed_at,
|
|
error: sync_error_payload(sync)
|
|
}
|
|
end
|
|
|
|
def sync_error_payload(sync)
|
|
return unless sync.failed? || sync.stale?
|
|
|
|
# Provider health treats stale connections as actionable even when the
|
|
# generic sync API suppresses stale-without-error payloads.
|
|
{
|
|
present: true,
|
|
message: sync.stale? ? "Sync became stale before completion" : "Sync failed"
|
|
}
|
|
end
|
|
|
|
def item_boolean(method_name)
|
|
item_value(method_name, false) == true
|
|
end
|
|
|
|
def item_value(method_name, default = nil)
|
|
return default unless item.respond_to?(method_name)
|
|
|
|
item.public_send(method_name)
|
|
end
|
|
end
|