From dc2a565b6ae7cf0674fb9c77295d60088c9ea4b9 Mon Sep 17 00:00:00 2001 From: Jake <69498448+Jookly123@users.noreply.github.com> Date: Sun, 21 Jun 2026 04:33:09 +1000 Subject: [PATCH] feat(up): add Up Bank (AU) provider integration (#2391) * feat(up): add Up Bank (AU) provider integration Adds Up Bank as a per-family, token-based bank sync provider, modelled on the existing Akahu integration. Up uses a JSON:API REST API with a personal access token (Bearer), cursor pagination via links.next, and returns both HELD (pending) and SETTLED transactions from one endpoint. New: - Provider::Up client (JSON:API unwrap, links.next pagination, retries, typed errors, /util/ping) + Provider::UpAdapter (Factory-registered, Depository + Loan). - UpItem / UpAccount models with Provided, Unlinking, Syncer, SyncCompleteEvent, Importer, Processor, Transactions::Processor, and UpEntry::Processor (amount sign flip, HELD->pending, foreignAmount FX, merchant from description, stale-pending pruning). - Family::UpConnectable, UpItemsController, routes, settings panel + connect flow views, accounts index wiring, initializer, en locale, and model tests. Core wiring: - "up" added to Transaction::PENDING_PROVIDERS, the three pending-match SQL blocks in Account::ProviderImportAdapter, Provider::Metadata::REGISTRY, ProviderMerchant/DataEnrichment source enums, ProviderConnectionStatus, settings provider panels, and financial data reset. Migration create_up_items_and_accounts must be run before use. No external API endpoints added (no OpenAPI changes). Co-Authored-By: Claude Opus 4.8 * fix(up): dump up tables to schema and make since filter TZ-safe The feature commit added the up_items/up_accounts migration but never re-dumped db/schema.rb, leaving the schema version and tables stale. Add the two table definitions and foreign keys and bump the schema version so a fresh DB load matches the migration. Also format a bare Date `since` as UTC midnight instead of the server's local zone, so `filter[since]` is deterministic regardless of where the app runs (previously shifted by the local UTC offset). Co-Authored-By: Claude Opus 4.8 * fix(up): address code review feedback Behavior/correctness: - Persist skipped accounts via a new up_accounts.ignored flag and a needs_setup scope, so skipped accounts stop resurfacing as "needs setup" on every sync. Linking clears the flag. - destroy now checks unlink_all! per-account results and aborts deletion (alert) if any unlink failed, instead of swallowing failures. - render_provider_panel_error redirect uses :see_other (was an invalid 4xx redirect status). - Up provider adapter falls back to item institution name/url when institution_metadata is absent (early return previously blocked it). Resilience/security: - fetch_all_resources guards against an API repeating the same links.next cursor (Set#add?), preventing infinite pagination. - HTTP client validates absolute URLs (from links.next) against Up's HTTPS host before sending the bearer token, preventing credential leakage to untrusted hosts. Diagnostics: - Route provider sync/import failures through DebugLogEntry.capture (controller, UpItem, syncer, unlinking) with family/account context. Low-level HTTP client and currency-normalization warnings keep Rails.logger to match existing provider conventions. Data integrity: - up_accounts.name and currency are NOT NULL (align with model presence validations); account_id stays nullable (allow_nil uniqueness). Forms: - select_existing_account radio is required; controller guards a blank/ unknown up_account_id with a friendly alert instead of RecordNotFound. Tests: - Add UpAccount needs_setup scope test, pagination loop guard test, untrusted-host rejection test; tighten filter[since] assertion to the exact UTC timestamp. Co-Authored-By: Claude Opus 4.8 * fix(up): address second-round review feedback - Capture sync/import failures via DebugLogEntry so swallowed errors in account/transaction fetching and transaction processing surface in /settings/debug instead of only Rails.logger. - Gate UP_DEBUG_RAW raw payload dump to local envs to avoid leaking PII (merchant names, amounts, account IDs) in managed/production logs. - Collapse linked/unlinked/total account counts into one memoized query instead of 3 separate COUNTs per rendered item. - Rename "Set Up Up Accounts" locale title to "Link Up Accounts". Co-Authored-By: Claude Opus 4.8 * docs(up): add method docstrings and align failed_result keys Add docstrings to all Up provider source files (controller, models, providers, concerns) to satisfy the 80% docstring coverage threshold. Third-round review: failed_result now mirrors import's result shape (accounts_updated/created/failed, transactions_imported/failed) instead of the stale accounts_imported key, so failure results stay consistent. Co-Authored-By: Claude Opus 4.8 --------- Co-authored-by: Claude Opus 4.8 --- CLAUDE.md | 1 + app/controllers/accounts_controller.rb | 8 + .../settings/providers_controller.rb | 6 + app/controllers/up_items_controller.rb | 442 ++++++++++++++++++ app/helpers/settings_helper.rb | 3 + app/models/account/provider_import_adapter.rb | 3 + app/models/data_enrichment.rb | 1 + app/models/family.rb | 1 + app/models/family/financial_data_reset.rb | 1 + app/models/family/up_connectable.rb | 28 ++ app/models/provider/metadata.rb | 1 + app/models/provider/up.rb | 224 +++++++++ app/models/provider/up_adapter.rb | 111 +++++ app/models/provider_connection_status.rb | 1 + app/models/provider_merchant.rb | 2 +- app/models/transaction.rb | 2 +- app/models/up_account.rb | 93 ++++ app/models/up_account/processor.rb | 91 ++++ .../up_account/transactions/processor.rb | 91 ++++ app/models/up_entry/processor.rb | 199 ++++++++ app/models/up_item.rb | 201 ++++++++ app/models/up_item/importer.rb | 250 ++++++++++ app/models/up_item/provided.rb | 15 + app/models/up_item/sync_complete_event.rb | 22 + app/models/up_item/syncer.rb | 141 ++++++ app/models/up_item/unlinking.rb | 54 +++ app/views/accounts/index.html.erb | 6 +- .../settings/providers/_up_panel.html.erb | 123 +++++ app/views/up_items/_up_item.html.erb | 114 +++++ app/views/up_items/select_accounts.html.erb | 50 ++ .../up_items/select_existing_account.html.erb | 50 ++ app/views/up_items/setup_accounts.html.erb | 72 +++ config/initializers/up.rb | 9 + config/locales/views/settings/en.yml | 4 + config/locales/views/up_items/en.yml | 116 +++++ config/routes.rb | 16 + ...0617120000_create_up_items_and_accounts.rb | 49 ++ db/schema.rb | 47 +- test/models/provider/up_test.rb | 160 +++++++ test/models/up_account_test.rb | 24 + test/models/up_entry/processor_test.rb | 116 +++++ test/models/up_item/importer_test.rb | 156 +++++++ 42 files changed, 3100 insertions(+), 4 deletions(-) create mode 100644 app/controllers/up_items_controller.rb create mode 100644 app/models/family/up_connectable.rb create mode 100644 app/models/provider/up.rb create mode 100644 app/models/provider/up_adapter.rb create mode 100644 app/models/up_account.rb create mode 100644 app/models/up_account/processor.rb create mode 100644 app/models/up_account/transactions/processor.rb create mode 100644 app/models/up_entry/processor.rb create mode 100644 app/models/up_item.rb create mode 100644 app/models/up_item/importer.rb create mode 100644 app/models/up_item/provided.rb create mode 100644 app/models/up_item/sync_complete_event.rb create mode 100644 app/models/up_item/syncer.rb create mode 100644 app/models/up_item/unlinking.rb create mode 100644 app/views/settings/providers/_up_panel.html.erb create mode 100644 app/views/up_items/_up_item.html.erb create mode 100644 app/views/up_items/select_accounts.html.erb create mode 100644 app/views/up_items/select_existing_account.html.erb create mode 100644 app/views/up_items/setup_accounts.html.erb create mode 100644 config/initializers/up.rb create mode 100644 config/locales/views/up_items/en.yml create mode 100644 db/migrate/20260617120000_create_up_items_and_accounts.rb create mode 100644 test/models/provider/up_test.rb create mode 100644 test/models/up_account_test.rb create mode 100644 test/models/up_entry/processor_test.rb create mode 100644 test/models/up_item/importer_test.rb diff --git a/CLAUDE.md b/CLAUDE.md index 5464699de..a1da3b7bb 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -118,6 +118,7 @@ Two primary data ingestion methods: - Set `SIMPLEFIN_INCLUDE_PENDING=0` to disable pending fetching for SimpleFIN. - Set `PLAID_INCLUDE_PENDING=0` to disable pending fetching for Plaid. - Set `SIMPLEFIN_DEBUG_RAW=1` to enable raw payload debug logging. + - Set `UP_DEBUG_RAW=1` to enable raw Up payload debug logging. DEV-ONLY: the dump contains PII and is gated to local environments, so it never logs in managed/production. Provider support notes: - SimpleFIN: supports pending + FX metadata (stored under `extra["simplefin"]`). diff --git a/app/controllers/accounts_controller.rb b/app/controllers/accounts_controller.rb index 6ce5e02c6..bd62cd9f8 100644 --- a/app/controllers/accounts_controller.rb +++ b/app/controllers/accounts_controller.rb @@ -16,6 +16,7 @@ class AccountsController < ApplicationController @simplefin_items = visible_provider_items(family.simplefin_items.ordered.includes(:syncs)) @lunchflow_items = visible_provider_items(family.lunchflow_items.ordered.includes(:syncs, :lunchflow_accounts)) @akahu_items = visible_provider_items(family.akahu_items.ordered.includes(:syncs, :akahu_accounts)) + @up_items = visible_provider_items(family.up_items.ordered.includes(:syncs, :up_accounts)) @enable_banking_items = visible_provider_items(family.enable_banking_items.ordered.includes(:syncs)) @coinstats_items = visible_provider_items(family.coinstats_items.ordered.includes(:coinstats_accounts, :accounts, :syncs)) @mercury_items = visible_provider_items(family.mercury_items.ordered.includes(:syncs, :mercury_accounts)) @@ -334,6 +335,13 @@ class AccountsController < ApplicationController @akahu_sync_stats_map[item.id] = latest_sync&.sync_stats || {} end + # Up sync stats + @up_sync_stats_map = {} + @up_items.each do |item| + latest_sync = item.syncs.ordered.first + @up_sync_stats_map[item.id] = latest_sync&.sync_stats || {} + end + # Enable Banking sync stats @enable_banking_sync_stats_map = {} @enable_banking_latest_sync_error_map = {} diff --git a/app/controllers/settings/providers_controller.rb b/app/controllers/settings/providers_controller.rb index 1a3a03ca1..8af7dd0ce 100644 --- a/app/controllers/settings/providers_controller.rb +++ b/app/controllers/settings/providers_controller.rb @@ -183,6 +183,7 @@ class Settings::ProvidersController < ApplicationController # them (see prepare_show_context). FAMILY_PANELS = [ { key: "akahu", title: "Akahu", turbo_id: "akahu", partial: "akahu_panel" }, + { key: "up", title: "Up", turbo_id: "up", partial: "up_panel" }, { key: "lunchflow", title: "Lunch Flow", turbo_id: "lunchflow", partial: "lunchflow_panel" }, { key: "simplefin", title: "SimpleFIN", turbo_id: "simplefin", partial: "simplefin_panel" }, { key: "enable_banking", title: "Enable Banking", turbo_id: "enable_banking", partial: "enable_banking_panel" }, @@ -203,6 +204,7 @@ class Settings::ProvidersController < ApplicationController # Maps panel key → ActiveRecord model name for sync health queries PANEL_SYNCABLE_TYPES = { "akahu" => "AkahuItem", + "up" => "UpItem", "simplefin" => "SimplefinItem", "lunchflow" => "LunchflowItem", "enable_banking" => "EnableBankingItem", @@ -222,6 +224,8 @@ class Settings::ProvidersController < ApplicationController case provider_key when "akahu" @akahu_items = Current.family.akahu_items.active.ordered + when "up" + @up_items = Current.family.up_items.active.ordered when "simplefin" @simplefin_items = Current.family.simplefin_items.ordered when "lunchflow" @@ -260,6 +264,7 @@ class Settings::ProvidersController < ApplicationController end @akahu_items = Current.family.akahu_items.active.ordered + @up_items = Current.family.up_items.active.ordered # Providers page only needs to know whether any SimpleFin/Lunchflow connections exist with valid credentials @simplefin_items = Current.family.simplefin_items.where.not(access_url: [ nil, "" ]).ordered.select(:id) @lunchflow_items = Current.family.lunchflow_items.where.not(api_key: [ nil, "" ]).ordered.select(:id) @@ -293,6 +298,7 @@ class Settings::ProvidersController < ApplicationController def family_panel_items { "akahu" => @akahu_items, + "up" => @up_items, "simplefin" => @simplefin_items, "lunchflow" => @lunchflow_items, "enable_banking" => @enable_banking_items, diff --git a/app/controllers/up_items_controller.rb b/app/controllers/up_items_controller.rb new file mode 100644 index 000000000..19998bd1c --- /dev/null +++ b/app/controllers/up_items_controller.rb @@ -0,0 +1,442 @@ +class UpItemsController < ApplicationController + before_action :set_up_item, only: [ :show, :edit, :update, :destroy, :sync, :setup_accounts, :complete_account_setup ] + before_action :require_admin!, only: [ + :new, :create, :preload_accounts, :select_accounts, :link_accounts, + :select_existing_account, :link_existing_account, :edit, :update, + :destroy, :sync, :setup_accounts, :complete_account_setup + ] + + # List the family's active Up connections in settings. + def index + @up_items = Current.family.up_items.active.ordered + render layout: "settings" + end + + # Show a single Up connection. + def show + end + + # Render the new-connection form. + def new + @up_item = Current.family.up_items.build + end + + # Render the edit-connection form. + def edit + end + + # Create an Up connection and kick off its first sync. + def create + @up_item = Current.family.up_items.build(up_item_params) + @up_item.name = t("up_items.provider_panel.default_connection_name") if @up_item.name.blank? + + if @up_item.save + @up_item.sync_later + render_provider_panel(:notice, t(".success")) + else + render_provider_panel_error(@up_item.errors.full_messages.join(", ")) + end + end + + # Update connection settings (name/token/start date). + def update + if @up_item.update(update_params) + render_provider_panel(:notice, t(".success")) + else + render_provider_panel_error(@up_item.errors.full_messages.join(", ")) + end + end + + # Unlink all accounts then schedule deletion of the connection. + def destroy + results = @up_item.unlink_all!(dry_run: false) + + if results.any? { |result| result[:error].present? } + DebugLogEntry.capture( + category: "provider_sync_error", + level: "warn", + message: "Up unlink during destroy failed", + source: self.class.name, + provider_key: "up", + family: @up_item.family, + metadata: { up_item_id: @up_item.id, failures: results.select { |r| r[:error].present? } } + ) + redirect_to settings_providers_path, alert: t(".unlink_failed"), status: :see_other + return + end + + @up_item.destroy_later + redirect_to settings_providers_path, notice: t(".success"), status: :see_other + rescue => e + DebugLogEntry.capture( + category: "provider_sync_error", + level: "warn", + message: "Up unlink during destroy failed", + source: self.class.name, + provider_key: "up", + family: @up_item&.family, + metadata: { up_item_id: @up_item&.id, error_class: e.class.name, error_message: e.message } + ) + redirect_to settings_providers_path, alert: t(".unlink_failed"), status: :see_other + end + + # Trigger a manual sync unless one is already running. + def sync + @up_item.sync_later unless @up_item.syncing? + + respond_to do |format| + format.html { redirect_back_or_to accounts_path } + format.json { head :ok } + end + end + + # Fetch accounts from the API (JSON) so the UI can show whether any exist. + def preload_accounts + up_item = requested_up_item + return render json: { success: false, error: "no_credentials", has_accounts: false } unless up_item.credentials_configured? + + error = fetch_up_accounts_from_api(up_item) + render json: { success: error.blank?, error_message: error, has_accounts: up_item.up_accounts.exists? } + end + + # Render the picker of unlinked Up accounts for a new Sure account. + def select_accounts + @accountable_type = params[:accountable_type] || "Depository" + @return_to = safe_return_to_path + @up_item = requested_up_item + + unless @up_item.credentials_configured? + redirect_to settings_providers_path, alert: t(".no_credentials_configured") + return + end + + @api_error = fetch_up_accounts_from_api(@up_item) + @up_accounts = @up_item.up_accounts + .left_joins(:account_provider) + .where(account_providers: { id: nil }) + .order(:name) + + render layout: false + end + + # Create new Sure accounts for the selected Up accounts and link them. + def link_accounts + up_item = requested_up_item + unless up_item.credentials_configured? + redirect_to settings_providers_path, alert: t(".no_credentials_configured") + return + end + + selected_ids = Array(params[:account_ids]).compact_blank + if selected_ids.empty? + redirect_to select_accounts_up_items_path(up_item_id: up_item.id, accountable_type: params[:accountable_type], return_to: safe_return_to_path), alert: t(".no_accounts_selected") + return + end + + account_type = params[:accountable_type].presence || "Depository" + unless Provider::UpAdapter.supported_account_types.include?(account_type) + redirect_to new_account_path, alert: t(".unsupported_account_type") + return + end + + created_accounts = [] + + ActiveRecord::Base.transaction do + up_item.up_accounts.where(id: selected_ids).find_each do |up_account| + next if up_account.account_provider.present? + + account = create_account_from_up(up_account, account_type) + AccountProvider.create!(account: account, provider: up_account) + created_accounts << account + end + end + + up_item.sync_later if created_accounts.any? + + if created_accounts.any? + redirect_to safe_return_to_path || accounts_path, notice: t(".success", count: created_accounts.count) + else + redirect_to select_accounts_up_items_path(up_item_id: up_item.id, accountable_type: account_type, return_to: safe_return_to_path), alert: t(".link_failed") + end + end + + # Render the picker to attach an Up account to an existing Sure account. + def select_existing_account + @account = Current.family.accounts.find(params[:account_id]) + + if @account.account_providers.exists? + redirect_to accounts_path, alert: t(".account_already_linked") + return + end + + @up_item = requested_up_item + unless @up_item.credentials_configured? + redirect_to settings_providers_path, alert: t(".no_credentials_configured") + return + end + + @api_error = fetch_up_accounts_from_api(@up_item) + @up_accounts = @up_item.up_accounts + .left_joins(:account_provider) + .where(account_providers: { id: nil }) + .order(:name) + @return_to = safe_return_to_path + + render layout: false + end + + # Link a selected Up account to an existing Sure account and sync. + def link_existing_account + account = Current.family.accounts.find(params[:account_id]) + up_item = requested_up_item + + unless up_item.credentials_configured? + redirect_to settings_providers_path, alert: t("up_items.select_existing_account.no_credentials_configured") + return + end + + if params[:up_account_id].blank? + redirect_to accounts_path, alert: t(".no_account_selected") + return + end + + up_account = up_item.up_accounts.find_by(id: params[:up_account_id]) + unless up_account + redirect_to accounts_path, alert: t(".no_account_selected") + return + end + + if account.account_providers.exists? + redirect_to accounts_path, alert: t(".account_already_linked") + return + end + + if up_account.account_provider.present? + redirect_to accounts_path, alert: t(".up_account_already_linked") + return + end + + AccountProvider.create!(account: account, provider: up_account) + up_item.sync_later + + redirect_to safe_return_to_path || accounts_path, notice: t(".success", account_name: account.name) + end + + # Render the post-sync setup screen for accounts still needing a decision. + def setup_accounts + @api_error = fetch_up_accounts_from_api(@up_item) + @up_accounts = @up_item.up_accounts.needs_setup.order(:name) + @account_type_options = [ + [ t(".account_types.skip"), "skip" ], + [ t(".account_types.depository"), "Depository" ], + [ t(".account_types.loan"), "Loan" ] + ] + @up_account_type_suggestions = @up_accounts.each_with_object({}) do |up_account, suggestions| + suggestions[up_account.id] = up_account.suggested_account_type || "skip" + end + end + + # Apply the user's per-account setup choices (create/link or skip). + def complete_account_setup + account_types = params[:account_types] || {} + created_accounts = [] + skipped_count = 0 + + ActiveRecord::Base.transaction do + account_types.each do |up_account_id, selected_type| + up_account = @up_item.up_accounts.find_by(id: up_account_id) + next unless up_account + + if selected_type.blank? || selected_type == "skip" + # Persist the skip so the account stops resurfacing as "needs setup" on every sync. + up_account.update!(ignored: true) unless up_account.account_provider.present? + skipped_count += 1 + next + end + + next unless Provider::UpAdapter.supported_account_types.include?(selected_type) + next if up_account.account_provider.present? + + account = create_account_from_up(up_account, selected_type) + AccountProvider.create!(account: account, provider: up_account) + created_accounts << account + end + end + + @up_item.sync_later if created_accounts.any? + + flash[:notice] = if created_accounts.any? + t(".success", count: created_accounts.count) + elsif skipped_count.positive? + t(".all_skipped") + else + t(".no_accounts") + end + + redirect_to accounts_path, status: :see_other + rescue ActiveRecord::RecordInvalid, ActiveRecord::RecordNotSaved => e + DebugLogEntry.capture( + category: "provider_sync_error", + level: "error", + message: "Up account setup failed", + source: self.class.name, + provider_key: "up", + family: @up_item&.family, + metadata: { up_item_id: @up_item&.id, error_class: e.class.name, error_message: e.message } + ) + redirect_to accounts_path, alert: t(".creation_failed"), status: :see_other + end + + private + + # Load the requested item scoped to the current family. + def set_up_item + @up_item = Current.family.up_items.find(params[:id]) + end + + # Strong params for creating/updating a connection. + def up_item_params + params.require(:up_item).permit(:name, :sync_start_date, :access_token) + end + + # Params for update, dropping a blank token so it isn't overwritten. + def update_params + permitted = up_item_params + permitted = permitted.except(:access_token) if permitted[:access_token].blank? + permitted + end + + # Load the active item referenced by up_item_id, scoped to the family. + def requested_up_item + Current.family.up_items.active.find_by!(id: params[:up_item_id]) + end + + # Fetch and upsert account snapshots from the API; returns an error string or nil. + def fetch_up_accounts_from_api(up_item) + return t("up_items.setup_accounts.no_credentials") unless up_item.credentials_configured? + + provider = up_item.up_provider + accounts = provider.get_accounts + accounts.each do |account_data| + account = account_data.with_indifferent_access + account_id = account[:id].presence + next if account_id.blank? || account[:displayName].blank? + + up_account = up_item.up_accounts.find_or_initialize_by(account_id: account_id.to_s) + up_account.upsert_up_snapshot!(account) + end + + nil + rescue Provider::Up::UpError => e + DebugLogEntry.capture( + category: "provider_sync_error", + level: "error", + message: "Up API error while fetching accounts", + source: self.class.name, + provider_key: "up", + family: up_item.family, + metadata: { up_item_id: up_item.id, error_class: e.class.name, error_message: e.message } + ) + t("up_items.setup_accounts.api_error") + rescue StandardError => e + DebugLogEntry.capture( + category: "provider_sync_error", + level: "error", + message: "Unexpected error fetching Up accounts", + source: self.class.name, + provider_key: "up", + family: up_item.family, + metadata: { up_item_id: up_item.id, error_class: e.class.name, error_message: e.message } + ) + t("up_items.setup_accounts.api_error") + end + + # Create and sync a Sure account from an Up account snapshot. + def create_account_from_up(up_account, account_type) + # Linking an account clears any prior skip so a future unlink re-prompts for setup. + up_account.update!(ignored: false) if up_account.ignored? + + balance = up_account.current_balance || 0 + balance = balance.abs if account_type == "Loan" + subtype = if account_type == "Depository" && up_account.suggested_account_type == account_type + up_account.suggested_subtype + end + + Account.create_and_sync( + { + family: Current.family, + name: up_account.name, + balance: balance, + cash_balance: balance, + currency: up_account.currency || "AUD", + accountable_type: account_type, + accountable_attributes: subtype.present? ? { subtype: subtype } : {} + }, + skip_initial_sync: true + ) + end + + # Re-render the providers settings panel (Turbo) or redirect with a flash. + def render_provider_panel(flash_type, message) + if turbo_frame_request? + flash.now[flash_type] = message + @up_items = Current.family.up_items.active.ordered + render turbo_stream: [ + turbo_stream.replace( + "up-providers-panel", + partial: "settings/providers/up_panel", + locals: { up_items: @up_items } + ), + *flash_notification_stream_items + ] + else + redirect_to settings_providers_path, { flash_type => message, status: :see_other } + end + end + + # Re-render the providers panel with an error (Turbo) or redirect with alert. + def render_provider_panel_error(message) + @error_message = message + if turbo_frame_request? + render turbo_stream: turbo_stream.replace( + "up-providers-panel", + partial: "settings/providers/up_panel", + locals: { error_message: @error_message } + ), status: :unprocessable_entity + else + redirect_to settings_providers_path, alert: @error_message, status: :see_other + end + end + + # Validate the return_to param as a safe in-app relative path, or nil. + def safe_return_to_path + return nil if params[:return_to].blank? + + return_to = params[:return_to].to_s.strip + return nil unless return_to.start_with?("/") + return nil if return_to[1] == "/" || return_to[1] == "\\" + return nil if return_to.include?("\\") || return_to.match?(/[[:cntrl:]]/) + return nil if encoded_path_separator?(return_to) + + uri = URI.parse(return_to) + return nil unless uri.relative? + + Rails.application.routes.recognize_path(uri.path, method: :get) + + return_to + rescue URI::InvalidURIError, ActionController::RoutingError + nil + end + + # True if the path's second char is a percent-encoded slash/backslash + # (used to block protocol-relative redirect bypasses). + def encoded_path_separator?(return_to) + encoded_second_character = return_to[1, 3] + return false unless encoded_second_character&.start_with?("%") + + decoded = URI.decode_www_form_component(encoded_second_character) + decoded == "/" || decoded == "\\" + rescue ArgumentError + true + end +end diff --git a/app/helpers/settings_helper.rb b/app/helpers/settings_helper.rb index 7d006ad58..6d821ecdb 100644 --- a/app/helpers/settings_helper.rb +++ b/app/helpers/settings_helper.rb @@ -60,6 +60,9 @@ module SettingsHelper when "akahu" return { status: :off } unless @akahu_items&.any? sync_based_summary(key) + when "up" + return { status: :off } unless @up_items&.any? + sync_based_summary(key) when "simplefin" return { status: :off } unless @simplefin_items&.any? sync_based_summary(key) diff --git a/app/models/account/provider_import_adapter.rb b/app/models/account/provider_import_adapter.rb index acd2fd7cf..61a095e32 100644 --- a/app/models/account/provider_import_adapter.rb +++ b/app/models/account/provider_import_adapter.rb @@ -770,6 +770,7 @@ class Account::ProviderImportAdapter OR (transactions.extra -> 'lunchflow' ->> 'pending')::boolean = true OR (transactions.extra -> 'enable_banking' ->> 'pending')::boolean = true OR (transactions.extra -> 'akahu' ->> 'pending')::boolean = true + OR (transactions.extra -> 'up' ->> 'pending')::boolean = true SQL .order(date: :desc) # Prefer most recent pending transaction @@ -818,6 +819,7 @@ class Account::ProviderImportAdapter OR (transactions.extra -> 'lunchflow' ->> 'pending')::boolean = true OR (transactions.extra -> 'enable_banking' ->> 'pending')::boolean = true OR (transactions.extra -> 'akahu' ->> 'pending')::boolean = true + OR (transactions.extra -> 'up' ->> 'pending')::boolean = true SQL # If merchant_id is provided, prioritize matching by merchant @@ -889,6 +891,7 @@ class Account::ProviderImportAdapter OR (transactions.extra -> 'lunchflow' ->> 'pending')::boolean = true OR (transactions.extra -> 'enable_banking' ->> 'pending')::boolean = true OR (transactions.extra -> 'akahu' ->> 'pending')::boolean = true + OR (transactions.extra -> 'up' ->> 'pending')::boolean = true SQL # For low confidence, require BOTH merchant AND name match (stronger signal needed) diff --git a/app/models/data_enrichment.rb b/app/models/data_enrichment.rb index 66c364660..59de42251 100644 --- a/app/models/data_enrichment.rb +++ b/app/models/data_enrichment.rb @@ -7,6 +7,7 @@ class DataEnrichment < ApplicationRecord simplefin: "simplefin", lunchflow: "lunchflow", akahu: "akahu", + up: "up", synth: "synth", ai: "ai", enable_banking: "enable_banking", diff --git a/app/models/family.rb b/app/models/family.rb index c5f8f2252..082316c1e 100644 --- a/app/models/family.rb +++ b/app/models/family.rb @@ -3,6 +3,7 @@ class Family < ApplicationRecord include PlaidConnectable, SimplefinConnectable, LunchflowConnectable, AkahuConnectable, EnableBankingConnectable include CoinbaseConnectable, BinanceConnectable, KrakenConnectable, CoinstatsConnectable, SnaptradeConnectable, MercuryConnectable, BrexConnectable, SophtronConnectable include IndexaCapitalConnectable, IbkrConnectable + include UpConnectable DATE_FORMATS = [ [ "MM-DD-YYYY", "%m-%d-%Y" ], diff --git a/app/models/family/financial_data_reset.rb b/app/models/family/financial_data_reset.rb index 021f1eab9..75141a5e4 100644 --- a/app/models/family/financial_data_reset.rb +++ b/app/models/family/financial_data_reset.rb @@ -54,6 +54,7 @@ class Family::FinancialDataReset simplefin_items snaptrade_items sophtron_items + up_items ].freeze Result = Struct.new(:user, :family, :dry_run, :before_counts, :deleted_counts, :after_counts, keyword_init: true) diff --git a/app/models/family/up_connectable.rb b/app/models/family/up_connectable.rb new file mode 100644 index 000000000..4f56e2398 --- /dev/null +++ b/app/models/family/up_connectable.rb @@ -0,0 +1,28 @@ +module Family::UpConnectable + extend ActiveSupport::Concern + + included do + has_many :up_items, dependent: :destroy + end + + # Whether this family may connect Up Bank accounts (always true). + def can_connect_up? + true + end + + # Create an Up item with the given token and trigger an initial sync. + def create_up_item!(access_token:, item_name: nil) + up_item = up_items.create!( + name: item_name || I18n.t("family.up.create_up_item.default_name"), + access_token: access_token + ) + + up_item.sync_later + up_item + end + + # True when any active Up item has usable credentials. + def has_up_credentials? + up_items.active.any?(&:credentials_configured?) + end +end diff --git a/app/models/provider/metadata.rb b/app/models/provider/metadata.rb index 805e84397..21d9436bf 100644 --- a/app/models/provider/metadata.rb +++ b/app/models/provider/metadata.rb @@ -4,6 +4,7 @@ class Provider akahu: { region: "NZ", kinds: %w[Bank Investment], maturity: :beta, logo_text: "AK", logo_bg: "bg-emerald-600" }, simplefin: { region: "US", kinds: %w[Bank Investment], maturity: :stable, logo_text: "SF", logo_bg: "bg-blue-600" }, lunchflow: { region: "Global", kinds: %w[Bank], maturity: :stable, logo_text: "LF", logo_bg: "bg-orange-500" }, + up: { region: "AU", kinds: %w[Bank], maturity: :beta, logo_text: "UP", logo_bg: "bg-orange-600" }, enable_banking: { region: "EU", kinds: %w[Bank], maturity: :beta, logo_text: "EB", logo_bg: "bg-purple-600" }, coinstats: { region: "Global", kinds: %w[Crypto], maturity: :beta, logo_text: "CS", logo_bg: "bg-pink-600" }, mercury: { region: "US", kinds: %w[Bank], maturity: :beta, logo_text: "ME", logo_bg: "bg-cyan-600" }, diff --git a/app/models/provider/up.rb b/app/models/provider/up.rb new file mode 100644 index 000000000..1346b03c8 --- /dev/null +++ b/app/models/provider/up.rb @@ -0,0 +1,224 @@ +class Provider::Up + include HTTParty + extend SslConfigurable + + DEFAULT_BASE_URL = "https://api.up.com.au/api/v1".freeze + DEFAULT_PAGE_SIZE = 100 + # Host that authenticated requests (bearer token) may be sent to. Absolute URLs + # taken from API responses (links.next) are validated against this. + ALLOWED_HOST = URI.parse(DEFAULT_BASE_URL).host.freeze + + headers "User-Agent" => "Sure Finance Up Client" + default_options.merge!({ timeout: 120 }.merge(httparty_ssl_options)) + + attr_reader :access_token + + # Build a client with the family's Up personal access token. Raises if blank. + def initialize(access_token) + @access_token = access_token.to_s.strip + + raise UpError.new("Up access token is required", :configuration_error) if @access_token.blank? + end + + # GET /util/ping - validates the personal access token. + # Returns the parsed payload (contains meta.id / meta.statusEmoji) or raises UpError. + def ping + get("util/ping") + end + + # GET /accounts - returns an array of flattened account hashes. + # Each hash: { id:, displayName:, accountType:, ownershipType:, balance: {...}, createdAt: } + def get_accounts + fetch_all_resources("accounts").map { |resource| flatten_account(resource) } + end + + # GET /accounts/{id}/transactions - returns an array of flattened transaction hashes. + # Both HELD (pending) and SETTLED (posted) transactions are returned; callers derive + # pending status from the :status field. + def get_account_transactions(account_id:, since: nil, until_date: nil, page_size: DEFAULT_PAGE_SIZE) + query = { "page[size]" => page_size } + query["filter[since]"] = format_api_time(since) if since.present? + query["filter[until]"] = format_api_time(until_date) if until_date.present? + + path = "accounts/#{ERB::Util.url_encode(account_id.to_s)}/transactions" + fetch_all_resources(path, query: query).map { |resource| flatten_transaction(resource) } + end + + private + + RETRYABLE_ERRORS = [ + SocketError, + Net::OpenTimeout, + Net::ReadTimeout, + Errno::ECONNRESET, + Errno::ECONNREFUSED, + Errno::ETIMEDOUT, + EOFError + ].freeze + + MAX_RETRIES = 3 + INITIAL_RETRY_DELAY = 2 + + # Follows JSON:API cursor pagination via links.next (absolute URLs) until exhausted, + # concatenating each page's `data` array. + def fetch_all_resources(path, query: {}) + results = [] + payload = get(path, query: query.presence) + seen_urls = Set.new + + loop do + results.concat(Array(payload[:data])) + next_url = payload.dig(:links, :next) + break if next_url.blank? + # Guard against an API that returns the same cursor repeatedly. + break unless seen_urls.add?(next_url) + + payload = get(next_url) + end + + results + end + + # Flattens a JSON:API account resource into a single hash with id + attributes. + def flatten_account(resource) + data = resource.with_indifferent_access + attributes = data[:attributes].is_a?(Hash) ? data[:attributes] : {} + + attributes.merge( + id: data[:id], + type: data[:type] + ).with_indifferent_access + end + + # Flattens a JSON:API transaction resource, lifting attributes to the top level and + # extracting the related account/category ids from relationships. + def flatten_transaction(resource) + data = resource.with_indifferent_access + attributes = data[:attributes].is_a?(Hash) ? data[:attributes] : {} + + attributes.merge( + id: data[:id], + account_id: data.dig(:relationships, :account, :data, :id), + category_id: data.dig(:relationships, :category, :data, :id) + ).with_indifferent_access + end + + def format_api_time(value) + return value if value.is_a?(String) + # A bare Date has no time/zone, so interpret it as UTC midnight rather than + # the server's local zone (which would shift `filter[since]` by the offset). + return value.to_time(:utc).iso8601 if value.instance_of?(Date) + + value.to_time.utc.iso8601 + end + + # Issues a GET request. `path_or_url` may be a relative path (prefixed with the base URL) + # or an absolute URL (used when following pagination links). + def get(path_or_url, query: nil) + with_retries("GET #{path_or_url}") do + url = resolve_url(path_or_url) + response = self.class.get(url, headers: auth_headers, query: query) + handle_response(response) + end + end + + # Resolves a relative path against the base URL, or validates an absolute URL + # so the bearer token is only ever sent to Up's HTTPS host. + def resolve_url(path_or_url) + value = path_or_url.to_s + return "#{DEFAULT_BASE_URL}/#{value}" unless value.start_with?("http") + + uri = URI.parse(value) + unless uri.scheme == "https" && uri.host == ALLOWED_HOST + raise UpError.new("Refusing to send credentials to untrusted host: #{uri.host.inspect}", :invalid_url) + end + + value + rescue URI::InvalidURIError + raise UpError.new("Invalid Up API URL", :invalid_url) + end + + # Bearer-auth headers sent with every request. + def auth_headers + { + "Authorization" => "Bearer #{access_token}", + "Accept" => "application/json" + } + end + + # Run the block, retrying transient network errors with exponential backoff. + def with_retries(operation_name, max_retries: MAX_RETRIES) + retries = 0 + + begin + yield + rescue *RETRYABLE_ERRORS => e + retries += 1 + if retries <= max_retries + delay = calculate_retry_delay(retries) + Rails.logger.warn( + "Up API: #{operation_name} failed (attempt #{retries}/#{max_retries}): " \ + "#{e.class}: #{e.message}. Retrying in #{delay}s..." + ) + Kernel.sleep(delay) + retry + end + + Rails.logger.error("Up API: #{operation_name} failed after #{max_retries} retries: #{e.class}: #{e.message}") + raise UpError.new("Network error after #{max_retries} retries: #{e.message}", :network_error) + end + end + + # Exponential backoff delay (with jitter), capped at 30 seconds. + def calculate_retry_delay(retry_count) + base_delay = INITIAL_RETRY_DELAY * (2 ** (retry_count - 1)) + jitter = base_delay * rand * 0.25 + [ base_delay + jitter, 30 ].min + end + + # Map an HTTP response to parsed data or a typed UpError by status code. + def handle_response(response) + case response.code + when 200, 201 + parse_response_body(response) + when 204 + {} + when 400 + raise UpError.new("Bad request to Up API (status=#{response.code})", :bad_request) + when 401 + raise UpError.new("Invalid Up access token", :unauthorized) + when 403 + raise UpError.new("Up access forbidden - check token permissions", :access_forbidden) + when 404 + raise UpError.new("Up resource not found", :not_found) + when 429 + raise UpError.new("Up rate limit exceeded. Please try again later.", :rate_limited) + when 500..599 + raise UpError.new("Up server error (#{response.code}). Please try again later.", :server_error) + else + Rails.logger.error "Up API: Unexpected response status=#{response.code}" + raise UpError.new("Failed to fetch Up data", :fetch_failed) + end + end + + # Parse a JSON response body into a symbol-keyed hash, raising on bad JSON. + def parse_response_body(response) + return {} if response.body.blank? + + JSON.parse(response.body, symbolize_names: true) + rescue JSON::ParserError => e + Rails.logger.error "Up API: Failed to parse response: #{e.class}" + raise UpError.new("Failed to parse Up API response", :parse_error) + end + + # Error raised for Up API failures, tagged with a symbolic +error_type+. + class UpError < StandardError + attr_reader :error_type + + # Build the error with a +message+ and a categorizing +error_type+. + def initialize(message, error_type = :unknown) + super(message) + @error_type = error_type + end + end +end diff --git a/app/models/provider/up_adapter.rb b/app/models/provider/up_adapter.rb new file mode 100644 index 000000000..1f0c2981b --- /dev/null +++ b/app/models/provider/up_adapter.rb @@ -0,0 +1,111 @@ +class Provider::UpAdapter < Provider::Base + include Provider::Syncable + include Provider::InstitutionMetadata + + Provider::Factory.register("UpAccount", self) + + # Sure accountable types that can be created from Up accounts. + def self.supported_account_types + %w[Depository Loan] + end + + # Connection config hashes for each of the family's configured Up items. + def self.connection_configs(family:) + return [] unless family.can_connect_up? + + family.up_items.active.ordered.select(&:credentials_configured?).map do |up_item| + connection_config_for(up_item) + end + end + + # Build an Up API client for the resolved item, or nil if none is usable. + def self.build_provider(family: nil, up_item_id: nil) + return nil unless family.present? + + up_item = resolve_up_item(family, up_item_id) + return nil unless up_item&.credentials_configured? + + Provider::Up.new(up_item.access_token) + end + + # Build the settings connection-config hash for a single Up item. + def self.connection_config_for(up_item) + path_params = ->(extra = {}) { extra.merge(up_item_id: up_item.id) } + + { + key: "up_#{up_item.id}", + name: up_item.name.presence || I18n.t("providers.up.name"), + description: I18n.t("providers.up.description"), + can_connect: true, + new_account_path: ->(accountable_type, return_to) { + Rails.application.routes.url_helpers.select_accounts_up_items_path( + path_params.call(accountable_type: accountable_type, return_to: return_to) + ) + }, + existing_account_path: ->(account_id) { + Rails.application.routes.url_helpers.select_existing_account_up_items_path( + path_params.call(account_id: account_id) + ) + } + } + end + private_class_method :connection_config_for + + # Provider key used across the sync/account-provider machinery. + def provider_name + "up" + end + + # Route to trigger a manual sync for this provider account's item. + def sync_path + Rails.application.routes.url_helpers.sync_up_item_path(item) + end + + # The UpItem backing this provider account. + def item + provider_account.up_item + end + + # Up holdings are never deletable by the sync machinery. + def can_delete_holdings? + false + end + + # Institution domain from account metadata, or nil. + def institution_domain + metadata = provider_account.institution_metadata + return nil unless metadata.present? + + metadata["domain"] + end + + # Institution name from account metadata, falling back to the item's. + def institution_name + metadata = provider_account.institution_metadata + metadata&.dig("name").presence || item&.institution_name + end + + # Institution URL from account metadata, falling back to the item's. + def institution_url + metadata = provider_account.institution_metadata + metadata&.dig("url").presence || item&.institution_url + end + + # Brand color for the institution, from the item. + def institution_color + item&.institution_color + end + + # Resolve the target Up item: the requested one, else the first configured. + def self.resolve_up_item(family, up_item_id) + if up_item_id.present? + item = family.up_items.active.find_by(id: up_item_id) + return item if item&.credentials_configured? + + return nil + end + + family.up_items.active.ordered.find(&:credentials_configured?) + end + private_class_method :resolve_up_item +end diff --git a/app/models/provider_connection_status.rb b/app/models/provider_connection_status.rb index 0c3720323..a995ace61 100644 --- a/app/models/provider_connection_status.rb +++ b/app/models/provider_connection_status.rb @@ -3,6 +3,7 @@ class ProviderConnectionStatus PROVIDERS = [ { key: "akahu", type: "AkahuItem", association: :akahu_items, accounts: :akahu_accounts }, + { key: "up", type: "UpItem", association: :up_items, accounts: :up_accounts }, { key: "plaid", type: "PlaidItem", association: :plaid_items, accounts: :plaid_accounts }, { key: "simplefin", type: "SimplefinItem", association: :simplefin_items, accounts: :simplefin_accounts }, { key: "lunchflow", type: "LunchflowItem", association: :lunchflow_items, accounts: :lunchflow_accounts }, diff --git a/app/models/provider_merchant.rb b/app/models/provider_merchant.rb index 261a1e660..719c6800f 100644 --- a/app/models/provider_merchant.rb +++ b/app/models/provider_merchant.rb @@ -1,5 +1,5 @@ class ProviderMerchant < Merchant - enum :source, { plaid: "plaid", simplefin: "simplefin", lunchflow: "lunchflow", akahu: "akahu", synth: "synth", ai: "ai", enable_banking: "enable_banking", coinstats: "coinstats", mercury: "mercury", brex: "brex", indexa_capital: "indexa_capital", sophtron: "sophtron" } + enum :source, { plaid: "plaid", simplefin: "simplefin", lunchflow: "lunchflow", akahu: "akahu", up: "up", synth: "synth", ai: "ai", enable_banking: "enable_banking", coinstats: "coinstats", mercury: "mercury", brex: "brex", indexa_capital: "indexa_capital", sophtron: "sophtron" } validates :name, uniqueness: { scope: [ :source ] } validates :source, presence: true diff --git a/app/models/transaction.rb b/app/models/transaction.rb index e564e3ea1..040a1ea69 100644 --- a/app/models/transaction.rb +++ b/app/models/transaction.rb @@ -94,7 +94,7 @@ class Transaction < ApplicationRecord INTERNAL_MOVEMENT_LABELS = [ "Transfer", "Sweep In", "Sweep Out", "Exchange" ].freeze # Providers that support pending transaction flags - PENDING_PROVIDERS = %w[simplefin plaid lunchflow enable_banking akahu].freeze + PENDING_PROVIDERS = %w[simplefin plaid lunchflow enable_banking akahu up].freeze # Pre-computed SQL fragment for subqueries that check if a transaction (aliased as "t") is pending. # Stored as a constant so static analysis can verify it contains no user input. diff --git a/app/models/up_account.rb b/app/models/up_account.rb new file mode 100644 index 000000000..d88a35c80 --- /dev/null +++ b/app/models/up_account.rb @@ -0,0 +1,93 @@ +class UpAccount < ApplicationRecord + include CurrencyNormalizable, Encryptable + + # Maps Up `accountType` values to Sure accountable types/subtypes. + UP_ACCOUNT_TYPE_MAP = { + "TRANSACTIONAL" => { accountable_type: "Depository", subtype: "checking" }, + "SAVER" => { accountable_type: "Depository", subtype: "savings" }, + "HOME_LOAN" => { accountable_type: "Loan" } + }.freeze + + INSTITUTION_NAME = "Up".freeze + INSTITUTION_DOMAIN = "up.com.au".freeze + + if encryption_ready? + encrypts :raw_payload + encrypts :raw_transactions_payload + end + + belongs_to :up_item + + has_one :account_provider, as: :provider, dependent: :destroy + has_one :account, through: :account_provider, source: :account + has_one :linked_account, through: :account_provider, source: :account + + validates :name, :currency, presence: true + validates :account_id, uniqueness: { scope: :up_item_id, allow_nil: true } + + # Up accounts with no linked Sure account. + scope :unlinked, -> { left_joins(:account_provider).where(account_providers: { id: nil }) } + # Unlinked accounts that still need a setup decision (i.e. not explicitly skipped). + scope :needs_setup, -> { unlinked.where(ignored: false) } + + # The linked Sure account, if any. + def current_account + account + end + + # Suggested Sure accountable type derived from Up's account type, or nil. + def suggested_account_type + UP_ACCOUNT_TYPE_MAP[account_type.to_s.upcase]&.fetch(:accountable_type) + end + + # Suggested Sure subtype (e.g. checking/savings) for this Up account, or nil. + def suggested_subtype + UP_ACCOUNT_TYPE_MAP[account_type.to_s.upcase]&.[](:subtype) + end + + # Persist the latest Up account snapshot, normalizing balance/currency/metadata. + def upsert_up_snapshot!(account_snapshot) + snapshot = account_snapshot.with_indifferent_access + balance = snapshot[:balance].is_a?(Hash) ? snapshot[:balance].with_indifferent_access : {} + + assign_attributes( + current_balance: parse_balance(balance[:value]), + currency: parse_currency(balance[:currencyCode]) || "AUD", + name: snapshot[:displayName].presence || I18n.t("up_account.fallback"), + account_id: snapshot[:id], + account_status: snapshot[:accountType], + account_type: snapshot[:accountType], + ownership_type: snapshot[:ownershipType], + provider: "up", + institution_metadata: { + name: INSTITUTION_NAME, + domain: INSTITUTION_DOMAIN + }.compact, + raw_payload: account_snapshot + ) + + save! + end + + # Persist the latest raw transactions payload for this account. + def upsert_up_transactions_snapshot!(transactions_snapshot) + assign_attributes(raw_transactions_payload: transactions_snapshot) + save! + end + + private + + # Parse an Up balance string into a BigDecimal, defaulting to 0 on bad input. + def parse_balance(value) + return 0 if value.blank? + + BigDecimal(value.to_s) + rescue ArgumentError + 0 + end + + # CurrencyNormalizable hook: warn when an Up currency code is unrecognized. + def log_invalid_currency(currency_value) + Rails.logger.warn("Invalid currency code '#{currency_value}' for Up account #{id}, defaulting to AUD") + end +end diff --git a/app/models/up_account/processor.rb b/app/models/up_account/processor.rb new file mode 100644 index 000000000..23d0f578b --- /dev/null +++ b/app/models/up_account/processor.rb @@ -0,0 +1,91 @@ +class UpAccount::Processor + include CurrencyNormalizable + + SanitizedProcessingError = Class.new(StandardError) + + attr_reader :up_account + + # Build a processor for the given +up_account+. + def initialize(up_account) + @up_account = up_account + end + + # Sync the linked account's balance and process its transactions. No-op when + # the Up account isn't linked to a Sure account. + def process + unless up_account.current_account.present? + Rails.logger.info "UpAccount::Processor - No linked account for up_account #{up_account.id}, skipping processing" + return + end + + process_account! + process_transactions + rescue StandardError => e + Rails.logger.error "UpAccount::Processor - Failed to process account up_account_id=#{up_account.id} error_class=#{e.class.name}" + report_exception(e, "account") + raise + end + + private + + # Update the linked Sure account's balance/currency from the Up snapshot. + def process_account! + account = up_account.current_account + balance = up_account.current_balance || 0 + + # Loan balances are stored as positive debt in Sure regardless of Up's sign. + balance = balance.abs if account.accountable_type == "Loan" + currency = parse_currency(up_account.currency) || account.currency || "AUD" + + account.update!( + balance: balance, + cash_balance: balance, + currency: currency + ) + end + + # Delegate to the transactions processor, capturing and logging failures. + def process_transactions + UpAccount::Transactions::Processor.new(up_account).process + rescue => e + report_exception(e, "transactions") + Rails.logger.error "UpAccount::Processor - Failed to process transactions up_account_id=#{up_account.id} error_class=#{e.class.name}" + DebugLogEntry.capture( + category: "provider_sync_error", + level: "error", + message: "Failed to process transactions", + source: self.class.name, + provider_key: "up", + family: up_account.up_item.family, + account_provider: up_account.account_provider, + metadata: { up_account_id: up_account.id, error_class: e.class.name, error_message: e.message } + ) + { success: false, failed: 1, errors: [ { error: I18n.t("up_item.errors.account_processing_failed") } ] } + end + + # Report a processing error to Sentry with a sanitized message and tags. + def report_exception(error, context) + safe_error = SanitizedProcessingError.new("Up account processing failed") + + Sentry.capture_exception(safe_error) do |scope| + scope.set_tags( + up_account_id: up_account.id, + context: context, + error_class: error.class.name + ) + scope.set_context( + "up_account_processor", + { + up_account_id: up_account.id, + context: context, + error_class: error.class.name + } + ) + end + end + + # CurrencyNormalizable hook: warn when an Up currency code is unrecognized. + def log_invalid_currency(currency_value) + Rails.logger.warn("Invalid currency code '#{currency_value}' for Up account #{up_account.id}, falling back to account currency") + end +end diff --git a/app/models/up_account/transactions/processor.rb b/app/models/up_account/transactions/processor.rb new file mode 100644 index 000000000..c56fadb4d --- /dev/null +++ b/app/models/up_account/transactions/processor.rb @@ -0,0 +1,91 @@ +class UpAccount::Transactions::Processor + attr_reader :up_account + + # Build a transactions processor for the given +up_account+. + def initialize(up_account) + @up_account = up_account + end + + # Process each stored raw transaction into a Sure entry, prune stale pending + # entries, and return a stats hash (total/imported/failed/pruned/errors). + def process + unless up_account.raw_transactions_payload.present? + Rails.logger.info "UpAccount::Transactions::Processor - No Up transactions available to process" + pruned_count = prune_stale_pending_entries([]) + return { success: true, total: 0, imported: 0, failed: 0, pruned_pending: pruned_count, errors: [] } + end + + total_count = up_account.raw_transactions_payload.count + imported_count = 0 + failed_count = 0 + errors = [] + current_pending_external_ids = pending_external_ids + + up_account.raw_transactions_payload.each_with_index do |transaction_data, index| + result = UpEntry::Processor.new( + transaction_data, + up_account: up_account + ).process + + if result.nil? + failed_count += 1 + errors << { index: index, transaction_id: transaction_id(transaction_data), error: "No linked account" } + else + imported_count += 1 + end + rescue ArgumentError => e + failed_count += 1 + errors << { index: index, transaction_id: transaction_id(transaction_data), error: "Validation error: #{e.message}" } + Rails.logger.error "UpAccount::Transactions::Processor - Validation error processing transaction #{transaction_id(transaction_data)}: #{e.message}" + rescue => e + failed_count += 1 + errors << { index: index, transaction_id: transaction_id(transaction_data), error: "#{e.class}: #{e.message}" } + Rails.logger.error "UpAccount::Transactions::Processor - Error processing transaction #{transaction_id(transaction_data)}: #{e.class} - #{e.message}" + Rails.logger.error e.backtrace.join("\n") + end + pruned_count = prune_stale_pending_entries(current_pending_external_ids) + + { + success: failed_count.zero?, + total: total_count, + imported: imported_count, + failed: failed_count, + pruned_pending: pruned_count, + errors: errors + } + end + + private + + # Extract the Up transaction id from raw data, or "unknown". + def transaction_id(transaction_data) + transaction_data.try(:[], :id) || transaction_data.try(:[], "id") || "unknown" + end + + # Canonical external ids of the currently-HELD (pending) transactions. + def pending_external_ids + up_account.raw_transactions_payload.filter_map do |transaction_data| + next unless transaction_data.is_a?(Hash) + next unless UpEntry::Processor.pending?(transaction_data) + + UpEntry::Processor.canonical_external_id(transaction_data) + end + end + + # Delete previously-imported pending entries no longer present in the latest + # fetch (cancelled/settled holds), returning how many were removed. + def prune_stale_pending_entries(current_pending_external_ids) + account = up_account.current_account + return 0 unless account.present? + + stale_pending_entries = account.entries + .joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'") + .where(source: "up") + .where("(transactions.extra -> 'up' ->> 'pending')::boolean = true") + stale_pending_entries = stale_pending_entries.where.not(external_id: current_pending_external_ids) if current_pending_external_ids.any? + + count = stale_pending_entries.count + stale_pending_entries.find_each(&:destroy!) if count.positive? + count + end +end diff --git a/app/models/up_entry/processor.rb b/app/models/up_entry/processor.rb new file mode 100644 index 000000000..52d6c760c --- /dev/null +++ b/app/models/up_entry/processor.rb @@ -0,0 +1,199 @@ +require "digest/md5" + +class UpEntry::Processor + include CurrencyNormalizable + + # Stable external id for a transaction: its Up id when present, else a content + # hash so pending entries lacking an id stay deduplicated across syncs. + def self.canonical_external_id(up_transaction) + data = up_transaction.with_indifferent_access + id = data[:id].presence + return "up_#{id}" if id.present? + + "up_pending_#{content_hash_for(data)}" + end + + # Up marks unsettled transactions with status "HELD"; settled ones are "SETTLED". + def self.pending?(up_transaction) + data = up_transaction.with_indifferent_access + data[:status].to_s.upcase == "HELD" + end + + # MD5 of account/date/amount/description, used to identify id-less pendings. + def self.content_hash_for(data) + amount = data[:amount].is_a?(Hash) ? data[:amount].with_indifferent_access : {} + attributes = [ + data[:account_id], + data[:createdAt], + amount[:value], + data[:description] + ].compact.join("|") + + Digest::MD5.hexdigest(attributes) + end + + # Build a processor for a single raw Up transaction tied to +up_account+. + def initialize(up_transaction, up_account:) + @up_transaction = up_transaction + @up_account = up_account + end + + # Import the transaction into the linked Sure account via the import adapter. + # Returns nil when the account isn't linked; re-raises on validation/save errors. + def process + unless account.present? + Rails.logger.warn "UpEntry::Processor - No linked account for up_account #{up_account.id}, skipping transaction #{external_id}" + return nil + end + + import_adapter.import_transaction( + external_id: external_id, + amount: amount, + currency: currency, + date: date, + name: name, + source: "up", + merchant: merchant, + notes: notes, + extra: extra_metadata + ) + rescue ArgumentError => e + Rails.logger.error "UpEntry::Processor - Validation error for transaction #{external_id}: #{e.message}" + raise + rescue ActiveRecord::RecordInvalid, ActiveRecord::RecordNotSaved => e + Rails.logger.error "UpEntry::Processor - Failed to save transaction #{external_id}: #{e.message}" + raise StandardError.new("Failed to import transaction: #{e.message}") + rescue => e + Rails.logger.error "UpEntry::Processor - Unexpected error processing transaction #{external_id}: #{e.class} - #{e.message}" + Rails.logger.error e.backtrace.join("\n") + raise StandardError.new("Unexpected error importing transaction: #{e.message}") + end + + private + + attr_reader :up_transaction, :up_account + + # Memoized adapter that writes provider transactions into the Sure account. + def import_adapter + @import_adapter ||= Account::ProviderImportAdapter.new(account) + end + + # The linked Sure account for this transaction, if any. + def account + @account ||= up_account.current_account + end + + # The raw transaction as an indifferent-access hash. + def data + @data ||= up_transaction.with_indifferent_access + end + + # Canonical external id for this transaction (see .canonical_external_id). + def external_id + @external_id ||= self.class.canonical_external_id(data) + end + + # Display name: the Up description, or a generic fallback. + def name + data[:description].presence || I18n.t("transactions.unknown_name") + end + + # Optional user-entered message attached to the transaction. + def notes + data[:message].presence + end + + # Find or create the merchant for this transaction's description, or nil. + def merchant + merchant_name = data[:description].to_s.strip.presence + return nil unless merchant_name + + provider_merchant_id = "up_merchant_#{Digest::MD5.hexdigest(merchant_name.downcase)}" + + @merchant ||= import_adapter.find_or_create_merchant( + provider_merchant_id: provider_merchant_id, + name: merchant_name, + source: "up" + ) + rescue ActiveRecord::RecordInvalid => e + Rails.logger.error "UpEntry::Processor - Failed to create merchant '#{merchant_name}': #{e.message}" + nil + end + + # Up amounts use banking convention: negative is money out, positive is money in. + # Sure stores expenses as positive and income as negative, so the sign is flipped. + def amount + raw_value = amount_data[:value] + parsed_amount = case raw_value + when String + BigDecimal(raw_value) + when Numeric + BigDecimal(raw_value.to_s) + else + BigDecimal("0") + end + + -parsed_amount + rescue ArgumentError => e + Rails.logger.error "Failed to parse Up transaction amount: #{e.class}" + raise ArgumentError, "Invalid transaction amount" + end + + # Transaction currency, falling back to the account/default currency. + def currency + parse_currency(amount_data[:currencyCode]) || up_account.currency || account&.currency || "AUD" + end + + # Settlement date (or creation date for pendings) as a Date. + def date + value = data[:settledAt].presence || data[:createdAt].presence + case value + when String + Time.parse(value).to_date + when Time, DateTime + value.to_date + when Date + value + else + Rails.logger.error("Up transaction has no usable date value") + raise ArgumentError, "Invalid date format" + end + rescue ArgumentError, TypeError => e + Rails.logger.error("Failed to parse Up transaction date: #{e.class}") + raise ArgumentError, "Unable to parse transaction date" + end + + # Provider metadata persisted on Transaction#extra (pending/status/fx/etc.). + def extra_metadata + { + "up" => { + "pending" => pending?, + "status" => data[:status], + "category_id" => data[:category_id], + "raw_text" => data[:rawText], + "fx_from" => foreign_amount_data[:currencyCode], + "fx_amount" => foreign_amount_data[:value] + }.compact + } + end + + # Whether this transaction is still HELD (unsettled) on Up. + def pending? + self.class.pending?(data) + end + + # The native amount object ({ value:, currencyCode: }) as a hash. + def amount_data + @amount_data ||= data[:amount].is_a?(Hash) ? data[:amount].with_indifferent_access : {} + end + + # The foreign-currency amount object for FX transactions, or empty hash. + def foreign_amount_data + @foreign_amount_data ||= data[:foreignAmount].is_a?(Hash) ? data[:foreignAmount].with_indifferent_access : {} + end + + # CurrencyNormalizable hook: warn when an Up currency code is unrecognized. + def log_invalid_currency(currency_value) + Rails.logger.warn("Invalid currency code '#{currency_value}' in Up transaction #{external_id}, falling back to account currency") + end +end diff --git a/app/models/up_item.rb b/app/models/up_item.rb new file mode 100644 index 000000000..27c4e6ce9 --- /dev/null +++ b/app/models/up_item.rb @@ -0,0 +1,201 @@ +class UpItem < ApplicationRecord + include Syncable, Provided, Unlinking, Encryptable + + enum :status, { good: "good", requires_update: "requires_update" }, default: :good + + if encryption_ready? + encrypts :access_token, deterministic: true + encrypts :raw_payload + encrypts :raw_institution_payload + end + + belongs_to :family + has_one_attached :logo, dependent: :purge_later + has_many :up_accounts, dependent: :destroy + has_many :accounts, through: :up_accounts + + validates :name, presence: true + validates :access_token, presence: true, on: :create + + scope :active, -> { where(scheduled_for_deletion: false) } + scope :syncable, -> { active } + scope :ordered, -> { order(created_at: :desc) } + scope :needs_update, -> { where(status: :requires_update) } + + # Mark the item for deletion and enqueue the background destroy job. + def destroy_later + update!(scheduled_for_deletion: true) + DestroyJob.perform_later(self) + end + + # Run the importer to fetch the latest accounts/transactions from Up. + def import_latest_up_data + provider = up_provider + unless provider + DebugLogEntry.capture( + category: "provider_sync_error", + level: "error", + message: "Cannot import: Up provider is not configured", + source: self.class.name, + provider_key: "up", + family: family, + metadata: { up_item_id: id } + ) + raise StandardError.new("Up provider is not configured") + end + + UpItem::Importer.new(self, up_provider: provider).import + rescue => e + DebugLogEntry.capture( + category: "provider_sync_error", + level: "error", + message: "Failed to import data", + source: self.class.name, + provider_key: "up", + family: family, + metadata: { up_item_id: id, error_class: e.class.name, error_message: e.message } + ) + raise + end + + # Process each linked, visible Up account, returning a per-account result array. + def process_accounts + return [] if up_accounts.empty? + + up_accounts.joins(:account).merge(Account.visible).map do |up_account| + result = UpAccount::Processor.new(up_account).process + if result.is_a?(Hash) && result.with_indifferent_access[:success] == false + { up_account_id: up_account.id, success: false, error: I18n.t("up_item.errors.account_processing_failed") } + else + { up_account_id: up_account.id, success: true, result: result } + end + rescue => e + DebugLogEntry.capture( + category: "provider_sync_error", + level: "error", + message: "Failed to process account", + source: self.class.name, + provider_key: "up", + family: family, + account_provider: up_account.account_provider, + metadata: { up_item_id: id, up_account_id: up_account.id, error_class: e.class.name, error_message: e.message } + ) + { up_account_id: up_account.id, success: false, error: I18n.t("up_item.errors.account_processing_failed") } + end + end + + # Enqueue a balance sync for each visible linked account, returning per-account results. + def schedule_account_syncs(parent_sync: nil, window_start_date: nil, window_end_date: nil) + return [] if accounts.empty? + + accounts.visible.map do |account| + account.sync_later( + parent_sync: parent_sync, + window_start_date: window_start_date, + window_end_date: window_end_date + ) + { account_id: account.id, success: true } + rescue => e + DebugLogEntry.capture( + category: "provider_sync_error", + level: "error", + message: "Failed to schedule sync for account", + source: self.class.name, + provider_key: "up", + family: family, + account: account, + metadata: { up_item_id: id, account_id: account.id, error_class: e.class.name, error_message: e.message } + ) + { account_id: account.id, success: false, error: I18n.t("up_item.errors.account_sync_schedule_failed") } + end + end + + # Persist the latest raw accounts payload for this item. + def upsert_up_snapshot!(accounts_snapshot) + assign_attributes(raw_payload: accounts_snapshot) + save! + end + + # True once at least one Up account has been linked to a Sure account. + def has_completed_initial_setup? + accounts.any? + end + + # Human-readable summary of linked vs. unlinked account counts. + def sync_status_summary + total_accounts = total_accounts_count + linked_count = linked_accounts_count + unlinked_count = unlinked_accounts_count + + if total_accounts.zero? + I18n.t("up_item.sync_status.no_accounts") + elsif unlinked_count.zero? + I18n.t("up_item.sync_status.all_synced", count: linked_count) + else + I18n.t("up_item.sync_status.partial", linked: linked_count, unlinked: unlinked_count) + end + end + + # Number of Up accounts linked to a Sure account. + def linked_accounts_count + account_counts[:linked] + end + + # Number of unlinked Up accounts still awaiting setup. + def unlinked_accounts_count + account_counts[:unlinked] + end + + # Total number of Up accounts under this item. + def total_accounts_count + account_counts[:total] + end + + # Best available display name for the connected institution. + def institution_display_name + institution_name.presence || institution_domain.presence || name + end + + # Distinct institution metadata across this item's accounts. + def connected_institutions + up_accounts.includes(:account) + .where.not(institution_metadata: nil) + .map(&:institution_metadata) + .uniq { |inst| inst["name"] || inst["domain"] } + end + + # Human-readable summary of connected institutions (none/one/count). + def institution_summary + institutions = connected_institutions + case institutions.count + when 0 + I18n.t("up_item.institution_summary.none") + when 1 + institutions.first["name"].presence || I18n.t("up_item.institution_summary.one") + else + I18n.t("up_item.institution_summary.count", count: institutions.count) + end + end + + # True when an access token is present and the item can call the Up API. + def credentials_configured? + access_token.present? + end + + private + + # Single query for all three account counts, reused across sync_status_summary + # and the settings partial to avoid 3+ separate COUNT queries per rendered item. + def account_counts + @account_counts ||= begin + rows = up_accounts + .left_joins(:account_provider) + .pluck(Arel.sql("account_providers.id IS NOT NULL"), :ignored) + + linked = rows.count { |has_provider, _ignored| has_provider } + unlinked = rows.count { |has_provider, ignored| !has_provider && !ignored } + + { linked: linked, unlinked: unlinked, total: rows.size } + end + end +end diff --git a/app/models/up_item/importer.rb b/app/models/up_item/importer.rb new file mode 100644 index 000000000..cc4b816ea --- /dev/null +++ b/app/models/up_item/importer.rb @@ -0,0 +1,250 @@ +# Imports Up Bank accounts and transactions for a single UpItem connection. +# Fetches account snapshots and per-account transaction history from the Up +# provider, persisting raw snapshots and returning aggregate import statistics. +class UpItem::Importer + attr_reader :up_item, :up_provider + + # Build an importer for the given +up_item+ using the supplied +up_provider+ client. + def initialize(up_item, up_provider:) + @up_item = up_item + @up_provider = up_provider + end + + # Run the full import (accounts then transactions) and return a result hash + # of success flag and per-entity counts. On a failed accounts fetch, returns + # a +failed_result+ with the same shape and zeroed counts. + def import + Rails.logger.info "UpItem::Importer - Starting import for item #{up_item.id}" + + accounts_data = fetch_accounts_data + return failed_result("Failed to fetch accounts data") unless accounts_data + + up_item.upsert_up_snapshot!(accounts_data) + + account_stats = import_accounts(accounts_data) + transaction_stats = import_transactions + + Rails.logger.info( + "UpItem::Importer - Completed import for item #{up_item.id}: " \ + "#{account_stats[:updated]} accounts updated, #{account_stats[:created]} new accounts discovered, " \ + "#{transaction_stats[:imported]} transactions" + ) + + { + success: account_stats[:failed].zero? && transaction_stats[:failed].zero?, + accounts_updated: account_stats[:updated], + accounts_created: account_stats[:created], + accounts_failed: account_stats[:failed], + transactions_imported: transaction_stats[:imported], + transactions_failed: transaction_stats[:failed] + } + end + + private + + # Fetch the current account list from Up, returning a hash of +items+ or + # +nil+ on any provider/parse error (which is logged and captured). + def fetch_accounts_data + items = up_provider.get_accounts + { items: items } + rescue Provider::Up::UpError => e + mark_requires_update! if e.error_type.in?([ :unauthorized, :access_forbidden ]) + Rails.logger.error "UpItem::Importer - Up API error: #{e.error_type}" + capture_sync_error("Failed to fetch accounts data", e, error_type: e.error_type) + nil + rescue JSON::ParserError => e + Rails.logger.error "UpItem::Importer - Failed to parse Up API response: #{e.class}" + capture_sync_error("Failed to parse Up accounts response", e) + nil + rescue => e + Rails.logger.error "UpItem::Importer - Unexpected error fetching accounts: #{e.class}" + Rails.logger.error e.backtrace.join("\n") + capture_sync_error("Unexpected error fetching accounts", e) + nil + end + + # Upsert snapshots for linked accounts and record newly discovered ones, + # returning a stats hash of +updated+, +created+, and +failed+ counts. + def import_accounts(accounts_data) + stats = { updated: 0, created: 0, failed: 0 } + accounts = Array(accounts_data[:items]) + linked_account_ids = up_item.up_accounts.joins(:account_provider).pluck(:account_id).map(&:to_s) + all_existing_ids = up_item.up_accounts.pluck(:account_id).map(&:to_s) + + accounts.each do |account_data| + account = account_data.with_indifferent_access + account_id = account[:id].presence + next if account_id.blank? + next if account[:displayName].blank? + + if linked_account_ids.include?(account_id.to_s) + import_account(account) + stats[:updated] += 1 + elsif !all_existing_ids.include?(account_id.to_s) + up_account = up_item.up_accounts.build(account_id: account_id.to_s) + up_account.upsert_up_snapshot!(account) + stats[:created] += 1 + end + rescue => e + stats[:failed] += 1 + Rails.logger.error "UpItem::Importer - Failed to import account #{account_id}: #{e.message}" + end + + stats + end + + # Upsert the snapshot for a single already-linked Up account. + def import_account(account_data) + account = account_data.with_indifferent_access + up_account = up_item.up_accounts.find_by(account_id: account[:id].to_s) + return unless up_account + + up_account.upsert_up_snapshot!(account) + end + + # Fetch and store transactions for every visible linked account, returning + # a stats hash of +imported+ and +failed+ counts. + def import_transactions + stats = { imported: 0, failed: 0 } + + up_item.up_accounts.joins(:account).merge(Account.visible).each do |up_account| + result = fetch_and_store_transactions(up_account) + if result[:success] + stats[:imported] += result[:transactions_count] + else + stats[:failed] += 1 + end + rescue => e + stats[:failed] += 1 + Rails.logger.error "UpItem::Importer - Failed to fetch/store transactions for Up account #{up_account.id}: #{e.class}" + end + + stats + end + + # Fetch transactions for +up_account+ since its sync start date and persist + # them, returning a result hash with +success+ and +transactions_count+. + def fetch_and_store_transactions(up_account) + start_date = determine_sync_start_date(up_account) + Rails.logger.info "UpItem::Importer - Fetching transactions for Up account #{up_account.id} since #{start_date}" + + transactions = up_provider.get_account_transactions( + account_id: up_account.account_id, + since: start_date + ) + + if Rails.configuration.x.up.debug_raw && Rails.env.local? + Rails.logger.debug "Up raw transactions response: #{transactions.to_json}" + end + + store_transactions(up_account, fresh_transactions: Array(transactions)) + + { success: true, transactions_count: Array(transactions).count } + rescue Provider::Up::UpError => e + mark_requires_update! if e.error_type.in?([ :unauthorized, :access_forbidden ]) + Rails.logger.error "UpItem::Importer - Up API error for account #{up_account.id}: #{e.error_type}" + capture_sync_error("Failed to fetch transactions", e, up_account: up_account, error_type: e.error_type) + { success: false, transactions_count: 0, error: I18n.t("up_item.errors.transactions_failed") } + rescue JSON::ParserError => e + Rails.logger.error "UpItem::Importer - Failed to parse transaction response for account #{up_account.id}: #{e.class}" + capture_sync_error("Failed to parse Up transactions response", e, up_account: up_account) + { success: false, transactions_count: 0, error: "Failed to parse response" } + rescue => e + Rails.logger.error "UpItem::Importer - Unexpected error fetching transactions for account #{up_account.id}: #{e.class}" + Rails.logger.error e.backtrace.join("\n") + capture_sync_error("Unexpected error fetching transactions", e, up_account: up_account) + { success: false, transactions_count: 0, error: I18n.t("up_item.errors.transactions_failed") } + end + + # Up returns both HELD (pending) and SETTLED transactions from the same endpoint. + # Settled history accumulates; HELD transactions are only retained while they remain + # present in the latest fetch, so cancelled/settled holds drop out of storage (and the + # transactions processor prunes their stale pending entries). + def store_transactions(up_account, fresh_transactions:) + existing = up_account.raw_transactions_payload.to_a + existing_settled = existing.reject { |tx| UpEntry::Processor.pending?(tx) } + + by_id = {} + existing_settled.each do |tx| + key = transaction_id(tx) + by_id[key] = tx if key.present? + end + fresh_transactions.each do |tx| + next unless tx.is_a?(Hash) + + key = transaction_id(tx) + by_id[key] = tx if key.present? + end + + final_transactions = by_id.values + + if final_transactions != existing + Rails.logger.info( + "UpItem::Importer - Storing #{final_transactions.count} transactions " \ + "(#{existing.count} existing) for account #{up_account.account_id}" + ) + up_account.upsert_up_transactions_snapshot!(final_transactions) + else + Rails.logger.info "UpItem::Importer - No transaction changes for account #{up_account.account_id}" + end + end + + # Extract the Up transaction id from a raw transaction hash, or +nil+. + def transaction_id(transaction) + data = transaction.with_indifferent_access + data[:id].presence + end + + # Resolve the date from which to fetch transactions for +up_account+, + # preferring explicit per-account/item start dates, then a recent window. + def determine_sync_start_date(up_account) + return up_account.sync_start_date if up_account.sync_start_date.present? + return up_item.sync_start_date if up_item.sync_start_date.present? + + has_stored_transactions = up_account.raw_transactions_payload.to_a.any? + if has_stored_transactions && up_item.last_synced_at + up_item.last_synced_at - 7.days + else + 90.days.ago + end + end + + # Record a provider sync error as a DebugLogEntry with structured metadata + # for support, attaching family and account provider when available. + def capture_sync_error(message, error, up_account: nil, error_type: nil) + metadata = { up_item_id: up_item.id, error_class: error.class.name, error_message: error.message } + metadata[:up_account_id] = up_account.id if up_account + metadata[:error_type] = error_type if error_type + + DebugLogEntry.capture( + category: "provider_sync_error", + level: "error", + message: message, + source: self.class.name, + provider_key: "up", + family: up_item.family, + account_provider: up_account&.account_provider, + metadata: metadata + ) + end + + # Flag the item as requiring re-authorization, swallowing update errors. + def mark_requires_update! + up_item.update!(status: :requires_update) + rescue => e + Rails.logger.error "UpItem::Importer - Failed to update item status: #{e.message}" + end + + # Build a failure result mirroring +import+'s shape with zeroed counts. + def failed_result(error) + { + success: false, + error: error, + accounts_updated: 0, + accounts_created: 0, + accounts_failed: 0, + transactions_imported: 0, + transactions_failed: 0 + } + end +end diff --git a/app/models/up_item/provided.rb b/app/models/up_item/provided.rb new file mode 100644 index 000000000..3efabecb8 --- /dev/null +++ b/app/models/up_item/provided.rb @@ -0,0 +1,15 @@ +module UpItem::Provided + extend ActiveSupport::Concern + + # Build an Up API client from this item's token, or nil if unconfigured. + def up_provider + return nil unless credentials_configured? + + Provider::Up.new(access_token) + end + + # The syncer responsible for importing and processing this item's data. + def syncer + UpItem::Syncer.new(self) + end +end diff --git a/app/models/up_item/sync_complete_event.rb b/app/models/up_item/sync_complete_event.rb new file mode 100644 index 000000000..5c7d4e8d6 --- /dev/null +++ b/app/models/up_item/sync_complete_event.rb @@ -0,0 +1,22 @@ +class UpItem::SyncCompleteEvent + attr_reader :up_item + + # Build the event for the given +up_item+. + def initialize(up_item) + @up_item = up_item + end + + # Broadcast sync-complete Turbo updates for the item, its accounts, and family. + def broadcast + up_item.accounts.each(&:broadcast_sync_complete) + + up_item.broadcast_replace_to( + up_item.family, + target: "up_item_#{up_item.id}", + partial: "up_items/up_item", + locals: { up_item: up_item } + ) + + up_item.family.broadcast_sync_complete + end +end diff --git a/app/models/up_item/syncer.rb b/app/models/up_item/syncer.rb new file mode 100644 index 000000000..b488c6078 --- /dev/null +++ b/app/models/up_item/syncer.rb @@ -0,0 +1,141 @@ +class UpItem::Syncer + include SyncStats::Collector + + SafeSyncError = Class.new(StandardError) + + # Error carrying the structured per-stage sync errors for health reporting. + class SyncError < StandardError + attr_reader :sync_errors + + # Build the error with a +message+ and the collected +sync_errors+. + def initialize(message, sync_errors:) + super(message) + @sync_errors = sync_errors + end + end + + attr_reader :up_item + + # Build a syncer for the given +up_item+. + def initialize(up_item) + @up_item = up_item + end + + # Run the full sync: import, account setup detection, transaction processing, + # balance sync scheduling, and stats/health collection. Raises on failures. + def perform_sync(sync) + sync.update!(status_text: "Importing accounts from Up...") if sync.respond_to?(:status_text) + import_result = up_item.import_latest_up_data + raise_if_failed_result!(import_result, stage: "Up import") + + sync.update!(status_text: "Checking account configuration...") if sync.respond_to?(:status_text) + collect_setup_stats(sync, provider_accounts: up_item.up_accounts) + + linked_accounts = up_item.up_accounts.joins(:account_provider) + unlinked_accounts = up_item.up_accounts.needs_setup + + if unlinked_accounts.any? + up_item.update!(pending_account_setup: true) + sync.update!(status_text: "#{unlinked_accounts.count} accounts need setup...") if sync.respond_to?(:status_text) + else + up_item.update!(pending_account_setup: false) + end + + if linked_accounts.any? + sync.update!(status_text: "Processing transactions...") if sync.respond_to?(:status_text) + mark_import_started(sync) + process_results = up_item.process_accounts + raise_if_failed_results!(process_results, stage: "Up account processing") + + sync.update!(status_text: "Calculating balances...") if sync.respond_to?(:status_text) + schedule_results = up_item.schedule_account_syncs( + parent_sync: sync, + window_start_date: sync.window_start_date, + window_end_date: sync.window_end_date + ) + raise_if_failed_results!(schedule_results, stage: "Up account sync scheduling") + + account_ids = linked_accounts.includes(:account_provider).filter_map { |ua| ua.current_account&.id } + collect_transaction_stats(sync, account_ids: account_ids, source: "up") + else + Rails.logger.info "UpItem::Syncer - No linked accounts to process" + end + + collect_health_stats(sync, errors: nil) + rescue SyncError => e + collect_health_stats(sync, errors: e.sync_errors) + raise + rescue => e + safe_message = I18n.t("up_item.errors.sync_failed") + DebugLogEntry.capture( + category: "provider_sync_error", + level: "error", + message: "Unexpected sync error", + source: self.class.name, + provider_key: "up", + family: up_item.family, + metadata: { up_item_id: up_item.id, error_class: e.class.name, error_message: e.message } + ) + collect_health_stats(sync, errors: [ { message: safe_message, category: "sync_error" } ]) + raise SafeSyncError.new(safe_message), cause: nil + end + + # Post-sync hook (no work required for Up). + def perform_post_sync + # no-op + end + + private + + # Raise a SyncError if a single result hash indicates failure. + def raise_if_failed_result!(result, stage:) + return unless failed_result?(result) + + errors = errors_from_result(result, stage: stage) + raise SyncError.new(error_message(stage, errors), sync_errors: errors) + end + + # Raise a SyncError if any result in the collection indicates failure. + def raise_if_failed_results!(results, stage:) + errors = Array(results).filter_map do |result| + next unless failed_result?(result) + + errors_from_result(result, stage: stage).first + end + + return if errors.empty? + + raise SyncError.new(error_message(stage, errors), sync_errors: errors) + end + + # True when +result+ is a hash explicitly flagged success: false. + def failed_result?(result) + result.is_a?(Hash) && result.with_indifferent_access[:success] == false + end + + # Normalize a failed result into an array of { message:, category: } errors. + def errors_from_result(result, stage:) + data = result.with_indifferent_access + messages = [] + messages << data[:error] if data[:error].present? + messages << "#{data[:accounts_failed]} accounts failed" if data[:accounts_failed].to_i.positive? + messages << "#{data[:transactions_failed]} transactions failed" if data[:transactions_failed].to_i.positive? + messages.concat(Array(data[:errors]).map { |error| error_message_value(error) }.compact) + messages << "#{stage} failed" if messages.empty? + + messages.map { |message| { message: "#{stage}: #{message}", category: "sync_error" } } + end + + # Join the error messages into a single stage summary string. + def error_message(stage, errors) + messages = errors.map { |error| error[:message] || error["message"] }.compact + messages.presence&.join(", ") || "#{stage} failed" + end + + # Extract a human-readable message from a heterogeneous error value. + def error_message_value(error) + return error[:message].presence || error["message"].presence || error[:error].presence || error["error"].presence if error.is_a?(Hash) + + error.to_s.presence + end +end diff --git a/app/models/up_item/unlinking.rb b/app/models/up_item/unlinking.rb new file mode 100644 index 000000000..fdd3d63c1 --- /dev/null +++ b/app/models/up_item/unlinking.rb @@ -0,0 +1,54 @@ +module UpItem::Unlinking + extend ActiveSupport::Concern + + # Unlink every Up account from its Sure account, detaching holdings and + # destroying the AccountProvider links. With +dry_run+, only reports what + # would be unlinked. Returns a per-account result array (with :error on failure). + def unlink_all!(dry_run: false) + results = [] + + up_accounts.find_each do |provider_account| + links = AccountProvider.joins(:account) + .where(provider: provider_account, accounts: { family_id: family_id }) + .to_a + link_ids = links.map(&:id) + result = { + provider_account_id: provider_account.id, + name: provider_account.name, + provider_link_ids: link_ids + } + results << result + + next if dry_run + + begin + ActiveRecord::Base.transaction do + links.each do |link| + Holding.where(account_id: link.account_id, account_provider_id: link.id).update_all(account_provider_id: nil) + link.destroy! + end + end + rescue StandardError => e + DebugLogEntry.capture( + category: "provider_sync_error", + level: "warn", + message: "Failed to fully unlink provider account", + source: self.class.name, + provider_key: "up", + family: family, + account_provider: links.first, + metadata: { + up_item_id: id, + up_account_id: provider_account.id, + link_ids: link_ids, + error_class: e.class.name, + error_message: e.message + } + ) + result[:error] = e.message + end + end + + results + end +end diff --git a/app/views/accounts/index.html.erb b/app/views/accounts/index.html.erb index 609d757eb..0145d1830 100644 --- a/app/views/accounts/index.html.erb +++ b/app/views/accounts/index.html.erb @@ -17,7 +17,7 @@ ) %> <% end %> -<% if @manual_accounts.empty? && @plaid_items.empty? && @simplefin_items.empty? && @lunchflow_items.empty? && @akahu_items.empty? && @enable_banking_items.empty? && @coinstats_items.empty? && @coinbase_items.empty? && @mercury_items.empty? && @brex_items.empty? && @ibkr_items.empty? && @snaptrade_items.empty? && @indexa_capital_items.empty? && @sophtron_items.empty? && @binance_items.empty? %> +<% if @manual_accounts.empty? && @plaid_items.empty? && @simplefin_items.empty? && @lunchflow_items.empty? && @akahu_items.empty? && @up_items.empty? && @enable_banking_items.empty? && @coinstats_items.empty? && @coinbase_items.empty? && @mercury_items.empty? && @brex_items.empty? && @ibkr_items.empty? && @snaptrade_items.empty? && @indexa_capital_items.empty? && @sophtron_items.empty? && @binance_items.empty? %> <%= render "empty" %> <% else %>
@@ -37,6 +37,10 @@ <%= render @akahu_items.sort_by(&:created_at) %> <% end %> + <% if @up_items.any? %> + <%= render @up_items.sort_by(&:created_at) %> + <% end %> + <% if @enable_banking_items.any? %> <%= render @enable_banking_items.sort_by(&:created_at) %> <% end %> diff --git a/app/views/settings/providers/_up_panel.html.erb b/app/views/settings/providers/_up_panel.html.erb new file mode 100644 index 000000000..e11604811 --- /dev/null +++ b/app/views/settings/providers/_up_panel.html.erb @@ -0,0 +1,123 @@ +
+ <% active_items = local_assigns[:up_items] || @up_items || Current.family.up_items.active.ordered %> + + <%= render "settings/providers/setup_steps", + steps: [ + t("settings.providers.up_panel.step_1_html", link: link_to(t("providers.up.name"), "https://api.up.com.au", target: "_blank", rel: "noopener noreferrer", class: "text-primary font-medium underline")), + t("settings.providers.up_panel.step_2"), + t("settings.providers.up_panel.step_3") + ] %> + + <% error_msg = local_assigns[:error_message] || @error_message %> + <% if error_msg.present? %> + <%= render DS::Alert.new(message: error_msg, variant: :error) %> + <% end %> + + <% if active_items.any? %> +
+ <% active_items.each do |item| %> + <%= render DS::Disclosure.new(variant: :card) do |disclosure| %> + <% disclosure.with_summary_content do %> +
+
+
+

<%= item.name.to_s.first.to_s.upcase %>

+
+
+

<%= item.name %>

+

<%= item.sync_status_summary %>

+
+
+
+ <% end %> + +
+
+ <%= render DS::Button.new( + text: t("up_items.provider_panel.sync"), + icon: "refresh-cw", + variant: :outline, + size: :sm, + href: sync_up_item_path(item), + method: :post, + disabled: item.syncing? + ) %> + <%= render DS::Button.new( + text: t("up_items.provider_panel.disconnect"), + icon: "trash-2", + variant: :outline_destructive, + size: :sm, + href: up_item_path(item), + method: :delete, + data: { turbo_confirm: t("up_items.provider_panel.disconnect_confirm", name: item.name) } + ) %> +
+ + <%= styled_form_with model: item, + url: up_item_path(item), + scope: :up_item, + method: :patch, + data: { turbo: true }, + class: "space-y-3" do |form| %> + <%= form.text_field :name, + label: t("up_items.provider_panel.connection_name_label"), + placeholder: t("up_items.provider_panel.connection_name_placeholder") %> + + <%= form.text_field :access_token, + label: t("up_items.provider_panel.access_token_label"), + placeholder: t("up_items.provider_panel.keep_access_token_placeholder"), + type: :password, + value: nil %> + +
+ <%= render DS::Link.new( + text: t("up_items.provider_panel.setup_accounts"), + icon: "settings", + variant: "secondary", + href: setup_accounts_up_item_path(item), + frame: :modal + ) %> + <%= render DS::Button.new( + text: t("up_items.provider_panel.update_connection"), + type: "submit" + ) %> +
+ <% end %> +
+ <% end %> + <% end %> +
+ <% end %> + + <% up_item = Current.family.up_items.build(name: t("up_items.provider_panel.default_connection_name")) %> + <% if active_items.any? %> +

+ <%= icon "plus", size: "sm" %> + <%= t("up_items.provider_panel.add_connection") %> +

+ <% end %> + + <%= styled_form_with model: up_item, + url: up_items_path, + scope: :up_item, + method: :post, + data: { turbo: true }, + class: "space-y-3" do |form| %> + <%= form.text_field :name, + label: t("up_items.provider_panel.connection_name_label"), + placeholder: t("up_items.provider_panel.connection_name_placeholder") %> + + <%= form.text_field :access_token, + label: t("up_items.provider_panel.access_token_label"), + placeholder: t("up_items.provider_panel.access_token_placeholder"), + type: :password, + value: nil %> + +
+ <%= render DS::Button.new( + text: t("up_items.provider_panel.add_connection"), + type: "submit" + ) %> +
+ <% end %> +
diff --git a/app/views/up_items/_up_item.html.erb b/app/views/up_items/_up_item.html.erb new file mode 100644 index 000000000..ddd637c9f --- /dev/null +++ b/app/views/up_items/_up_item.html.erb @@ -0,0 +1,114 @@ +<%# locals: (up_item:) %> + +<%= tag.div id: dom_id(up_item) do %> + <%= render DS::Disclosure.new(variant: :card, open: true) do |disclosure| %> + <% disclosure.with_summary_content do %> +
+
+ <%= icon "chevron-right", class: "group-open:rotate-90 motion-safe:transition-transform motion-safe:duration-150" %> + +
+

<%= up_item.name.to_s.first.to_s.upcase %>

+
+ +
+
+ <%= tag.p up_item.name, class: "font-medium text-primary" %> + <% if up_item.scheduled_for_deletion? %> +

<%= t(".deletion_in_progress") %>

+ <% end %> +
+ + <% if up_item.accounts.any? %> +

<%= up_item.institution_summary %>

+ <% end %> + + <% if up_item.syncing? %> +
+ <%= icon "loader", size: "sm", class: "animate-spin" %> + <%= tag.span t(".syncing") %> +
+ <% elsif up_item.sync_error.present? %> +
+ <%= render DS::Tooltip.new(text: up_item.sync_error, icon: "alert-circle", size: "sm", color: "destructive", as: :span) %> + <%= tag.span t(".error"), class: "text-destructive" %> +
+ <% else %> +

+ <% if up_item.last_synced_at %> + <%= t(".status_with_summary", timestamp: time_ago_in_words(up_item.last_synced_at), summary: up_item.sync_status_summary) %> + <% else %> + <%= t(".status_never") %> + <% end %> +

+ <% end %> +
+
+ + <% if Current.user&.admin? %> +
+ <%= render DS::Menu.new do |menu| %> + <% menu.with_item( + variant: "button", + text: t(".delete"), + icon: "trash-2", + href: up_item_path(up_item), + method: :delete, + confirm: CustomConfirm.for_resource_deletion(up_item.name, high_severity: true) + ) %> + <% end %> +
+ <% end %> +
+ <% end %> + + <% unless up_item.scheduled_for_deletion? %> +
+ <% if up_item.accounts.any? %> + <%= render "accounts/index/account_groups", accounts: up_item.accounts %> + <% end %> + + <% stats = if defined?(@up_sync_stats_map) && @up_sync_stats_map + @up_sync_stats_map[up_item.id] || {} + else + up_item.syncs.ordered.first&.sync_stats || {} + end %> + <%= render ProviderSyncSummary.new( + stats: stats, + provider_item: up_item, + institutions_count: up_item.connected_institutions.size + ) %> + + <% unlinked_count = up_item.unlinked_accounts_count %> + <% linked_count = up_item.linked_accounts_count %> + <% total_count = up_item.total_accounts_count %> + + <% if unlinked_count > 0 %> +
+

<%= t(".setup_needed") %>

+

<%= t(".setup_description", linked: linked_count, total: total_count) %>

+ <%= render DS::Link.new( + text: t(".setup_action"), + icon: "settings", + variant: "primary", + href: setup_accounts_up_item_path(up_item), + frame: :modal + ) %> +
+ <% elsif up_item.accounts.empty? && total_count == 0 %> +
+

<%= t(".no_accounts_title") %>

+

<%= t(".no_accounts_description") %>

+ <%= render DS::Link.new( + text: t(".setup_action"), + icon: "settings", + variant: "primary", + href: setup_accounts_up_item_path(up_item), + frame: :modal + ) %> +
+ <% end %> +
+ <% end %> + <% end %> +<% end %> diff --git a/app/views/up_items/select_accounts.html.erb b/app/views/up_items/select_accounts.html.erb new file mode 100644 index 000000000..f9a602a86 --- /dev/null +++ b/app/views/up_items/select_accounts.html.erb @@ -0,0 +1,50 @@ +<%= turbo_frame_tag "modal" do %> + <%= render DS::Dialog.new do |dialog| %> + <% dialog.with_header(title: t(".title")) %> + + <% dialog.with_body do %> +
+ <% if @api_error.present? %> + <%= render DS::Alert.new(message: @api_error, variant: :error) %> + <% elsif @up_accounts.empty? %> +

<%= t(".no_accounts_found") %>

+ <% else %> +

<%= t(".description") %>

+ + <%= form_with url: link_accounts_up_items_path, + method: :post, + data: { turbo_frame: "_top" }, + class: "space-y-4" do %> + <%= hidden_field_tag :up_item_id, @up_item.id %> + <%= hidden_field_tag :accountable_type, @accountable_type %> + <%= hidden_field_tag :return_to, @return_to %> + +
+ <% @up_accounts.each do |up_account| %> + + <% end %> +
+ +
+ <%= render DS::Link.new( + text: t(".cancel"), + href: @return_to || accounts_path, + variant: :secondary, + data: { turbo_frame: "_top", action: "DS--dialog#close" } + ) %> + <%= render DS::Button.new(text: t(".link_accounts"), type: "submit") %> +
+ <% end %> + <% end %> +
+ <% end %> + <% end %> +<% end %> diff --git a/app/views/up_items/select_existing_account.html.erb b/app/views/up_items/select_existing_account.html.erb new file mode 100644 index 000000000..913a59be9 --- /dev/null +++ b/app/views/up_items/select_existing_account.html.erb @@ -0,0 +1,50 @@ +<%= turbo_frame_tag "modal" do %> + <%= render DS::Dialog.new do |dialog| %> + <% dialog.with_header(title: t(".title", account_name: @account.name)) %> + + <% dialog.with_body do %> +
+ <% if @api_error.present? %> + <%= render DS::Alert.new(message: @api_error, variant: :error) %> + <% elsif @up_accounts.empty? %> +

<%= t(".no_accounts_found") %>

+ <% else %> +

<%= t(".description") %>

+ + <%= form_with url: link_existing_account_up_items_path, + method: :post, + data: { turbo_frame: "_top" }, + class: "space-y-4" do %> + <%= hidden_field_tag :up_item_id, @up_item.id %> + <%= hidden_field_tag :account_id, @account.id %> + <%= hidden_field_tag :return_to, @return_to %> + +
+ <% @up_accounts.each do |up_account| %> + + <% end %> +
+ +
+ <%= render DS::Link.new( + text: t(".cancel"), + href: @return_to || accounts_path, + variant: :secondary, + data: { turbo_frame: "_top", action: "DS--dialog#close" } + ) %> + <%= render DS::Button.new(text: t(".link_account"), type: "submit") %> +
+ <% end %> + <% end %> +
+ <% end %> + <% end %> +<% end %> diff --git a/app/views/up_items/setup_accounts.html.erb b/app/views/up_items/setup_accounts.html.erb new file mode 100644 index 000000000..09b916862 --- /dev/null +++ b/app/views/up_items/setup_accounts.html.erb @@ -0,0 +1,72 @@ +<% content_for :title, t(".title") %> + +<%= render DS::Dialog.new do |dialog| %> + <% dialog.with_header(title: t(".title")) do %> +
+ <%= icon "landmark", class: "text-primary" %> + <%= t(".subtitle") %> +
+ <% end %> + + <% dialog.with_body do %> + <%= form_with url: complete_account_setup_up_item_path(@up_item), + method: :post, + local: true, + data: { turbo_frame: "_top" }, + class: "space-y-6" do %> +
+ <% if @api_error.present? %> +
+ <%= icon "alert-circle", size: "lg", class: "text-destructive" %> +

<%= t(".fetch_failed") %>

+

<%= @api_error %>

+
+ <% elsif @up_accounts.empty? %> +
+ <%= icon "check-circle", size: "lg", class: "text-success" %> +

<%= t(".no_accounts_to_setup") %>

+

<%= t(".all_accounts_linked") %>

+
+ <% else %> +
+

<%= t(".choose_account_type") %>

+

<%= t(".choose_account_type_description") %>

+
+ + <% @up_accounts.each do |up_account| %> +
+
+

<%= up_account.name %>

+

+ <%= [up_account.currency, up_account.account_type].compact.join(" · ") %> +

+
+ + <%= label_tag "account_types[#{up_account.id}]", t(".account_type_label"), + class: "block text-sm font-medium text-primary mb-2" %> + <%= select_tag "account_types[#{up_account.id}]", + options_for_select(@account_type_options, @up_account_type_suggestions.fetch(up_account.id, "skip")), + class: "appearance-none bg-container border border-primary rounded-md px-3 py-2 text-sm leading-6 text-primary focus:border-primary focus:ring-1 focus:ring-primary focus:outline-none w-full" %> +
+ <% end %> + <% end %> +
+ +
+ <%= render DS::Button.new( + text: t(".create_accounts"), + variant: "primary", + icon: "plus", + type: "submit", + class: "flex-1", + disabled: @api_error.present? || @up_accounts.empty? + ) %> + <%= render DS::Link.new( + text: t(".cancel"), + variant: "secondary", + href: accounts_path + ) %> +
+ <% end %> + <% end %> +<% end %> diff --git a/config/initializers/up.rb b/config/initializers/up.rb new file mode 100644 index 000000000..8eb414d51 --- /dev/null +++ b/config/initializers/up.rb @@ -0,0 +1,9 @@ +# Up Bank integration runtime configuration +Rails.application.configure do + # Debug logging for raw Up API responses. + # When enabled, logs the full raw payload returned by the Up API. + # DEVELOPMENT-ONLY: the raw dump contains PII (merchant names, amounts, account IDs) + # and is gated to local environments so it never fires in managed/production. + # Default: false (only log summary info) + config.x.up.debug_raw = ENV["UP_DEBUG_RAW"].to_s.strip.downcase.in?(%w[1 true yes]) +end diff --git a/config/locales/views/settings/en.yml b/config/locales/views/settings/en.yml index 0280986d2..c9913ee8e 100644 --- a/config/locales/views/settings/en.yml +++ b/config/locales/views/settings/en.yml @@ -478,6 +478,10 @@ en: step_1_html: "Go to %{link} and create a personal app." step_2: "Copy your app token and user token." step_3: "Paste the tokens below, save, then link your synced accounts." + up_panel: + step_1_html: "Go to %{link} and generate a personal access token." + step_2: "Copy your personal access token." + step_3: "Paste the token below, save, then link your synced accounts." simplefin_panel: step_1_html: "Go to %{link} for a one-time setup token." step_2: "Paste the token below and connect." diff --git a/config/locales/views/up_items/en.yml b/config/locales/views/up_items/en.yml new file mode 100644 index 000000000..bdc8e07f1 --- /dev/null +++ b/config/locales/views/up_items/en.yml @@ -0,0 +1,116 @@ +--- +en: + providers: + up: + name: Up + description: Connect Australian Up bank accounts via a personal access token + family: + up: + create_up_item: + default_name: Up Connection + up_account: + fallback: Up account + up_item: + errors: + account_processing_failed: Could not sync Up account + account_sync_schedule_failed: Could not schedule Up account sync + transactions_failed: Could not fetch Up transactions + sync_failed: Could not sync Up connection + sync_status: + no_accounts: No accounts found + all_synced: + one: 1 account synced + other: "%{count} accounts synced" + partial: "%{linked} synced, %{unlinked} need setup" + institution_summary: + none: No institutions connected + one: 1 institution + count: + one: 1 institution + other: "%{count} institutions" + up_items: + provider_panel: + default_connection_name: Up Connection + add_connection: Add Up connection + update_connection: Update connection + connection_name_label: Connection name + connection_name_placeholder: Main Up + access_token_label: Personal Access Token + access_token_placeholder: Paste your Up personal access token + keep_access_token_placeholder: Leave blank to keep the existing token + setup_accounts: Setup accounts + syncing: Syncing... + sync: Sync + disconnect: Disconnect + disconnect_confirm: "Are you sure you want to disconnect %{name}?" + create: + success: Up connection saved. + update: + success: Up connection updated. + destroy: + success: Scheduled Up connection for deletion. + unlink_failed: Could not disconnect Up connection + select_accounts: + title: Link Up accounts + description: Choose the Up accounts to add. + no_accounts_found: No unlinked Up accounts were found. + no_credentials_configured: Configure Up in provider settings first. + cancel: Cancel + link_accounts: Link accounts + link_accounts: + success: + one: Linked 1 Up account. + other: "Linked %{count} Up accounts." + no_accounts_selected: Select at least one account. + no_credentials_configured: Configure Up in provider settings first. + unsupported_account_type: Up does not support that account type. + link_failed: No accounts were linked. + select_existing_account: + title: "Link Up account to %{account_name}" + description: Choose an unlinked Up account to connect to this account. + no_accounts_found: No unlinked Up accounts were found. + no_credentials_configured: Configure Up in provider settings first. + account_already_linked: This account is already linked to a provider. + cancel: Cancel + link_account: Link account + link_existing_account: + success: "Linked Up account to %{account_name}." + account_already_linked: This account is already linked to a provider. + up_account_already_linked: This Up account is already linked. + no_account_selected: Select an Up account to link. + setup_accounts: + title: Link Up Accounts + subtitle: Choose how each Up account should appear in Sure. + no_credentials: Configure Up credentials first. + api_error: Could not fetch Up accounts. + fetch_failed: Could not fetch accounts + no_accounts_to_setup: No accounts to set up + all_accounts_linked: All Up accounts are already linked. + choose_account_type: Choose an account type + choose_account_type_description: Skip accounts you do not want to track. + account_type_label: Account type + account_types: + skip: Skip + depository: Cash + loan: Loan + create_accounts: Create accounts + cancel: Cancel + complete_account_setup: + success: + one: Created 1 Up account. + other: "Created %{count} Up accounts." + all_skipped: No Up accounts were created. + no_accounts: No Up accounts were selected. + creation_failed: Could not create Up accounts. + up_item: + deletion_in_progress: Deletion in progress + syncing: Syncing + error: Error + status_with_summary: "Synced %{timestamp} ago · %{summary}" + status_never: Never synced + delete: Delete + setup_needed: Account setup needed + setup_description: "%{linked} of %{total} accounts linked." + setup_action: Setup accounts + no_accounts_title: No accounts imported yet + no_accounts_description: Fetch Up accounts and choose which ones to link. diff --git a/config/routes.rb b/config/routes.rb index 69a096c44..af0e04bc4 100644 --- a/config/routes.rb +++ b/config/routes.rb @@ -649,6 +649,22 @@ Rails.application.routes.draw do end end + resources :up_items, only: %i[index new create show edit update destroy] do + collection do + get :preload_accounts + get :select_accounts + post :link_accounts + get :select_existing_account + post :link_existing_account + end + + member do + post :sync + get :setup_accounts + post :complete_account_setup + end + end + resources :sophtron_items, only: %i[index new create show edit update destroy] do collection do get :preload_accounts diff --git a/db/migrate/20260617120000_create_up_items_and_accounts.rb b/db/migrate/20260617120000_create_up_items_and_accounts.rb new file mode 100644 index 000000000..89288641a --- /dev/null +++ b/db/migrate/20260617120000_create_up_items_and_accounts.rb @@ -0,0 +1,49 @@ +class CreateUpItemsAndAccounts < ActiveRecord::Migration[7.2] + def change + create_table :up_items, id: :uuid do |t| + t.references :family, null: false, foreign_key: true, type: :uuid + t.string :name + t.string :institution_id + t.string :institution_name + t.string :institution_domain + t.string :institution_url + t.string :institution_color + t.string :status, default: "good", null: false + t.boolean :scheduled_for_deletion, default: false, null: false + t.boolean :pending_account_setup, default: false, null: false + t.date :sync_start_date + t.jsonb :raw_payload + t.jsonb :raw_institution_payload + t.text :access_token + t.timestamps + end + + add_index :up_items, :status + + create_table :up_accounts, id: :uuid do |t| + t.references :up_item, null: false, foreign_key: true, type: :uuid + t.string :name, null: false + t.string :account_id + t.string :currency, null: false + t.decimal :current_balance, precision: 19, scale: 4 + t.string :account_status + t.string :account_type + t.string :ownership_type + t.string :provider + t.boolean :ignored, default: false, null: false + t.jsonb :institution_metadata + t.jsonb :raw_payload + t.jsonb :raw_transactions_payload + t.date :sync_start_date + + t.timestamps + end + + add_index :up_accounts, :account_id + add_index :up_accounts, + [ :up_item_id, :account_id ], + unique: true, + where: "account_id IS NOT NULL", + name: "index_up_accounts_on_item_and_account_id" + end +end diff --git a/db/schema.rb b/db/schema.rb index 4c0be7fcf..240ec6790 100644 --- a/db/schema.rb +++ b/db/schema.rb @@ -10,7 +10,7 @@ # # It's strongly recommended that you check this file into your version control system. -ActiveRecord::Schema[7.2].define(version: 2026_06_08_160000) do +ActiveRecord::Schema[7.2].define(version: 2026_06_17_120000) do # These are extensions that must be enabled in order to support this database enable_extension "pgcrypto" enable_extension "plpgsql" @@ -1967,6 +1967,49 @@ ActiveRecord::Schema[7.2].define(version: 2026_06_08_160000) do t.index ["status"], name: "index_transfers_on_status" end + create_table "up_accounts", id: :uuid, default: -> { "gen_random_uuid()" }, force: :cascade do |t| + t.uuid "up_item_id", null: false + t.string "name", null: false + t.string "account_id" + t.string "currency", null: false + t.decimal "current_balance", precision: 19, scale: 4 + t.string "account_status" + t.string "account_type" + t.string "ownership_type" + t.string "provider" + t.boolean "ignored", default: false, null: false + t.jsonb "institution_metadata" + t.jsonb "raw_payload" + t.jsonb "raw_transactions_payload" + t.date "sync_start_date" + t.datetime "created_at", null: false + t.datetime "updated_at", null: false + t.index ["account_id"], name: "index_up_accounts_on_account_id" + t.index ["up_item_id", "account_id"], name: "index_up_accounts_on_item_and_account_id", unique: true, where: "(account_id IS NOT NULL)" + t.index ["up_item_id"], name: "index_up_accounts_on_up_item_id" + end + + create_table "up_items", id: :uuid, default: -> { "gen_random_uuid()" }, force: :cascade do |t| + t.uuid "family_id", null: false + t.string "name" + t.string "institution_id" + t.string "institution_name" + t.string "institution_domain" + t.string "institution_url" + t.string "institution_color" + t.string "status", default: "good", null: false + t.boolean "scheduled_for_deletion", default: false, null: false + t.boolean "pending_account_setup", default: false, null: false + t.date "sync_start_date" + t.jsonb "raw_payload" + t.jsonb "raw_institution_payload" + t.text "access_token" + t.datetime "created_at", null: false + t.datetime "updated_at", null: false + t.index ["family_id"], name: "index_up_items_on_family_id" + t.index ["status"], name: "index_up_items_on_status" + end + create_table "users", id: :uuid, default: -> { "gen_random_uuid()" }, force: :cascade do |t| t.uuid "family_id", null: false t.string "first_name" @@ -2162,6 +2205,8 @@ ActiveRecord::Schema[7.2].define(version: 2026_06_08_160000) do add_foreign_key "transactions", "merchants" add_foreign_key "transfers", "transactions", column: "inflow_transaction_id", on_delete: :cascade add_foreign_key "transfers", "transactions", column: "outflow_transaction_id", on_delete: :cascade + add_foreign_key "up_accounts", "up_items" + add_foreign_key "up_items", "families" add_foreign_key "users", "accounts", column: "default_account_id", on_delete: :nullify add_foreign_key "users", "chats", column: "last_viewed_chat_id" add_foreign_key "users", "families" diff --git a/test/models/provider/up_test.rb b/test/models/provider/up_test.rb new file mode 100644 index 000000000..332ffc051 --- /dev/null +++ b/test/models/provider/up_test.rb @@ -0,0 +1,160 @@ +require "test_helper" + +class Provider::UpTest < ActiveSupport::TestCase + FakeResponse = Struct.new(:code, :body, :message, keyword_init: true) + + test "fetches paginated account transactions following JSON:API links.next with bearer auth" do + next_url = "https://api.up.com.au/api/v1/accounts/acc_123/transactions?page%5Bafter%5D=cursor2" + responses = [ + FakeResponse.new( + code: 200, + message: "OK", + body: { + data: [ { type: "transactions", id: "tx_1", attributes: { status: "SETTLED" }, relationships: { account: { data: { id: "acc_123" } } } } ], + links: { prev: nil, next: next_url } + }.to_json + ), + FakeResponse.new( + code: 200, + message: "OK", + body: { + data: [ { type: "transactions", id: "tx_2", attributes: { status: "HELD" }, relationships: { account: { data: { id: "acc_123" } } } } ], + links: { prev: nil, next: nil } + }.to_json + ) + ] + requests = [] + + Provider::Up.stub(:get, ->(url, headers:, query: nil) { + requests << { url: url, headers: headers, query: query } + responses.shift + }) do + client = Provider::Up.new("up-access-token") + + transactions = client.get_account_transactions( + account_id: "acc_123", + since: Date.new(2026, 1, 1) + ) + + assert_equal [ "tx_1", "tx_2" ], transactions.map { |tx| tx[:id] } + assert_equal [ "acc_123", "acc_123" ], transactions.map { |tx| tx[:account_id] } + assert_equal "SETTLED", transactions.first[:status] + end + + assert_equal 2, requests.size + assert_match "/accounts/acc_123/transactions", requests.first[:url] + assert_equal "Bearer up-access-token", requests.first[:headers]["Authorization"] + assert_equal "2026-01-01T00:00:00Z", requests.first[:query]["filter[since]"] + assert_equal 100, requests.first[:query]["page[size]"] + # Pagination follows the absolute next URL with no extra query params. + assert_equal next_url, requests.second[:url] + assert_nil requests.second[:query] + end + + test "stops paginating when the API repeats the same next cursor" do + repeating_url = "https://api.up.com.au/api/v1/accounts/acc_123/transactions?page%5Bafter%5D=loop" + page = FakeResponse.new( + code: 200, + message: "OK", + body: { + data: [ { type: "transactions", id: "tx_1", attributes: { status: "SETTLED" }, relationships: { account: { data: { id: "acc_123" } } } } ], + links: { prev: nil, next: repeating_url } + }.to_json + ) + request_count = 0 + + Provider::Up.stub(:get, ->(url, headers:, query: nil) { + request_count += 1 + raise "infinite pagination loop" if request_count > 5 + + page + }) do + client = Provider::Up.new("up-access-token") + transactions = client.get_account_transactions(account_id: "acc_123") + + # First request + one follow of the repeated cursor, then the guard stops. + assert_equal 2, request_count + assert_equal [ "tx_1", "tx_1" ], transactions.map { |tx| tx[:id] } + end + end + + test "refuses to follow a pagination link pointing at a non-Up host" do + evil_url = "https://evil.example.com/api/v1/accounts/acc_123/transactions?page%5Bafter%5D=x" + first_page = FakeResponse.new( + code: 200, + message: "OK", + body: { + data: [ { type: "transactions", id: "tx_1", attributes: { status: "SETTLED" }, relationships: { account: { data: { id: "acc_123" } } } } ], + links: { prev: nil, next: evil_url } + }.to_json + ) + requested_urls = [] + + Provider::Up.stub(:get, ->(url, headers:, query: nil) { + requested_urls << url + first_page + }) do + client = Provider::Up.new("up-access-token") + + error = assert_raises(Provider::Up::UpError) do + client.get_account_transactions(account_id: "acc_123") + end + assert_equal :invalid_url, error.error_type + end + + # The bearer token must never be sent to the foreign host. + assert_not_includes requested_urls, evil_url + end + + test "flattens JSON:API account resources" do + response = FakeResponse.new( + code: 200, + message: "OK", + body: { + data: [ { + type: "accounts", + id: "acc_123", + attributes: { + displayName: "Spending", + accountType: "TRANSACTIONAL", + ownershipType: "INDIVIDUAL", + balance: { currencyCode: "AUD", value: "123.45", valueInBaseUnits: 12345 } + } + } ], + links: { prev: nil, next: nil } + }.to_json + ) + + Provider::Up.stub(:get, ->(_url, headers:, query: nil) { response }) do + accounts = Provider::Up.new("up-access-token").get_accounts + + assert_equal 1, accounts.size + account = accounts.first + assert_equal "acc_123", account[:id] + assert_equal "Spending", account[:displayName] + assert_equal "TRANSACTIONAL", account[:accountType] + assert_equal "AUD", account.dig(:balance, :currencyCode) + assert_equal "123.45", account.dig(:balance, :value) + end + end + + test "raises typed errors for unauthorized responses" do + response = FakeResponse.new(code: 401, message: "Unauthorized", body: "{}") + + Provider::Up.stub(:get, ->(_url, headers:, query: nil) { response }) do + error = assert_raises Provider::Up::UpError do + Provider::Up.new("invalid-token").get_accounts + end + + assert_equal :unauthorized, error.error_type + end + end + + test "raises configuration error when token blank" do + error = assert_raises Provider::Up::UpError do + Provider::Up.new("") + end + + assert_equal :configuration_error, error.error_type + end +end diff --git a/test/models/up_account_test.rb b/test/models/up_account_test.rb new file mode 100644 index 000000000..d751b1563 --- /dev/null +++ b/test/models/up_account_test.rb @@ -0,0 +1,24 @@ +require "test_helper" + +class UpAccountTest < ActiveSupport::TestCase + setup do + @family = families(:empty) + @up_item = UpItem.create!(family: @family, name: "Test Up", access_token: "up-access-token") + end + + test "needs_setup excludes linked and ignored accounts" do + unlinked = UpAccount.create!(up_item: @up_item, name: "Unlinked", account_id: "acc_unlinked", currency: "AUD") + ignored = UpAccount.create!(up_item: @up_item, name: "Skipped", account_id: "acc_ignored", currency: "AUD", ignored: true) + + linked = UpAccount.create!(up_item: @up_item, name: "Linked", account_id: "acc_linked", currency: "AUD") + account = Account.create!(family: @family, name: "Linked", accountable: Depository.new(subtype: "checking"), balance: 0, currency: "AUD") + AccountProvider.create!(account: account, provider: linked) + + needs_setup = @up_item.up_accounts.needs_setup + + assert_includes needs_setup, unlinked + assert_not_includes needs_setup, ignored + assert_not_includes needs_setup, linked + assert_equal 1, @up_item.unlinked_accounts_count + end +end diff --git a/test/models/up_entry/processor_test.rb b/test/models/up_entry/processor_test.rb new file mode 100644 index 000000000..45ba6228c --- /dev/null +++ b/test/models/up_entry/processor_test.rb @@ -0,0 +1,116 @@ +require "test_helper" + +class UpEntry::ProcessorTest < ActiveSupport::TestCase + setup do + @family = families(:empty) + @up_item = UpItem.create!( + family: @family, + name: "Test Up", + access_token: "up-access-token" + ) + @up_account = UpAccount.create!( + up_item: @up_item, + name: "Up Spending", + account_id: "acc_123", + currency: "AUD" + ) + @account = Account.create!( + family: @family, + name: "Spending", + accountable: Depository.new(subtype: "checking"), + balance: 1000, + currency: "AUD" + ) + + AccountProvider.create!(account: @account, provider: @up_account) + end + + test "imports settled Up transaction with sign conversion and provider metadata" do + transaction_data = { + id: "tx_123", + account_id: "acc_123", + status: "SETTLED", + description: "Coffee Shop", + message: "Morning coffee", + rawText: "COFFEE SHOP SYDNEY", + amount: { currencyCode: "AUD", value: "-12.50", valueInBaseUnits: -1250 }, + foreignAmount: nil, + settledAt: "2026-01-15T08:30:00+11:00", + createdAt: "2026-01-15T08:00:00+11:00", + category_id: "restaurants-and-cafes" + } + + entry = UpEntry::Processor.new(transaction_data, up_account: @up_account).process + + assert_equal "up_tx_123", entry.external_id + assert_equal "up", entry.source + assert_equal BigDecimal("12.5"), entry.amount + assert_equal "AUD", entry.currency + assert_equal Date.new(2026, 1, 15), entry.date + assert_equal "Coffee Shop", entry.name + assert_equal "Morning coffee", entry.notes + + transaction = entry.entryable + assert_equal false, transaction.pending? + assert_equal false, transaction.extra.dig("up", "pending") + assert_equal "SETTLED", transaction.extra.dig("up", "status") + assert_equal "restaurants-and-cafes", transaction.extra.dig("up", "category_id") + assert_equal "Coffee Shop", transaction.merchant.name + end + + test "marks HELD transactions as pending and falls back to createdAt for the date" do + entry = UpEntry::Processor.new( + { + id: "tx_held_1", + account_id: "acc_123", + status: "HELD", + description: "Pending card auth", + amount: { currencyCode: "AUD", value: "-8.00", valueInBaseUnits: -800 }, + settledAt: nil, + createdAt: "2026-01-20T12:00:00+11:00" + }, + up_account: @up_account + ).process + + assert entry.entryable.pending? + assert_equal true, entry.entryable.extra.dig("up", "pending") + assert_equal Date.new(2026, 1, 20), entry.date + end + + test "stores foreign amount FX metadata" do + entry = UpEntry::Processor.new( + { + id: "tx_fx_1", + account_id: "acc_123", + status: "SETTLED", + description: "US Merchant", + amount: { currencyCode: "AUD", value: "-15.00", valueInBaseUnits: -1500 }, + foreignAmount: { currencyCode: "USD", value: "-10.00", valueInBaseUnits: -1000 }, + settledAt: "2026-01-18T00:00:00+11:00", + createdAt: "2026-01-18T00:00:00+11:00" + }, + up_account: @up_account + ).process + + transaction = entry.entryable + assert_equal "USD", transaction.extra.dig("up", "fx_from") + assert_equal "-10.00", transaction.extra.dig("up", "fx_amount") + end + + test "income (positive Up amount) is stored as negative in Sure" do + entry = UpEntry::Processor.new( + { + id: "tx_income_1", + account_id: "acc_123", + status: "SETTLED", + description: "Salary", + amount: { currencyCode: "AUD", value: "2500.00", valueInBaseUnits: 250000 }, + settledAt: "2026-01-15T00:00:00+11:00", + createdAt: "2026-01-15T00:00:00+11:00" + }, + up_account: @up_account + ).process + + assert_equal BigDecimal("-2500.0"), entry.amount + end +end diff --git a/test/models/up_item/importer_test.rb b/test/models/up_item/importer_test.rb new file mode 100644 index 000000000..73ff16925 --- /dev/null +++ b/test/models/up_item/importer_test.rb @@ -0,0 +1,156 @@ +require "test_helper" + +class UpItem::ImporterTest < ActiveSupport::TestCase + class FakeUpProvider + attr_reader :transaction_calls + + def initialize(accounts: nil, transactions: nil) + @transaction_calls = [] + @accounts = accounts + @transactions = transactions + end + + def get_accounts + @accounts || [ + { + id: "acc_123", + displayName: "Up Spending", + accountType: "TRANSACTIONAL", + ownershipType: "INDIVIDUAL", + balance: { currencyCode: "AUD", value: "123.45", valueInBaseUnits: 12345 } + } + ] + end + + def get_account_transactions(account_id:, since: nil) + @transaction_calls << { account_id: account_id, since: since } + @transactions || [ settled_transaction ] + end + + private + + def settled_transaction + { + id: "tx_1", + account_id: "acc_123", + status: "SETTLED", + description: "Posted transaction", + amount: { currencyCode: "AUD", value: "-20.00", valueInBaseUnits: -2000 }, + settledAt: "2026-01-19T00:00:00+11:00", + createdAt: "2026-01-19T00:00:00+11:00" + } + end + end + + setup do + @family = families(:empty) + @up_item = UpItem.create!( + family: @family, + name: "Test Up", + access_token: "up-access-token" + ) + @up_account = UpAccount.create!( + up_item: @up_item, + name: "Old name", + account_id: "acc_123", + currency: "AUD" + ) + @account = Account.create!( + family: @family, + name: "Spending", + accountable: Depository.new(subtype: "checking"), + balance: 0, + currency: "AUD" + ) + AccountProvider.create!(account: @account, provider: @up_account) + end + + test "imports account snapshot and stores transactions" do + provider = FakeUpProvider.new + + result = UpItem::Importer.new(@up_item, up_provider: provider).import + + assert result[:success] + assert_equal 1, result[:accounts_updated] + assert_equal 1, result[:transactions_imported] + + @up_account.reload + assert_equal "Up Spending", @up_account.name + assert_equal BigDecimal("123.45"), @up_account.current_balance + assert_equal "TRANSACTIONAL", @up_account.account_type + assert_equal "acc_123", provider.transaction_calls.first[:account_id] + + assert_equal [ "tx_1" ], @up_account.raw_transactions_payload.map { |tx| tx["id"] } + end + + test "held transaction settling under the same id clears pending without duplicates" do + import_with(transactions: [ held_transaction(id: "tx_h1", amount: "-8.00") ]) + process_transactions + + assert_equal 1, pending_entries.count + + import_with(transactions: [ settled(id: "tx_h1", amount: "-8.00") ]) + process_transactions + + assert_empty pending_entries + assert_equal 1, @account.entries.where(source: "up").count + entry = @account.entries.find_by(external_id: "up_tx_h1", source: "up") + assert_equal false, entry.entryable.pending? + end + + test "removes held transactions that disappear from the latest fetch" do + import_with(transactions: [ held_transaction(id: "tx_h2", amount: "-8.00") ]) + process_transactions + + assert_equal 1, pending_entries.count + + import_with(transactions: []) + process_transactions + + @up_account.reload + assert_empty pending_entries + assert_empty @up_account.raw_transactions_payload.select { |tx| UpEntry::Processor.pending?(tx) } + end + + private + + def import_with(transactions:) + provider = FakeUpProvider.new(transactions: transactions) + UpItem::Importer.new(@up_item, up_provider: provider).import + end + + def process_transactions + UpAccount::Transactions::Processor.new(@up_account.reload).process + end + + def pending_entries + @account.entries + .joins("INNER JOIN transactions ON transactions.id = entries.entryable_id AND entries.entryable_type = 'Transaction'") + .where(source: "up") + .where("(transactions.extra -> 'up' ->> 'pending')::boolean = true") + end + + def held_transaction(id:, amount:, date: "2026-01-15T00:00:00+11:00") + { + id: id, + account_id: "acc_123", + status: "HELD", + description: "Pending card auth", + amount: { currencyCode: "AUD", value: amount, valueInBaseUnits: (amount.to_f * 100).to_i }, + settledAt: nil, + createdAt: date + } + end + + def settled(id:, amount:, date: "2026-01-16T00:00:00+11:00") + { + id: id, + account_id: "acc_123", + status: "SETTLED", + description: "Posted card auth", + amount: { currencyCode: "AUD", value: amount, valueInBaseUnits: (amount.to_f * 100).to_i }, + settledAt: date, + createdAt: date + } + end +end