QT/sure
1
0
Fork 0
mirror of https://github.com/we-promise/sure.git synced 2026-04-25 06:54:07 +00:00
Code Issues Packages Projects Releases Wiki Activity

Simplefin enhancements v2 (#267)

Browse Source 
* SimpleFin: metadata + merge fixes; holdings (incl. crypto) + Day Change; Sync Summary; ops rakes; lint

# Conflicts:
#	db/schema.rb

# Conflicts:
#	app/controllers/simplefin_items_controller.rb

* fix testing

* fix linting

* xfix linting x2

* Review PR #267 on we-promise/sure (SimpleFin enhancements v2). Address all 15 actionable CodeRabbit comments: Add UUID validations in rakes (e.g., simplefin_unlink), swap Ruby pattern matching/loops for efficient DB queries (e.g., where LOWER(name) LIKE ?), generate docstrings for low-coverage areas (31%), consolidate routes for simplefin_items, move view logic to helpers (e.g., format_transaction_extra), strengthen tests with exact assertions/fixtures for dedup/relink failures. Also, check for overlaps with merged #262 (merchants fix): Ensure merchant creation in simplefin_entry/processor.rb aligns with new payee-based flow and MD5 IDs; add tests for edge cases like empty payees or over-merging pendings. Prioritize security (PII redaction in logs, no hardcoded secrets).

* SimpleFin: address CodeRabbit comments (batch 1)

- Consolidate simplefin_items routes under a single resources block; keep URLs stable
- Replace inline JS with Stimulus auto-relink controller; auto-load relink modal via global modal frame
- Improve a11y in relink modal by wrapping rows in labels
- Harden unlink rake: default dry_run=true, UUID validation, redact PII in outputs, clearer errors
- Backfill rake: default dry_run=true, UUID validation; groundwork for per-SFA counters
- Fix-was-merged rake: default dry_run=true, UUID validation; clearer outputs
- Idempotent transfer auto-match (find_or_create_by! + RecordNotUnique rescue)
- Extract SimpleFin error tooltip assembly into helper and use it in view

RuboCop: maintain 2-space indentation, spaces inside array brackets, spaces after commas, and no redundant returns

* Linter noise

* removed filed commited by mistake.

* manual relink flow and tighten composite matching

* enforce manual relink UI; fix adapter keywords; guarantee extra.simplefin hash

* refactor(simplefin): extract relink service; enforce manual relink UI; tighten composite match; migration 7.2

* add provider date parser; refactor rake; move view queries; partial resilience

* run balances-only import in background job. make update flow enqueue balances-only job

* persists across all update redirects and initialize
used_manual_ids to prevent NameError in relink candidate computation.

* SimpleFin: metadata + merge fixes; holdings (incl. crypto) + Day Change; Sync Summary; ops rakes; lint

* Fixed failed test after rebase.

* scan_ruby fix

* Calming the rabbit:
Fix AccountProvider linking when accounts change
Drop the legacy unique index instead of duplicating it
Fix dynamic constant assignment
Use fixtures consistently; avoid rescue for control flow.
Replace bare rescue with explicit exception class.
Move business logic out of the view.
Critical: Transaction boundary excludes recompute phase, risking data loss.
Inconsistency between documentation and implementation for zero-error case.
Refactor to use the compute_unlinked_count helper for consistency.
Fix cleanup task default: it deletes by default.
Move sync stats computation to controller to avoid N+1 queries.
Consolidate duplicate sync query.
Clarify the intent of setting flash notice on the error path.
Fix Date/Time comparison in should_be_inactive?.
Move stats retrieval logic to controller.
Remove duplicate Sync summary section.
Remove the unnecessary sleep statement; use Capybara's built-in waiting.
Add label wrappers for accessibility and consistency.

* FIX SimpleFIN new account modal

Now new account properly loads as a Modal, instead of new page.
Fixes also form showing dashboard instead of settings page.

* Remove SimpleFin legacy UI components, migrate schema, and refine linking behavior.

# Conflicts:
#	app/helpers/settings_helper.rb

* Extract SimpleFin-related logic to `prepare_show_context` helper and refactor for consistency. Adjust conditional checks and ensure controller variables are properly initialized.

* Remove unused SimpleFin maps from prepare_show_context; select IDs to avoid N+1
Replace Tailwind bg-green-500 with semantic bg-success in _simplefin_panel/_provider_form
Add f.label :setup_token in simplefin_items/new for a11y
Remove duplicate require in AccountsControllerSimplefinCtaTest

* Remove unnecessary blank lines

* Reduce unnecessary changes

This reduces the diff against main

* Simplefin Account Setup: Display in modal

This fixes an issue with the `X` dismiss button in the top right corner

* Removed unnecessary comment.

* removed unnecessary function.

* fixed broken links

* Removed unnecessary file

* changed to database query

* set to use UTC and gaurd against null

* set dry_run=true

* Fixed comment

* Changed to use a database-level query

* matched test name to test behavior.

* Eliminate code duplication and Time.zone dependency

* make final summary surface failures

* lint fix

* Revised timezone comment. better handle missing selectors.

* sanitized LIKE wildcards

* Fixed SimpleFin import to avoid “Currency can’t be blank” validation failures when providers return an empty currency string.

* Added helper methods for admin and self-hosted checks

* Specify exception types in rescue clauses.

* Refined logic to determine transaction dates for credit accounts.

* Refined stats calculation for `total_accounts` to track the maximum unique accounts per run instead of accumulating totals.

* Moved `unlink_all!` logic to `SimplefinItem::Unlinking` concern and deprecated `SimplefinItem::Unlinker`. Updated related references.

* Refined legacy unlinking logic, improved `current_holdings` formatting, and added ENV-based overrides for self-hosted checks.

* Enhanced `unlink_all!` with explicit error handling, improved transaction safety, and refined ENV-based self-hosted checks. Adjusted exception types and cleaned up private method handling.

* Improved currency assignment logic by adding fallback to `current_account` and `family` currencies.

* Enhanced error tracking during SimpleFin account imports by adding categorized error buckets, limiting stored errors to the last 5, and improving `stats` calculations.

* typo fix

* Didn't realize rabbit was still mad...
Refactored SimpleFin error handling and CTA logic: centralized duplicate detection and relink visibility into controller, improved task counters, adjusted redirect notices, and fixed form indexing.

* Dang rabbit never stops... Centralized SimpleFin maps logic into `MapsHelper` concern and integrated it into relevant controllers and rake tasks. Optimized queries, reduced redundancy, and improved unlinked counts and manual account checks with batch processing. Adjusted task arguments for clarity.

* Persistent rabbit. Optimized SimpleFin maps logic by implementing batch queries for manual account and unlinked count checks, reducing N+1 issues. Improved clarity of rake task argument descriptions and error messages for better usability.

* Lost a commit somehow, resolved here. Refactored transaction extra details logic by introducing `build_transaction_extra_details` helper to improve clarity, reusability, and reduce view complexity. Enhanced rake tasks with strict dry-run validation and better error handling. Updated schema to allow nullable `merchant_id` and added conditional unique indexes for recurring transactions.

* Refactored sensitive data redaction in `simplefin_unlink` task for recursive handling, optimized SQL sanitization in `simplefin_holdings_backfill`, improved error handling in `transactions_helper`, and streamlined day change calculation logic in `Holding` model.

* Lint fix

* Removed per PR comments.

* Also removing per PR comment.

* git commit -m "SimpleFIN polish: preserve #manual-accounts wrapper, unify \"manual\" scope, and correct unlinked counts
- Preserve #manual-accounts wrapper: switch non-empty updates to turbo_stream.update and background broadcast_update_to; keep empty-path replace to render <div id=\"manual-accounts\"></div>
- Unify definition of manual accounts via Account.visible_manual (visible + legacy-nil + no AccountProvider); reuse in controllers, jobs, and helper
- Correct setup/unlinked counts: SimplefinItem::Syncer#finalize_setup_counts and maps now consider AccountProvider links (legacy account AND provider must be absent)
Deleted:
- app/models/simplefin_item/relink_service.rb
- app/controllers/concerns/simplefin_items/relink_helpers.rb
- app/javascript/controllers/auto_relink_controller.js
- app/views/simplefin_items/_relink_modal.html.erb
- app/views/simplefin_items/manual_relink.html.erb
- app/views/simplefin_items/relink.html.erb
- test/services/simplefin_item/relink_service_test.rb
Refs: PR #318 unified link/unlink; PR #267 SimpleFIN; follow-up to fix wrapper ID loss and counting drift."

* Extend unlinked account check to include "Investment" type

* set SimpleFIN item for `balances`, remove redundant unpacking, and improve holdings task error

* SimpleFIN: add `errors` action + modal; do not reintroduce legacy relink actions; removed dead helper

* FIX simpleFIN linking

* Add delay back, tests benefit from it

* Put cache back in

* Remove empty `rake` task

* Small spelling fixes.

---------

Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: sokie <sokysrm@gmail.com>
Co-authored-by: Dylan Corrales <deathcamel58@gmail.com>
This commit is contained in:
LPW
2025-11-17 15:51:37 -05:00
committed by GitHub
parent 0d6ca8f25c
commit 61eb611529
67 changed files with 2919 additions and 409 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
lib/simplefin/date_utils.rb Normal file
View File

@@ -0,0 +1,28 @@
# frozen_string_literal: true
module Simplefin
module DateUtils
module_function
# Parses provider-supplied dates that may be String (ISO), Numeric (epoch seconds),
# Time/DateTime, or Date. Returns a Date or nil when unparseable.
def parse_provider_date(val)
return nil if val.nil?
case val
when Date
val
when Time, DateTime
val.to_date
when Integer, Float
Time.at(val).utc.to_date
when String
Date.parse(val)
else
nil
end
rescue ArgumentError, TypeError
nil
end
end
end

83
lib/tasks/holdings_tools.rake Normal file
View File

@@ -0,0 +1,83 @@
# frozen_string_literal: true
# Utilities for demonstrating holdings UI features (e.g., Day Change)
#
# Seed a prior snapshot for an existing holding to visualize Day Change immediately.
# Example:
# # Preview (no write):
# # bin/rails 'sure:holdings:seed_prev_snapshot[holding_id=HOLDING_UUID,change_pct=2,days_ago=1,dry_run=true]'
# # Apply (writes):
# # bin/rails 'sure:holdings:seed_prev_snapshot[holding_id=HOLDING_UUID,change_pct=2,days_ago=1,dry_run=false]'
#
# Remove a previously seeded snapshot by id:
# # bin/rails 'sure:holdings:remove_snapshot[id=HOLDING_UUID]'
namespace :sure do
namespace :holdings do
desc "Seed a previous snapshot for Day Change demo. Args: holding_id, change_pct=2, days_ago=1, dry_run=true"
task :seed_prev_snapshot, [ :holding_id, :change_pct, :days_ago, :dry_run ] => :environment do |_, args|
kv = {}
[ args[:holding_id], args[:change_pct], args[:days_ago], args[:dry_run] ].each do |raw|
next unless raw.is_a?(String) && raw.include?("=")
k, v = raw.split("=", 2)
kv[k.to_s] = v
end
holding_id = (kv["holding_id"] || args[:holding_id]).presence
change_pct = ((kv["change_pct"] || args[:change_pct] || 2).to_f) / 100.0
days_ago = (kv["days_ago"] || args[:days_ago] || 1).to_i
raw_dry = kv.key?("dry_run") ? kv["dry_run"] : args[:dry_run]
dry_raw = raw_dry.to_s.downcase
# Default to dry_run=true unless explicitly disabled, and validate input strictly
if raw_dry.nil? || dry_raw.blank?
dry_run = true
elsif %w[1 true yes y].include?(dry_raw)
dry_run = true
elsif %w[0 false no n].include?(dry_raw)
dry_run = false
else
puts({ ok: false, error: "invalid_argument", message: "dry_run must be one of: true/yes/1 or false/no/0" }.to_json)
exit 1
end
unless holding_id
puts({ ok: false, error: "usage", message: "Provide holding_id" }.to_json)
exit 1
end
h = Holding.find(holding_id)
prev = h.dup
prev.date = h.date - days_ago
# Apply percentage change to price and amount (positive change_pct decreases values, negative increases)
factor = (1.0 - change_pct)
prev.price = (h.price * factor).round(4)
prev.amount = (h.amount * factor).round(4)
prev.external_id = nil
if dry_run
puts({ ok: true, dry_run: true, holding_id: h.id, would_create: prev.attributes.slice("account_id", "security_id", "date", "qty", "price", "amount", "currency") }.to_json)
else
prev.save!
puts({ ok: true, created_prev_id: prev.id, date: prev.date, amount: prev.amount, price: prev.price }.to_json)
end
rescue => e
puts({ ok: false, error: e.class.name, message: e.message }.to_json)
exit 1
end
desc "Remove a seeded snapshot by its id. Args: snapshot_id"
task :remove_snapshot, [ :snapshot_id ] => :environment do |_, args|
id = args[:snapshot_id]
unless id
puts({ ok: false, error: "usage", message: "Provide id" }.to_json)
exit 1
end
h = Holding.find(id)
h.destroy!
puts({ ok: true, removed: id }.to_json)
rescue => e
puts({ ok: false, error: e.class.name, message: e.message }.to_json)
exit 1
end
end
end

83
lib/tasks/simplefin.rake
View File

@@ -31,5 +31,88 @@ namespace :sure do
puts({ error: e.class.name, message: e.message, backtrace: e.backtrace&.take(3) }.to_json)
exit 1
end
desc "Encrypt existing SimpleFin access_url values (idempotent). Args: batch_size, limit, dry_run"
task :encrypt_access_urls, [ :batch_size, :limit, :dry_run ] => :environment do |_, args|
Rake::Task["sure:encrypt_access_urls"].invoke(args[:batch_size], args[:limit], args[:dry_run])
end
end
desc "Encrypt existing SimpleFin access_url values (idempotent). Args: batch_size, limit, dry_run"
task :encrypt_access_urls, [ :batch_size, :limit, :dry_run ] => :environment do |_, args|
# Parse args or fall back to ENV overrides for convenience
raw_batch = args[:batch_size].presence || ENV["BATCH_SIZE"].presence || ENV["SURE_BATCH_SIZE"].presence
raw_limit = args[:limit].presence || ENV["LIMIT"].presence || ENV["SURE_LIMIT"].presence
raw_dry = args[:dry_run].presence || ENV["DRY_RUN"].presence || ENV["SURE_DRY_RUN"].presence
batch_size = raw_batch.to_i
batch_size = 100 if batch_size <= 0
limit = raw_limit.to_i
limit = nil if limit <= 0
# Default to non-destructive (dry run) unless explicitly disabled
dry_run = case raw_dry.to_s.strip.downcase
when "0", "false", "no", "n" then false
when "1", "true", "yes", "y" then true
else
true
end
# Guard: ensure encryption is configured (centralized on the model)
encryption_ready = SimplefinItem.encryption_ready?
unless encryption_ready
puts({
ok: false,
error: "encryption_not_configured",
message: "Rails.application.credentials.active_record_encryption is missing; cannot encrypt access_url"
}.to_json)
exit 1
end
total_seen = 0
total_updated = 0
failed = []
scope = SimplefinItem.order(:id)
begin
scope.in_batches(of: batch_size) do |batch|
batch.each do |item|
break if limit && total_seen >= limit
total_seen += 1
next if dry_run
begin
# Reassign to trigger encryption on write
item.update!(access_url: item.access_url)
total_updated += 1
rescue ActiveRecord::RecordInvalid => e
failed << { id: item.id, error: e.class.name, message: e.message }
rescue ActiveRecord::StatementInvalid => e
failed << { id: item.id, error: e.class.name, message: e.message }
rescue => e
failed << { id: item.id, error: e.class.name, message: e.message }
end
end
break if limit && total_seen >= limit
end
puts({
ok: true,
dry_run: dry_run,
batch_size: batch_size,
limit: limit,
processed: total_seen,
updated: total_updated,
failed_count: failed.size,
failed_samples: failed.take(5)
}.to_json)
rescue => e
puts({ ok: false, error: e.class.name, message: e.message, backtrace: e.backtrace&.take(3) }.to_json)
exit 1
end
end
end

256
lib/tasks/simplefin_backfill.rake Normal file
View File

@@ -0,0 +1,256 @@
# frozen_string_literal: true
# Backfill and maintenance tasks for SimpleFin transactions metadata and demo cleanup
#
# Usage examples:
# # Preview (no writes) a 45-day backfill for a single item
# # NOTE: Use your real item id
# bin/rails 'sure:simplefin:backfill_extra[item_id=ec255931-62ff-4a68-abda-16067fad0429,days=45,dry_run=true]'
#
# # Execute the backfill (writes enabled)
# bin/rails 'sure:simplefin:backfill_extra[item_id=ec255931-62ff-4a68-abda-16067fad0429,days=45,dry_run=false]'
#
# # Limit to a single linked account by Account ID (UUID from your UI/db)
# bin/rails 'sure:simplefin:backfill_extra[account_id=8b46387c-5aa4-4a92-963a-4392c10999c9,days=30,dry_run=false]'
#
# # Clean up known demo entries for a specific account (dry-run first)
# bin/rails 'sure:simplefin:cleanup_demo_entries[account_id=8b46387c-5aa4-4a92-963a-4392c10999c9,dry_run=true]'
# bin/rails 'sure:simplefin:cleanup_demo_entries[account_id=8b46387c-5aa4-4a92-963a-4392c10999c9,dry_run=false]'
namespace :sure do
namespace :simplefin do
desc "Backfill transactions.extra for SimpleFin imports over a recent window. Args (named): item_id, account_id, days=30, dry_run=true, force=false"
task :backfill_extra, [ :item_id, :account_id, :days, :dry_run, :force ] => :environment do |_, args|
# Support both positional and named (key=value) args; prefer named
kv = {}
[ args[:item_id], args[:account_id], args[:days], args[:dry_run], args[:force] ].each do |raw|
next unless raw.is_a?(String) && raw.include?("=")
k, v = raw.split("=", 2)
kv[k.to_s] = v
end
item_id = (kv["item_id"] || args[:item_id]).presence
account_id = (kv["account_id"] || args[:account_id]).presence
days_i = (kv["days"] || args[:days] || 30).to_i
dry_raw = (kv["dry_run"] || args[:dry_run]).to_s.downcase
force_raw = (kv["force"] || args[:force]).to_s.downcase
# Default to dry_run=true unless explicitly disabled, and validate input strictly
if dry_raw.blank?
dry_run = true
elsif %w[1 true yes y].include?(dry_raw)
dry_run = true
elsif %w[0 false no n].include?(dry_raw)
dry_run = false
else
puts({ ok: false, error: "invalid_argument", message: "dry_run must be one of: true/yes/1 or false/no/0" }.to_json)
exit 1
end
force = %w[1 true yes y].include?(force_raw)
days_i = 30 if days_i <= 0
window_start = days_i.days.ago.to_date
window_end = Date.today
# Basic UUID validation when provided
uuid_rx = /\A[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}\z/i
if item_id.present? && !item_id.match?(uuid_rx)
puts({ ok: false, error: "invalid_argument", message: "item_id must be a hyphenated UUID" }.to_json)
exit 1
end
if account_id.present? && !account_id.match?(uuid_rx)
puts({ ok: false, error: "invalid_argument", message: "account_id must be a hyphenated UUID" }.to_json)
exit 1
end
# Select SimplefinAccounts to process
sfas = if item_id.present?
item = SimplefinItem.find(item_id)
item.simplefin_accounts
elsif account_id.present?
acct = Account.find(account_id)
# Prefer new provider linkage, fallback to legacy foreign key
sfa = if acct.account_providers.where(provider_type: "SimplefinAccount").exists?
AccountProvider.find_by(account: acct, provider_type: "SimplefinAccount")&.provider
else
SimplefinAccount.find_by(account: acct)
end
Array.wrap(sfa)
else
puts({ ok: false, error: "usage", message: "Provide item_id or account_id" }.to_json)
exit 1
end
# Ensure sfas is an ActiveRecord::Relation so downstream can call find_each safely
unless sfas.respond_to?(:find_each)
sfa_ids = Array.wrap(sfas).compact.map { |x| x.is_a?(SimplefinAccount) ? x.id : x }
sfas = SimplefinAccount.where(id: sfa_ids)
end
total_seen = 0
total_matched = 0
total_updated = 0
total_skipped = 0
total_errors = 0
sfas.find_each do |sfa|
# Per-SFA counters (reset each iteration)
s_seen = s_matched = s_updated = s_skipped = s_errors = 0
acct = sfa.current_account
unless acct
puts({ warn: "no_linked_account", sfa_id: sfa.id, name: sfa.name }.to_json)
next
end
txs = Array(sfa.raw_transactions_payload).map { |t| t.with_indifferent_access }
if txs.empty?
puts({ info: "no_raw_transactions", sfa_id: sfa.id, name: sfa.name }.to_json)
next
end
txs.each do |t|
begin
posted = t[:posted]
trans = t[:transacted_at]
# convert to Date where possible for window filtering
posted_d = case posted
when String then Date.parse(posted) rescue nil
when Numeric then Time.zone.at(posted).to_date rescue nil
when Date then posted
when Time, DateTime then posted.to_date
else nil
end
trans_d = case trans
when String then Date.parse(trans) rescue nil
when Numeric then Time.zone.at(trans).to_date rescue nil
when Date then trans
when Time, DateTime then trans.to_date
else nil
end
best = posted_d || trans_d
# If neither date is available, skip (cannot window-match safely)
if best.nil? || best < window_start || best > window_end
s_skipped += 1
total_skipped += 1
next
end
s_seen += 1
total_seen += 1
# Build extra payload exactly like SimplefinEntry::Processor
sf = {}
sf["payee"] = t[:payee] if t.key?(:payee)
sf["memo"] = t[:memo] if t.key?(:memo)
sf["description"] = t[:description] if t.key?(:description)
sf["extra"] = t[:extra] if t[:extra].is_a?(Hash)
extra_hash = sf.empty? ? nil : { "simplefin" => sf }
# Skip if no metadata to add (unless forcing overwrite)
if extra_hash.nil? && !force
s_skipped += 1
total_skipped += 1
next
end
# Reuse the import adapter path so we merge onto the existing entry
adapter = Account::ProviderImportAdapter.new(acct)
external_id = t[:id].present? ? "simplefin_#{t[:id]}" : nil
if external_id.nil?
s_skipped += 1
total_skipped += 1
puts({ warn: "missing_transaction_id", sfa_id: sfa.id, account_id: acct.id, name: sfa.name }.to_json)
next
end
if dry_run
# Simulate: check if we can composite-match; we won't persist
entry = external_id && acct.entries.find_by(external_id: external_id, source: "simplefin")
processor = SimplefinEntry::Processor.new(t, simplefin_account: sfa)
window_days = (acct.accountable_type.in?([ "CreditCard", "Loan" ]) ? 5 : 3)
entry ||= adapter.composite_match(
source: "simplefin",
name: processor.send(:name),
amount: processor.send(:amount),
date: (posted_d || trans_d),
window_days: window_days
)
matched = entry.present?
if matched
s_matched += 1
total_matched += 1
end
else
processed = SimplefinEntry::Processor.new(t, simplefin_account: sfa).process
if processed&.transaction&.extra.present?
s_updated += 1
total_updated += 1
else
s_skipped += 1
total_skipped += 1
end
end
rescue => e
s_errors += 1
total_errors += 1
puts({ error: e.class.name, message: e.message }.to_json)
end
end
puts({ sfa_id: sfa.id, account_id: acct.id, name: sfa.name, seen: s_seen, matched: s_matched, updated: s_updated, skipped: s_skipped, errors: s_errors, window_start: window_start, window_end: window_end, dry_run: dry_run, force: force }.to_json)
end
puts({ ok: true, total_seen: total_seen, total_matched: total_matched, total_updated: total_updated, total_skipped: total_skipped, total_errors: total_errors, window_start: window_start, window_end: window_end, dry_run: dry_run, force: force }.to_json)
end
desc "List and optionally delete known demo SimpleFin entries for a given Account. Args (named): account_id, dry_run=true, pattern"
task :cleanup_demo_entries, [ :account_id, :dry_run, :pattern ] => :environment do |_, args|
kv = {}
[ args[:account_id], args[:dry_run], args[:pattern] ].each do |raw|
next unless raw.is_a?(String) && raw.include?("=")
k, v = raw.split("=", 2)
kv[k.to_s] = v
end
account_id = (kv["account_id"] || args[:account_id]).presence
dry_raw = (kv["dry_run"] || args[:dry_run]).to_s.downcase
pattern = (kv["pattern"] || args[:pattern]).presence || "simplefin_posted_demo_%|simplefin_posted_ui"
dry_run = dry_raw.blank? ? true : %w[1 true yes y].include?(dry_raw)
unless account_id.present?
puts({ ok: false, error: "usage", message: "Provide account_id" }.to_json)
exit 1
end
acct = Account.find(account_id)
patterns = pattern.split("|")
scope = acct.entries.where(source: "simplefin", entryable_type: "Transaction")
# Apply LIKE filters combined with OR
like_sql = patterns.map { |p| "external_id LIKE ?" }.join(" OR ")
like_vals = patterns.map { |p| p }
candidates = scope.where(like_sql, *like_vals)
out = candidates.order(date: :desc).map { |e| { id: e.id, external_id: e.external_id, date: e.date, name: e.name, amount: e.amount } }
puts({ account_id: acct.id, count: candidates.count, entries: out }.to_json)
if candidates.any? && !dry_run
deleted = 0
ActiveRecord::Base.transaction do
candidates.each do |e|
e.destroy!
deleted += 1
end
end
puts({ ok: true, deleted: deleted }.to_json)
else
puts({ ok: true, deleted: 0, dry_run: dry_run }.to_json)
end
end
end
end

106
lib/tasks/simplefin_debug.rake Normal file
View File

@@ -0,0 +1,106 @@
# frozen_string_literal: true
require "json"
require "time"
namespace :sure do
namespace :simplefin do
desc "Print last N raw SimpleFin transactions for a given item/account name. Args: item_id, account_name, limit (default 15)"
task :tx_debug, [ :item_id, :account_name, :limit ] => :environment do |_, args|
unless args[:item_id].present? && args[:account_name].present?
puts({ error: "usage", example: "bin/rails sure:simplefin:tx_debug[ITEM_ID,ACCOUNT_NAME,15]" }.to_json)
exit 1
end
item = SimplefinItem.find(args[:item_id])
limit = (args[:limit] || 15).to_i
limit = 15 if limit <= 0
sfa = item.simplefin_accounts.order(updated_at: :desc).find do |acc|
acc.name.to_s.downcase.include?(args[:account_name].to_s.downcase)
end
unless sfa
puts({ error: "not_found", message: "No SimplefinAccount matched", item_id: item.id, account_name: args[:account_name] }.to_json)
exit 1
end
txs = Array(sfa.raw_transactions_payload)
# Sort by best-known date: posted -> transacted_at -> as-is
txs = txs.map { |t| t.with_indifferent_access }
txs.sort_by! do |t|
posted = t[:posted]
trans = t[:transacted_at]
ts = if posted.is_a?(Numeric)
posted
elsif trans.is_a?(Numeric)
trans
else
0
end
-ts
end
sample = txs.first(limit)
out = sample.map do |t|
posted = t[:posted]
trans = t[:transacted_at]
{
id: t[:id],
amount: t[:amount],
description: t[:description],
payee: t[:payee],
memo: t[:memo],
posted: posted,
transacted_at: trans,
pending_flag: t[:pending],
inferred_pending: (trans.present? && posted.present? && posted.to_i > trans.to_i)
}
end
puts({ item_id: item.id, sfa_id: sfa.id, sfa_name: sfa.name, count: txs.size, sample: out }.to_json)
rescue => e
puts({ error: e.class.name, message: e.message, backtrace: e.backtrace&.take(3) }.to_json)
exit 1
end
desc "Print last N imported Entries for an account by name (linked to SimpleFin). Args: account_name, limit (default 15)"
task :entries_debug, [ :account_name, :limit ] => :environment do |_, args|
unless args[:account_name].present?
puts({ error: "usage", example: "bin/rails sure:simplefin:entries_debug[ACCOUNT_NAME,15]" }.to_json)
exit 1
end
acct = Account
.where("LOWER(name) LIKE ?", "%#{ActiveRecord::Base.sanitize_sql_like(args[:account_name].to_s.downcase)}%")
.order(updated_at: :desc)
.first
unless acct
puts({ error: "not_found", message: "No Account matched", account_name: args[:account_name] }.to_json)
exit 1
end
limit = (args[:limit] || 15).to_i
limit = 15 if limit <= 0
entries = acct.entries.includes(:entryable).where(entryable_type: "Transaction").order(date: :desc).limit(limit)
out = entries.map do |e|
{
id: e.id,
external_id: e.external_id,
source: e.source,
name: e.name,
amount: e.amount,
date: e.date,
was_merged: (e.entryable.respond_to?(:was_merged) ? e.entryable.was_merged : nil)
}
end
puts({ account_id: acct.id, account_name: acct.name, entries: out }.to_json)
rescue => e
puts({ error: e.class.name, message: e.message, backtrace: e.backtrace&.take(3) }.to_json)
exit 1
end
end
end

141
lib/tasks/simplefin_holdings_backfill.rake Normal file
View File

@@ -0,0 +1,141 @@
# frozen_string_literal: true
# Backfill holdings for SimpleFin-linked investment accounts using the existing
# SimplefinAccount::Investments::HoldingsProcessor. This is provider-agnostic at the
# UI/model level and works for any brokerage piped through SimpleFin (including Robinhood).
#
# Examples:
# # By SimpleFin item id (process all linked accounts under the item)
# # bin/rails 'sure:simplefin:backfill_holdings[item_id=ec255931-62ff-4a68-abda-16067fad0429,dry_run=true]'
# # Apply:
# # bin/rails 'sure:simplefin:backfill_holdings[item_id=ec255931-62ff-4a68-abda-16067fad0429,dry_run=false]'
#
# # By Account name contains (e.g., "Robinhood")
# # bin/rails 'sure:simplefin:backfill_holdings[account_name=Robinhood,dry_run=true]'
#
# # By Account id (UUID in your DB)
# # bin/rails 'sure:simplefin:backfill_holdings[account_id=<ACCOUNT_UUID>,dry_run=false]'
#
# Args (named or positional key=value):
# item_id - SimplefinItem id
# account_id - Account id (we will find its linked SimplefinAccount)
# account_name - Case-insensitive contains match to pick a single Account
# dry_run - default true; when true, do not write, just report what would be processed
# sleep_ms - per-account sleep to be polite to quotas (default 200ms)
namespace :sure do
namespace :simplefin do
desc "Backfill holdings for SimpleFin-linked investment accounts. Args: item_id, account_id, account_name, dry_run=true, sleep_ms=200"
task :backfill_holdings, [ :item_id, :account_id, :account_name, :dry_run, :sleep_ms ] => :environment do |_, args|
kv = {}
[ args[:item_id], args[:account_id], args[:account_name], args[:dry_run], args[:sleep_ms] ].each do |raw|
next unless raw.is_a?(String) && raw.include?("=")
k, v = raw.split("=", 2)
kv[k.to_s] = v
end
# Prefer named args parsed into kv; fall back to positional only when it is not a key=value string
fetch = ->(sym_key, str_key) do
if kv.key?(str_key)
kv[str_key]
else
v = args[sym_key]
v.is_a?(String) && v.include?("=") ? nil : v
end
end
item_id = fetch.call(:item_id, "item_id").presence
account_id = fetch.call(:account_id, "account_id").presence
account_name = fetch.call(:account_name, "account_name").presence
dry_raw = (kv["dry_run"] || args[:dry_run]).to_s.downcase
sleep_ms = ((kv["sleep_ms"] || args[:sleep_ms] || 200).to_i).clamp(0, 5000)
# Default to dry_run=true unless explicitly disabled, and validate input strictly
if dry_raw.blank?
dry_run = true
elsif %w[1 true yes y].include?(dry_raw)
dry_run = true
elsif %w[0 false no n].include?(dry_raw)
dry_run = false
else
puts({ ok: false, error: "invalid_argument", message: "dry_run must be one of: true/yes/1 or false/no/0" }.to_json)
exit 1
end
# Select SimplefinAccounts to process
sfas = []
if item_id.present?
begin
item = SimplefinItem.find(item_id)
sfas = item.simplefin_accounts.joins(:account)
rescue ActiveRecord::RecordNotFound
puts({ ok: false, error: "not_found", message: "SimplefinItem not found", item_id: item_id }.to_json)
exit 1
end
elsif account_id.present?
begin
acct = Account.find(account_id)
ap = acct.account_providers.where(provider_type: "SimplefinAccount").first
sfa = ap&.provider || SimplefinAccount.find_by(account: acct)
sfas = Array.wrap(sfa).compact
rescue ActiveRecord::RecordNotFound
puts({ ok: false, error: "not_found", message: "Account not found", account_id: account_id }.to_json)
exit 1
end
elsif account_name.present?
sanitized = ActiveRecord::Base.sanitize_sql_like(account_name.to_s.downcase)
acct = Account.where("LOWER(name) LIKE ?", "%#{sanitized}%")
.order(updated_at: :desc)
.first
unless acct
puts({ ok: false, error: "not_found", message: "No Account matched", account_name: account_name }.to_json)
exit 1
end
ap = acct.account_providers.where(provider_type: "SimplefinAccount").first
sfa = ap&.provider || SimplefinAccount.find_by(account: acct)
sfas = Array.wrap(sfa).compact
else
success = errors.empty?
puts({ ok: false, error: "usage", message: "Provide one of item_id, account_id, or account_name" }.to_json)
exit 1
end
total_accounts = 0
total_holdings_seen = 0
total_holdings_written = 0
errors = []
sfas.each do |sfa|
begin
account = sfa.current_account
next unless [ "Investment", "Crypto" ].include?(account&.accountable_type)
total_accounts += 1
holdings_data = Array(sfa.raw_holdings_payload)
if holdings_data.empty?
puts({ info: "no_raw_holdings", sfa_id: sfa.id, account_id: account.id, name: sfa.name }.to_json)
next
end
count = holdings_data.size
total_holdings_seen += count
if dry_run
puts({ dry_run: true, sfa_id: sfa.id, account_id: account.id, name: sfa.name, would_process: count }.to_json)
else
SimplefinAccount::Investments::HoldingsProcessor.new(sfa).process
total_holdings_written += count
puts({ ok: true, sfa_id: sfa.id, account_id: account.id, name: sfa.name, processed: count }.to_json)
end
sleep(sleep_ms / 1000.0) if sleep_ms.positive?
rescue => e
errors << { sfa_id: sfa.id, error: e.class.name, message: e.message }
end
end
puts({ ok: true, accounts_processed: total_accounts, holdings_seen: total_holdings_seen, holdings_written: total_holdings_written, errors: errors }.to_json)
end
end
end

64
lib/tasks/simplefin_unlink.rake Normal file
View File

@@ -0,0 +1,64 @@
# frozen_string_literal: true
namespace :sure do
namespace :simplefin do
desc "Unlink all provider links for a SimpleFin item so its accounts move to 'Other accounts'. Args: item_id, dry_run=true"
task :unlink_item, [ :item_id, :dry_run ] => :environment do |_, args|
require "json"
item_id = args[:item_id].to_s.strip.presence
dry_raw = args[:dry_run].to_s.downcase
# Default to non-destructive (dry run) unless explicitly disabled
# Accept only explicit true/false values; abort on invalid input to prevent accidental destructive runs
if dry_raw.blank?
dry_run = true
elsif %w[1 true yes y].include?(dry_raw)
dry_run = true
elsif %w[0 false no n].include?(dry_raw)
dry_run = false
else
puts({ ok: false, error: "invalid_argument", message: "dry_run must be one of: true/yes/1 or false/no/0" }.to_json)
exit 1
end
unless item_id.present?
puts({ ok: false, error: "usage", example: "bin/rails 'sure:simplefin:unlink_item[ITEM_UUID,true]'" }.to_json)
exit 1
end
# Basic UUID v4 validation (hyphenated 36 chars)
uuid_v4 = /\A[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}\z/i
unless item_id.match?(uuid_v4)
puts({ ok: false, error: "invalid_argument", message: "item_id must be a hyphenated UUID (v4)" }.to_json)
exit 1
end
item = SimplefinItem.find(item_id)
results = item.unlink_all!(dry_run: dry_run)
# Redact potentially sensitive names or identifiers in output
# Recursively redact sensitive fields from output
def redact_sensitive(obj)
case obj
when Hash
obj.except(:name, :payee, :account_number).transform_values { |v| redact_sensitive(v) }
when Array
obj.map { |item| redact_sensitive(item) }
else
obj
end
end
safe_details = redact_sensitive(Array(results))
puts({ ok: true, dry_run: dry_run, item_id: item.id, unlinked_count: safe_details.size, details: safe_details }.to_json)
rescue ActiveRecord::RecordNotFound
puts({ ok: false, error: "not_found", message: "SimplefinItem not found for given item_id" }.to_json)
exit 1
rescue => e
puts({ ok: false, error: e.class.name, message: e.message }.to_json)
exit 1
end
end
end