mirror of
https://github.com/we-promise/sure.git
synced 2026-04-08 14:54:49 +00:00
* Add tests and enhance logic for SimpleFin account synchronization and reconciliation - Added retry logic with exponential backoff for network errors in `Provider::Simplefin`. - Introduced tests to verify retry functionality and error handling for rate-limit, server errors, and stale data. - Updated `SimplefinItem` to detect stale sync status and reconciliation issues. - Enhanced UI to display stale sync warnings and data integrity notices. - Improved SimpleFin account matching during updates with multi-tier strategy (ID, fingerprint, fuzzy match). - Added transaction reconciliation logic to detect data gaps, transaction count drops, and duplicate transaction IDs. * Introduce `SimplefinConnectionUpdateJob` for asynchronous SimpleFin connection updates - Moved SimpleFin connection update logic to `SimplefinConnectionUpdateJob` to improve response times by offloading network retries, data fetching, and reconciliation tasks. - Enhanced SimpleFin account matching with a multi-tier strategy (ID, fingerprint, fuzzy name match). - Added retry logic and bounded latency for token claim requests in `Provider::Simplefin`. - Updated tests to cover the new job flow and ensure correct account reconciliation during updates. * Remove unused SimpleFin account matching logic and improve error handling in `SimplefinConnectionUpdateJob` - Deleted the multi-tier account matching logic from `SimplefinItemsController` as it is no longer used. - Enhanced error handling in `SimplefinConnectionUpdateJob` to gracefully handle import failures, ensuring orphaned items can be manually resolved. - Updated job flow to conditionally set item status based on the success of import operations. * Fix SimpleFin sync: check both legacy FK and AccountProvider for linked accounts * Add crypto, checking, savings, and cash account detection; refine subtype selection and linking - Enhanced `Simplefin::AccountTypeMapper` to include detection for crypto, checking, savings, and standalone cash accounts. - Improved subtype selection UI with validation and warning indicators for missing selections. - Updated SimpleFin account linking to handle both legacy FK and `AccountProvider` associations consistently. - Refined job flow and importer logic for better handling of linked accounts and subtype inference. * Improve `SimplefinConnectionUpdateJob` and holdings processing logic - Fixed race condition in `SimplefinConnectionUpdateJob` by moving `destroy_later` calls outside of transactions. - Updated fuzzy name match logic to use Levenshtein distance for better accuracy. - Enhanced synthetic ticker generation in holdings processor with hash suffix for uniqueness. * Refine SimpleFin entry processing logic and ensure `extra` data persistence - Simplified pending flag determination to rely solely on provider-supplied values. - Fixed potential stale values in `extra` by ensuring deep merge overwrite with `entry.transaction.save!`. * Replace hardcoded fallback transaction description with localized string * Refine pending flag logic in SimpleFin processor tests - Adjust test to prevent falsely inferring pending status from missing posted dates. - Ensure provider explicitly sets pending flag for transactions. * Add `has_many :holdings` association to `AccountProvider` with `dependent: :nullify` --------- Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
171 lines
6.1 KiB
Ruby
171 lines
6.1 KiB
Ruby
class Provider::Simplefin
|
|
# Pending: some institutions do not return pending transactions even with `pending=1`.
|
|
# This is provider variability (not a bug). For troubleshooting, you can set
|
|
# `SIMPLEFIN_INCLUDE_PENDING=1` and/or `SIMPLEFIN_DEBUG_RAW=1` (both default-off).
|
|
# These are centralized in `Rails.configuration.x.simplefin.*` via
|
|
# `config/initializers/simplefin.rb`.
|
|
include HTTParty
|
|
|
|
headers "User-Agent" => "Sure Finance SimpleFin Client"
|
|
default_options.merge!(verify: true, ssl_verify_mode: OpenSSL::SSL::VERIFY_PEER, timeout: 120)
|
|
|
|
# Retry configuration for transient network failures
|
|
MAX_RETRIES = 3
|
|
INITIAL_RETRY_DELAY = 2 # seconds
|
|
MAX_RETRY_DELAY = 30 # seconds
|
|
|
|
# Errors that are safe to retry (transient network issues)
|
|
RETRYABLE_ERRORS = [
|
|
SocketError,
|
|
Net::OpenTimeout,
|
|
Net::ReadTimeout,
|
|
Errno::ECONNRESET,
|
|
Errno::ECONNREFUSED,
|
|
Errno::ETIMEDOUT,
|
|
EOFError
|
|
].freeze
|
|
|
|
def initialize
|
|
end
|
|
|
|
def claim_access_url(setup_token)
|
|
# Decode the base64 setup token to get the claim URL
|
|
claim_url = Base64.decode64(setup_token)
|
|
|
|
# Use retry logic for transient network failures during token claim
|
|
# Claim should be fast; keep request-path latency bounded.
|
|
response = with_retries("POST /claim", max_retries: 1, sleep: false) do
|
|
HTTParty.post(claim_url, timeout: 15)
|
|
end
|
|
|
|
case response.code
|
|
when 200
|
|
# The response body contains the access URL with embedded credentials
|
|
response.body.strip
|
|
when 403
|
|
raise SimplefinError.new("Setup token may be compromised, expired, or already used", :token_compromised)
|
|
else
|
|
raise SimplefinError.new("Failed to claim access URL: #{response.code} #{response.message}", :claim_failed)
|
|
end
|
|
end
|
|
|
|
def get_accounts(access_url, start_date: nil, end_date: nil, pending: nil)
|
|
# Build query parameters
|
|
query_params = {}
|
|
|
|
# SimpleFin expects Unix timestamps for dates
|
|
if start_date
|
|
start_timestamp = start_date.to_time.to_i
|
|
query_params["start-date"] = start_timestamp.to_s
|
|
end
|
|
|
|
if end_date
|
|
end_timestamp = end_date.to_time.to_i
|
|
query_params["end-date"] = end_timestamp.to_s
|
|
end
|
|
|
|
query_params["pending"] = pending ? "1" : "0" unless pending.nil?
|
|
|
|
accounts_url = "#{access_url}/accounts"
|
|
accounts_url += "?#{URI.encode_www_form(query_params)}" unless query_params.empty?
|
|
|
|
# The access URL already contains HTTP Basic Auth credentials
|
|
# Use retry logic with exponential backoff for transient network failures
|
|
response = with_retries("GET /accounts") do
|
|
HTTParty.get(accounts_url)
|
|
end
|
|
|
|
case response.code
|
|
when 200
|
|
JSON.parse(response.body, symbolize_names: true)
|
|
when 400
|
|
Rails.logger.error "SimpleFin API: Bad request - #{response.body}"
|
|
raise SimplefinError.new("Bad request to SimpleFin API: #{response.body}", :bad_request)
|
|
when 403
|
|
raise SimplefinError.new("Access URL is no longer valid", :access_forbidden)
|
|
when 402
|
|
raise SimplefinError.new("Payment required to access this account", :payment_required)
|
|
when 429
|
|
Rails.logger.warn "SimpleFin API: Rate limited - #{response.body}"
|
|
raise SimplefinError.new("SimpleFin rate limit exceeded. Please try again later.", :rate_limited)
|
|
when 500..599
|
|
Rails.logger.error "SimpleFin API: Server error - Code: #{response.code}, Body: #{response.body}"
|
|
raise SimplefinError.new("SimpleFin server error (#{response.code}). Please try again later.", :server_error)
|
|
else
|
|
Rails.logger.error "SimpleFin API: Unexpected response - Code: #{response.code}, Body: #{response.body}"
|
|
raise SimplefinError.new("Failed to fetch accounts: #{response.code} #{response.message} - #{response.body}", :fetch_failed)
|
|
end
|
|
end
|
|
|
|
def get_info(base_url)
|
|
response = HTTParty.get("#{base_url}/info")
|
|
|
|
case response.code
|
|
when 200
|
|
response.body.strip.split("\n")
|
|
else
|
|
raise SimplefinError.new("Failed to get server info: #{response.code} #{response.message}", :info_failed)
|
|
end
|
|
end
|
|
|
|
class SimplefinError < StandardError
|
|
attr_reader :error_type
|
|
|
|
def initialize(message, error_type = :unknown)
|
|
super(message)
|
|
@error_type = error_type
|
|
end
|
|
end
|
|
|
|
private
|
|
|
|
# Execute a block with retry logic and exponential backoff for transient network errors.
|
|
# This helps handle temporary network issues that cause autosync failures while
|
|
# manual sync (with user retry) succeeds.
|
|
def with_retries(operation_name, max_retries: MAX_RETRIES, sleep: true)
|
|
retries = 0
|
|
|
|
begin
|
|
yield
|
|
rescue *RETRYABLE_ERRORS => e
|
|
retries += 1
|
|
|
|
if retries <= max_retries
|
|
delay = calculate_retry_delay(retries)
|
|
Rails.logger.warn(
|
|
"SimpleFin API: #{operation_name} failed (attempt #{retries}/#{max_retries}): " \
|
|
"#{e.class}: #{e.message}. Retrying in #{delay}s..."
|
|
)
|
|
Kernel.sleep(delay) if sleep && delay.to_f.positive?
|
|
retry
|
|
else
|
|
Rails.logger.error(
|
|
"SimpleFin API: #{operation_name} failed after #{max_retries} retries: " \
|
|
"#{e.class}: #{e.message}"
|
|
)
|
|
raise SimplefinError.new(
|
|
"Network error after #{max_retries} retries: #{e.message}",
|
|
:network_error
|
|
)
|
|
end
|
|
rescue SimplefinError => e
|
|
# Preserve original error type and message.
|
|
raise
|
|
rescue => e
|
|
# Non-retryable errors are logged and re-raised immediately
|
|
Rails.logger.error "SimpleFin API: #{operation_name} failed with non-retryable error: #{e.class}: #{e.message}"
|
|
raise SimplefinError.new("Exception during #{operation_name}: #{e.message}", :request_failed)
|
|
end
|
|
end
|
|
|
|
# Calculate delay with exponential backoff and jitter
|
|
def calculate_retry_delay(retry_count)
|
|
# Exponential backoff: 2^retry * initial_delay
|
|
base_delay = INITIAL_RETRY_DELAY * (2 ** (retry_count - 1))
|
|
# Add jitter (0-25% of base delay) to prevent thundering herd
|
|
jitter = base_delay * rand * 0.25
|
|
# Cap at max delay
|
|
[ base_delay + jitter, MAX_RETRY_DELAY ].min
|
|
end
|
|
end
|