Files
sure/app/models/up_entry/processor.rb
T
4e010493c7 feat(up): map Up category slugs to Sure categories on import (#2487)
* feat(up): map Up category slugs to Sure categories on import

UpEntry::Processor captured Up's category slug into extra but never applied it, so
Up transactions imported uncategorised even though the user had already tagged them
in the Up app.

Add UpAccount::Transactions::CategoryTaxonomy + CategoryMatcher, mirroring
PlaidAccount::Transactions::CategoryMatcher: map Up's child category slugs onto the
family's existing/default Sure categories by alias, and wire the matcher through
UpAccount::Transactions::Processor into UpEntry::Processor. The category is applied via
the adapter's enrich_attribute, so a category the user has set or locked is preserved
on re-sync.

High-confidence mappings only. Up-specific categories with no honest Sure default
(Booze, Pets, Apps & Games, Life Admin, Technology, ...) intentionally stay
uncategorised for the user's own rules / AI, since a wrong auto-category is worse than
none. Adds a matcher unit test and processor wiring tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* refactor(up): match category slugs as strings in CategoryMatcher

Up category ids are string slugs; compare them against the taxonomy keys as strings
so the lookup does not depend on the keys being symbols. No behaviour change (the
"slug": hash syntax already produces symbol keys that matched the symbolized input,
covered by the matcher unit test), but it removes a subtle footgun and reads clearer.
Flagged by the Codex review on the PR.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(up): make category import non-destructive; word-boundary the alias match

Per review feedback: do not bootstrap Sure's default categories during a sync.
family_categories now returns the family's existing categories without creating
defaults, so a family that has none (deliberately cleared, or pre-onboarding) gets
uncategorised transactions rather than having the full default set silently created.
Matching resumes once the user sets up categories through the normal UI flow.

Also word-boundary the "and" stripping in the matcher normalization so it strips only
the standalone conjunction, not "and" inside a word (e.g. errand). Adds a processor
test for the non-destructive guarantee.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Gavin Matthews <matthews.gav@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-22 08:26:48 +02:00

241 lines
8.6 KiB
Ruby

require "digest/md5"
class UpEntry::Processor
include CurrencyNormalizable
# Stable external id for a transaction: its Up id when present, else a content
# hash so pending entries lacking an id stay deduplicated across syncs.
def self.canonical_external_id(up_transaction)
data = up_transaction.with_indifferent_access
id = data[:id].presence
return "up_#{id}" if id.present?
"up_pending_#{content_hash_for(data)}"
end
# Up marks unsettled transactions with status "HELD"; settled ones are "SETTLED".
def self.pending?(up_transaction)
data = up_transaction.with_indifferent_access
data[:status].to_s.upcase == "HELD"
end
# MD5 of account/date/amount/description, used to identify id-less pendings.
def self.content_hash_for(data)
amount = data[:amount].is_a?(Hash) ? data[:amount].with_indifferent_access : {}
attributes = [
data[:account_id],
data[:createdAt],
amount[:value],
data[:description]
].compact.join("|")
Digest::MD5.hexdigest(attributes)
end
# Build a processor for a single raw Up transaction tied to +up_account+.
#
# category_matcher (optional) maps Up's category slug onto one of the family's
# existing Sure categories. It is injected by UpAccount::Transactions::Processor so a
# single matcher (built once per account) is reused across the account's transactions.
def initialize(up_transaction, up_account:, category_matcher: nil)
@up_transaction = up_transaction
@up_account = up_account
@category_matcher = category_matcher
end
# Import the transaction into the linked Sure account via the import adapter.
# Returns nil when the account isn't linked; re-raises on validation/save errors.
def process
unless account.present?
Rails.logger.warn "UpEntry::Processor - No linked account for up_account #{up_account.id}, skipping transaction #{external_id}"
return nil
end
import_adapter.import_transaction(
external_id: external_id,
amount: amount,
currency: currency,
date: date,
name: name,
source: "up",
category_id: matched_category_id,
kind: kind,
merchant: merchant,
notes: notes,
extra: extra_metadata
)
rescue ArgumentError => e
Rails.logger.error "UpEntry::Processor - Validation error for transaction #{external_id}: #{e.message}"
raise
rescue ActiveRecord::RecordInvalid, ActiveRecord::RecordNotSaved => e
Rails.logger.error "UpEntry::Processor - Failed to save transaction #{external_id}: #{e.message}"
raise StandardError.new("Failed to import transaction: #{e.message}")
rescue => e
Rails.logger.error "UpEntry::Processor - Unexpected error processing transaction #{external_id}: #{e.class} - #{e.message}"
Rails.logger.error e.backtrace.join("\n")
raise StandardError.new("Unexpected error importing transaction: #{e.message}")
end
private
attr_reader :up_transaction, :up_account
# Memoized adapter that writes provider transactions into the Sure account.
def import_adapter
@import_adapter ||= Account::ProviderImportAdapter.new(account)
end
# The linked Sure account for this transaction, if any.
def account
@account ||= up_account.current_account
end
# The raw transaction as an indifferent-access hash.
def data
@data ||= up_transaction.with_indifferent_access
end
# Canonical external id for this transaction (see .canonical_external_id).
def external_id
@external_id ||= self.class.canonical_external_id(data)
end
# Display name: the Up description, or a generic fallback.
def name
data[:description].presence || I18n.t("transactions.unknown_name")
end
# The id of the Sure category that Up's category slug maps to, or nil when no
# matcher was injected, the transaction has no Up category (transfers/income), or
# the slug has no confident equivalent among the family's categories. The import
# adapter applies this via enrich_attribute, so it never overwrites a category the
# user has set or locked.
def matched_category_id
return nil unless @category_matcher
@category_matcher.match(data[:category_id])&.id
end
# The id of the other account in an internal money movement, if any (see
# Provider::Up#flatten_transaction). Present for transfers between the user's own
# accounts and for round-ups swept into a Saver; nil for ordinary income/expense.
def transfer_account_id
data[:transfer_account_id].presence
end
# Mark internal movements as funds_movement so they are excluded from income,
# expense, and budget analytics. Two-sided transfers between two linked accounts are
# additionally paired into a Transfer by Family#auto_match_transfers!; one-sided moves
# (counterpart not linked in Sure) and round-ups rely on this flag, since the matcher
# has no opposing entry to pair them with.
def kind
transfer_account_id ? "funds_movement" : nil
end
# Optional user-entered message attached to the transaction.
def notes
data[:message].presence
end
# Find or create the merchant for this transaction's description, or nil.
def merchant
merchant_name = data[:description].to_s.strip.presence
return nil unless merchant_name
provider_merchant_id = "up_merchant_#{Digest::MD5.hexdigest(merchant_name.downcase)}"
@merchant ||= import_adapter.find_or_create_merchant(
provider_merchant_id: provider_merchant_id,
name: merchant_name,
source: "up"
)
rescue ActiveRecord::RecordInvalid => e
Rails.logger.error "UpEntry::Processor - Failed to create merchant '#{merchant_name}': #{e.message}"
nil
end
# Up amounts use banking convention: negative is money out, positive is money in.
# Sure stores expenses as positive and income as negative, so the sign is flipped.
def amount
raw_value = amount_data[:value]
parsed_amount = case raw_value
when String
BigDecimal(raw_value)
when Numeric
BigDecimal(raw_value.to_s)
else
BigDecimal("0")
end
-parsed_amount
rescue ArgumentError => e
Rails.logger.error "Failed to parse Up transaction amount: #{e.class}"
raise ArgumentError, "Invalid transaction amount"
end
# Transaction currency, falling back to the account/default currency.
def currency
parse_currency(amount_data[:currencyCode]) || up_account.currency || account&.currency || "AUD"
end
# Settlement date (or creation date for pendings) as a Date.
def date
value = data[:settledAt].presence || data[:createdAt].presence
case value
when String
if value.include?("T") || value.include?(":")
Time.parse(value).in_time_zone(account&.family&.timezone).to_date
else
Date.parse(value)
end
when Integer, Float
Time.at(value).in_time_zone(account&.family&.timezone).to_date
when Time, DateTime
value.in_time_zone(account&.family&.timezone).to_date
when Date
value
else
Rails.logger.error("Up transaction has no usable date value")
raise ArgumentError, "Invalid date format"
end
rescue ArgumentError, TypeError => e
Rails.logger.error("Failed to parse Up transaction date: #{e.class}")
raise ArgumentError, "Unable to parse transaction date"
end
# Provider metadata persisted on Transaction#extra (pending/status/fx/etc.).
def extra_metadata
{
"up" => {
"pending" => pending?,
"status" => data[:status],
"category_id" => data[:category_id],
"transfer_account_id" => transfer_account_id,
"raw_text" => data[:rawText],
"fx_from" => foreign_amount_data[:currencyCode],
"fx_amount" => foreign_amount_data[:value]
}.compact
}
end
# Whether this transaction is still HELD (unsettled) on Up.
def pending?
self.class.pending?(data)
end
# The native amount object ({ value:, currencyCode: }) as a hash.
def amount_data
@amount_data ||= data[:amount].is_a?(Hash) ? data[:amount].with_indifferent_access : {}
end
# The foreign-currency amount object for FX transactions, or empty hash.
def foreign_amount_data
@foreign_amount_data ||= data[:foreignAmount].is_a?(Hash) ? data[:foreignAmount].with_indifferent_access : {}
end
# CurrencyNormalizable hook: warn when an Up currency code is unrecognized.
def log_invalid_currency(currency_value)
Rails.logger.warn("Invalid currency code '#{currency_value}' in Up transaction #{external_id}, falling back to account currency")
end
end