mirror of
https://github.com/we-promise/sure.git
synced 2026-09-03 13:51:29 +00:00
feat(up): map Up category slugs to Sure categories on import (#2487)
* feat(up): map Up category slugs to Sure categories on import UpEntry::Processor captured Up's category slug into extra but never applied it, so Up transactions imported uncategorised even though the user had already tagged them in the Up app. Add UpAccount::Transactions::CategoryTaxonomy + CategoryMatcher, mirroring PlaidAccount::Transactions::CategoryMatcher: map Up's child category slugs onto the family's existing/default Sure categories by alias, and wire the matcher through UpAccount::Transactions::Processor into UpEntry::Processor. The category is applied via the adapter's enrich_attribute, so a category the user has set or locked is preserved on re-sync. High-confidence mappings only. Up-specific categories with no honest Sure default (Booze, Pets, Apps & Games, Life Admin, Technology, ...) intentionally stay uncategorised for the user's own rules / AI, since a wrong auto-category is worse than none. Adds a matcher unit test and processor wiring tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(up): match category slugs as strings in CategoryMatcher Up category ids are string slugs; compare them against the taxonomy keys as strings so the lookup does not depend on the keys being symbols. No behaviour change (the "slug": hash syntax already produces symbol keys that matched the symbolized input, covered by the matcher unit test), but it removes a subtle footgun and reads clearer. Flagged by the Codex review on the PR. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(up): make category import non-destructive; word-boundary the alias match Per review feedback: do not bootstrap Sure's default categories during a sync. family_categories now returns the family's existing categories without creating defaults, so a family that has none (deliberately cleared, or pre-onboarding) gets uncategorised transactions rather than having the full default set silently created. Matching resumes once the user sets up categories through the normal UI flow. Also word-boundary the "and" stripping in the matcher normalization so it strips only the standalone conjunction, not "and" inside a word (e.g. errand). Adds a processor test for the non-destructive guarantee. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Gavin Matthews <matthews.gav@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
threatsurfer
co-authored by
Claude Opus 4.8
Gavin Matthews
Claude Opus 4.8
Gavin Matthews
parent
4d1d33f91d
commit
4e010493c7
@@ -0,0 +1,86 @@
|
||||
# Auto-matches Up Bank's spending categories to the family's existing Sure categories.
|
||||
# Mirrors PlaidAccount::Transactions::CategoryMatcher: a fast, cheap, high-confidence
|
||||
# pass that never creates categories and never overwrites user data (the import adapter
|
||||
# applies the result via enrich_attribute, which respects locks). Up tags a transaction
|
||||
# with a child category slug (relationships.category.data.id); we map the honest
|
||||
# equivalents onto Sure's default categories and leave the rest for user rules / AI.
|
||||
class UpAccount::Transactions::CategoryMatcher
|
||||
include UpAccount::Transactions::CategoryTaxonomy
|
||||
|
||||
def initialize(user_categories = [])
|
||||
@user_categories = user_categories
|
||||
end
|
||||
|
||||
# up_category_slug is the value of relationships.category.data.id, e.g.
|
||||
# "restaurants-and-cafes". Returns a matching Category, or nil when the slug is
|
||||
# unknown or has no confident equivalent among the user's categories.
|
||||
def match(up_category_slug)
|
||||
details = category_details(up_category_slug)
|
||||
return nil unless details
|
||||
|
||||
# Try an exact match against the slug name first (rare), then the category's own
|
||||
# aliases, then its parent group's aliases.
|
||||
exact = normalized_user_categories.find { |c| c[:name] == details[:key].to_s }
|
||||
return user_categories.find { |c| c.id == exact[:id] } if exact
|
||||
|
||||
match_aliases(details[:aliases]) || match_aliases(details[:parent_aliases])
|
||||
end
|
||||
|
||||
private
|
||||
attr_reader :user_categories
|
||||
|
||||
def match_aliases(aliases)
|
||||
return nil if aliases.blank?
|
||||
|
||||
hit = normalized_user_categories.find do |category|
|
||||
name = category[:name]
|
||||
aliases.any? do |a|
|
||||
alias_str = a.to_s
|
||||
next true if name == alias_str
|
||||
next true if name.singularize == alias_str || name.pluralize == alias_str
|
||||
next true if alias_str.singularize == name || alias_str.pluralize == name
|
||||
|
||||
# Strip the standalone "and" conjunction (word-boundaried so it does not eat
|
||||
# "and" inside a word, e.g. errand), plus "&" and whitespace.
|
||||
normalized_name = name.gsub(/(\band\b|&|\s+)/, "").strip
|
||||
normalized_alias = alias_str.gsub(/(\band\b|&|\s+)/, "").strip
|
||||
normalized_name == normalized_alias
|
||||
end
|
||||
end
|
||||
|
||||
hit && user_categories.find { |c| c.id == hit[:id] }
|
||||
end
|
||||
|
||||
def category_details(up_category_slug)
|
||||
return nil if up_category_slug.blank?
|
||||
|
||||
# Up category ids are string slugs (e.g. "restaurants-and-cafes"); compare as
|
||||
# strings so the lookup does not depend on the taxonomy keys being symbols.
|
||||
slug = up_category_slug.to_s.downcase
|
||||
detailed_categories.find { |c| c[:key].to_s == slug }
|
||||
end
|
||||
|
||||
def detailed_categories
|
||||
@detailed_categories ||= CATEGORIES_MAP.flat_map do |parent_key, parent_data|
|
||||
parent_data[:detailed_categories].map do |child_key, child_data|
|
||||
{
|
||||
key: child_key,
|
||||
classification: child_data[:classification],
|
||||
aliases: child_data[:aliases],
|
||||
parent_key: parent_key,
|
||||
parent_aliases: parent_data[:aliases]
|
||||
}
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
def normalized_user_categories
|
||||
@normalized_user_categories ||= user_categories.map do |user_category|
|
||||
{ id: user_category.id, name: normalize_user_category_name(user_category.name) }
|
||||
end
|
||||
end
|
||||
|
||||
def normalize_user_category_name(name)
|
||||
name.to_s.downcase.gsub(/[^a-z0-9]/, " ").strip
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,81 @@
|
||||
# Up Bank's spending category taxonomy (GET /api/v1/categories): four parent groups
|
||||
# and ~40 child categories. A transaction references a CHILD slug via
|
||||
# relationships.category.data.id (e.g. "restaurants-and-cafes"); Up does not tag
|
||||
# transfers or income.
|
||||
#
|
||||
# We map those child slugs onto the family's existing/default Sure categories via
|
||||
# aliases, mirroring PlaidAccount::Transactions::CategoryTaxonomy. Aliases are written
|
||||
# in Sure's default-category vocabulary and matched case/punctuation-insensitively.
|
||||
#
|
||||
# Up categories with no honest Sure-default equivalent (e.g. Booze, Adult, Pets, Apps
|
||||
# & Games, Life Admin, Technology) are intentionally left without a mapping alias, so
|
||||
# they stay uncategorised for the user's own rules / AI to handle. A wrong
|
||||
# auto-category is worse than none.
|
||||
module UpAccount::Transactions::CategoryTaxonomy
|
||||
CATEGORIES_MAP = {
|
||||
"good-life": {
|
||||
classification: :expense,
|
||||
aliases: [],
|
||||
detailed_categories: {
|
||||
"restaurants-and-cafes": { classification: :expense, aliases: [ "food and drink", "dining", "restaurants" ] },
|
||||
"takeaway": { classification: :expense, aliases: [ "food and drink", "takeout" ] },
|
||||
"tv-and-music": { classification: :expense, aliases: [ "subscriptions" ] },
|
||||
"hobbies": { classification: :expense, aliases: [ "entertainment" ] },
|
||||
"events-and-gigs": { classification: :expense, aliases: [ "entertainment" ] },
|
||||
"holidays-and-travel": { classification: :expense, aliases: [ "travel" ] },
|
||||
"pubs-and-bars": { classification: :expense, aliases: [] },
|
||||
"booze": { classification: :expense, aliases: [] },
|
||||
"games-and-software": { classification: :expense, aliases: [] },
|
||||
"adult": { classification: :expense, aliases: [] },
|
||||
"lottery-and-gambling": { classification: :expense, aliases: [] },
|
||||
"tobacco-and-vaping": { classification: :expense, aliases: [] }
|
||||
}
|
||||
},
|
||||
"home": {
|
||||
classification: :expense,
|
||||
aliases: [],
|
||||
detailed_categories: {
|
||||
"groceries": { classification: :expense, aliases: [ "groceries" ] },
|
||||
"utilities": { classification: :expense, aliases: [ "utilities" ] },
|
||||
"internet": { classification: :expense, aliases: [ "utilities" ] },
|
||||
"rent-and-mortgage": { classification: :expense, aliases: [ "mortgage and rent", "rent" ] },
|
||||
"home-maintenance-and-improvements": { classification: :expense, aliases: [ "home improvement" ] },
|
||||
"homeware-and-appliances": { classification: :expense, aliases: [ "shopping" ] },
|
||||
"home-insurance-and-rates": { classification: :expense, aliases: [ "insurance" ] },
|
||||
"pets": { classification: :expense, aliases: [] }
|
||||
}
|
||||
},
|
||||
"personal": {
|
||||
classification: :expense,
|
||||
aliases: [],
|
||||
detailed_categories: {
|
||||
"health-and-medical": { classification: :expense, aliases: [ "healthcare" ] },
|
||||
"fitness-and-wellbeing": { classification: :expense, aliases: [ "sports and fitness" ] },
|
||||
"hair-and-beauty": { classification: :expense, aliases: [ "personal care" ] },
|
||||
"clothing-and-accessories": { classification: :expense, aliases: [ "shopping" ] },
|
||||
"gifts-and-charity": { classification: :expense, aliases: [ "gifts and donations" ] },
|
||||
"investments": { classification: :expense, aliases: [ "savings and investments" ] },
|
||||
"mobile-phone": { classification: :expense, aliases: [ "subscriptions" ] },
|
||||
"technology": { classification: :expense, aliases: [] },
|
||||
"life-admin": { classification: :expense, aliases: [] },
|
||||
"education-and-student-loans": { classification: :expense, aliases: [] },
|
||||
"family": { classification: :expense, aliases: [] },
|
||||
"news-magazines-and-books": { classification: :expense, aliases: [] }
|
||||
}
|
||||
},
|
||||
"transport": {
|
||||
classification: :expense,
|
||||
aliases: [ "transportation" ],
|
||||
detailed_categories: {
|
||||
"fuel": { classification: :expense, aliases: [ "transportation" ] },
|
||||
"public-transport": { classification: :expense, aliases: [ "transportation" ] },
|
||||
"taxis-and-share-cars": { classification: :expense, aliases: [ "transportation" ] },
|
||||
"parking": { classification: :expense, aliases: [ "transportation" ] },
|
||||
"toll-roads": { classification: :expense, aliases: [ "transportation" ] },
|
||||
"cycling": { classification: :expense, aliases: [ "transportation" ] },
|
||||
"car-insurance-and-maintenance": { classification: :expense, aliases: [ "transportation" ] },
|
||||
"car-repayments": { classification: :expense, aliases: [ "loan payments" ] }
|
||||
}
|
||||
}
|
||||
}.freeze
|
||||
end
|
||||
@@ -24,7 +24,8 @@ class UpAccount::Transactions::Processor
|
||||
up_account.raw_transactions_payload.each_with_index do |transaction_data, index|
|
||||
result = UpEntry::Processor.new(
|
||||
transaction_data,
|
||||
up_account: up_account
|
||||
up_account: up_account,
|
||||
category_matcher: category_matcher
|
||||
).process
|
||||
|
||||
if result.nil?
|
||||
@@ -57,6 +58,22 @@ class UpAccount::Transactions::Processor
|
||||
|
||||
private
|
||||
|
||||
# A single category matcher reused across this account's transactions, built from
|
||||
# the family's existing categories.
|
||||
def category_matcher
|
||||
@category_matcher ||= UpAccount::Transactions::CategoryMatcher.new(family_categories)
|
||||
end
|
||||
|
||||
# The family's existing categories. Importing transactions is intentionally
|
||||
# non-destructive with respect to the family's category structure: we do NOT
|
||||
# bootstrap Sure's defaults here. A family that has no categories (deliberately
|
||||
# cleared, or pre-onboarding) simply gets uncategorised transactions, and matching
|
||||
# resumes once the user sets up categories through the normal UI flow. Returns []
|
||||
# when the account isn't linked (each entry is skipped before the matcher is used).
|
||||
def family_categories
|
||||
@family_categories ||= up_account.current_account&.family&.categories&.to_a || []
|
||||
end
|
||||
|
||||
# Extract the Up transaction id from raw data, or "unknown".
|
||||
def transaction_id(transaction_data)
|
||||
transaction_data.try(:[], :id) || transaction_data.try(:[], "id") || "unknown"
|
||||
|
||||
@@ -33,9 +33,14 @@ class UpEntry::Processor
|
||||
end
|
||||
|
||||
# Build a processor for a single raw Up transaction tied to +up_account+.
|
||||
def initialize(up_transaction, up_account:)
|
||||
#
|
||||
# category_matcher (optional) maps Up's category slug onto one of the family's
|
||||
# existing Sure categories. It is injected by UpAccount::Transactions::Processor so a
|
||||
# single matcher (built once per account) is reused across the account's transactions.
|
||||
def initialize(up_transaction, up_account:, category_matcher: nil)
|
||||
@up_transaction = up_transaction
|
||||
@up_account = up_account
|
||||
@category_matcher = category_matcher
|
||||
end
|
||||
|
||||
# Import the transaction into the linked Sure account via the import adapter.
|
||||
@@ -53,6 +58,7 @@ class UpEntry::Processor
|
||||
date: date,
|
||||
name: name,
|
||||
source: "up",
|
||||
category_id: matched_category_id,
|
||||
kind: kind,
|
||||
merchant: merchant,
|
||||
notes: notes,
|
||||
@@ -99,6 +105,17 @@ class UpEntry::Processor
|
||||
data[:description].presence || I18n.t("transactions.unknown_name")
|
||||
end
|
||||
|
||||
# The id of the Sure category that Up's category slug maps to, or nil when no
|
||||
# matcher was injected, the transaction has no Up category (transfers/income), or
|
||||
# the slug has no confident equivalent among the family's categories. The import
|
||||
# adapter applies this via enrich_attribute, so it never overwrites a category the
|
||||
# user has set or locked.
|
||||
def matched_category_id
|
||||
return nil unless @category_matcher
|
||||
|
||||
@category_matcher.match(data[:category_id])&.id
|
||||
end
|
||||
|
||||
# The id of the other account in an internal money movement, if any (see
|
||||
# Provider::Up#flatten_transaction). Present for transfers between the user's own
|
||||
# accounts and for round-ups swept into a Saver; nil for ordinary income/expense.
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
require "test_helper"
|
||||
|
||||
class UpAccount::Transactions::CategoryMatcherTest < ActiveSupport::TestCase
|
||||
setup do
|
||||
@family = families(:empty)
|
||||
@family.categories.bootstrap! # creates Sure's default categories
|
||||
@matcher = UpAccount::Transactions::CategoryMatcher.new(@family.categories.to_a)
|
||||
end
|
||||
|
||||
test "maps high-confidence Up categories to default Sure categories" do
|
||||
expectations = {
|
||||
"groceries" => "Groceries",
|
||||
"takeaway" => "Food & Drink",
|
||||
"restaurants-and-cafes" => "Food & Drink",
|
||||
"tv-and-music" => "Subscriptions",
|
||||
"health-and-medical" => "Healthcare",
|
||||
"fitness-and-wellbeing" => "Sports & Fitness",
|
||||
"hair-and-beauty" => "Personal Care",
|
||||
"clothing-and-accessories" => "Shopping",
|
||||
"gifts-and-charity" => "Gifts & Donations",
|
||||
"rent-and-mortgage" => "Mortgage / Rent",
|
||||
"utilities" => "Utilities",
|
||||
"fuel" => "Transportation",
|
||||
"taxis-and-share-cars" => "Transportation",
|
||||
"home-maintenance-and-improvements" => "Home Improvement"
|
||||
}
|
||||
|
||||
expectations.each do |slug, expected_name|
|
||||
matched = @matcher.match(slug)
|
||||
assert_not_nil matched, "expected '#{slug}' to match a default category"
|
||||
assert_equal expected_name, matched.name, "'#{slug}' mapped to the wrong category"
|
||||
end
|
||||
end
|
||||
|
||||
test "resolves transport children via the parent alias too" do
|
||||
assert_equal "Transportation", @matcher.match("parking")&.name
|
||||
assert_equal "Transportation", @matcher.match("public-transport")&.name
|
||||
end
|
||||
|
||||
test "leaves Up-specific categories with no honest default uncategorised" do
|
||||
%w[
|
||||
booze adult tobacco-and-vaping lottery-and-gambling pubs-and-bars
|
||||
games-and-software technology life-admin pets
|
||||
].each do |slug|
|
||||
assert_nil @matcher.match(slug), "expected '#{slug}' to have no confident match"
|
||||
end
|
||||
end
|
||||
|
||||
test "returns nil for unknown or blank slugs" do
|
||||
assert_nil @matcher.match("not-a-real-up-category")
|
||||
assert_nil @matcher.match(nil)
|
||||
assert_nil @matcher.match("")
|
||||
end
|
||||
|
||||
test "does not match when the family has none of the target categories" do
|
||||
bare_family = families(:dylan_family)
|
||||
bare_family.categories.destroy_all
|
||||
matcher = UpAccount::Transactions::CategoryMatcher.new(bare_family.categories.reload.to_a)
|
||||
|
||||
assert_nil matcher.match("groceries"), "no Groceries category exists, so no match"
|
||||
end
|
||||
end
|
||||
@@ -0,0 +1,68 @@
|
||||
require "test_helper"
|
||||
|
||||
class UpAccount::Transactions::ProcessorTest < ActiveSupport::TestCase
|
||||
setup do
|
||||
@family = families(:empty)
|
||||
@up_item = UpItem.create!(family: @family, name: "Up", access_token: "tok")
|
||||
@up_account = UpAccount.create!(
|
||||
up_item: @up_item, name: "Spending", account_id: "acc_1", currency: "AUD"
|
||||
)
|
||||
@account = Account.create!(
|
||||
family: @family, name: "Spending",
|
||||
accountable: Depository.new(subtype: "checking"), balance: 100, currency: "AUD"
|
||||
)
|
||||
AccountProvider.create!(account: @account, provider: @up_account)
|
||||
end
|
||||
|
||||
# Importing must not create categories as a side effect: a family that has none
|
||||
# (deliberately cleared, or pre-onboarding) keeps none, and transactions stay
|
||||
# uncategorised until the user sets up categories themselves.
|
||||
test "does not bootstrap default categories during import" do
|
||||
assert_equal 0, @family.categories.count, "family starts with no categories"
|
||||
|
||||
@up_account.update!(raw_transactions_payload: [
|
||||
{
|
||||
"id" => "tx_1",
|
||||
"status" => "SETTLED",
|
||||
"description" => "Woolworths",
|
||||
"amount" => { "currencyCode" => "AUD", "value" => "-40.00", "valueInBaseUnits" => -4000 },
|
||||
"settledAt" => "2026-01-15T00:00:00+11:00",
|
||||
"createdAt" => "2026-01-15T00:00:00+11:00",
|
||||
"account_id" => "acc_1",
|
||||
"category_id" => "groceries"
|
||||
}
|
||||
])
|
||||
|
||||
result = UpAccount::Transactions::Processor.new(@up_account).process
|
||||
|
||||
assert result[:success]
|
||||
assert_equal 0, @family.categories.reload.count, "import must not create categories"
|
||||
|
||||
entry = @account.entries.find_by(external_id: "up_tx_1")
|
||||
assert_not_nil entry, "the transaction was still imported"
|
||||
assert_nil entry.transaction.category_id, "stays uncategorised when the family has no categories"
|
||||
end
|
||||
|
||||
# With the default categories present, the same Up category resolves and is applied.
|
||||
test "applies matched categories when the family already has them" do
|
||||
@family.categories.bootstrap!
|
||||
|
||||
@up_account.update!(raw_transactions_payload: [
|
||||
{
|
||||
"id" => "tx_2",
|
||||
"status" => "SETTLED",
|
||||
"description" => "Woolworths",
|
||||
"amount" => { "currencyCode" => "AUD", "value" => "-40.00", "valueInBaseUnits" => -4000 },
|
||||
"settledAt" => "2026-01-15T00:00:00+11:00",
|
||||
"createdAt" => "2026-01-15T00:00:00+11:00",
|
||||
"account_id" => "acc_1",
|
||||
"category_id" => "groceries"
|
||||
}
|
||||
])
|
||||
|
||||
UpAccount::Transactions::Processor.new(@up_account).process
|
||||
|
||||
entry = @account.entries.find_by(external_id: "up_tx_2")
|
||||
assert_equal "Groceries", entry.transaction.category&.name
|
||||
end
|
||||
end
|
||||
@@ -151,4 +151,50 @@ class UpEntry::ProcessorTest < ActiveSupport::TestCase
|
||||
assert_equal "standard", entry.entryable.kind
|
||||
assert_nil entry.entryable.extra.dig("up", "transfer_account_id")
|
||||
end
|
||||
|
||||
test "applies the matched Sure category when a category_matcher is provided" do
|
||||
@family.categories.bootstrap!
|
||||
matcher = UpAccount::Transactions::CategoryMatcher.new(@family.categories.to_a)
|
||||
|
||||
entry = UpEntry::Processor.new(
|
||||
{
|
||||
id: "tx_cat_1",
|
||||
account_id: "acc_123",
|
||||
status: "SETTLED",
|
||||
description: "Woolworths",
|
||||
amount: { currencyCode: "AUD", value: "-40.00", valueInBaseUnits: -4000 },
|
||||
settledAt: "2026-01-15T00:00:00+11:00",
|
||||
createdAt: "2026-01-15T00:00:00+11:00",
|
||||
category_id: "groceries"
|
||||
},
|
||||
up_account: @up_account,
|
||||
category_matcher: matcher
|
||||
).process
|
||||
|
||||
assert_equal "Groceries", entry.transaction.category&.name
|
||||
# The raw Up slug is still preserved in extra for reference.
|
||||
assert_equal "groceries", entry.transaction.extra.dig("up", "category_id")
|
||||
end
|
||||
|
||||
test "leaves the transaction uncategorised when the Up category has no confident match" do
|
||||
@family.categories.bootstrap!
|
||||
matcher = UpAccount::Transactions::CategoryMatcher.new(@family.categories.to_a)
|
||||
|
||||
entry = UpEntry::Processor.new(
|
||||
{
|
||||
id: "tx_cat_2",
|
||||
account_id: "acc_123",
|
||||
status: "SETTLED",
|
||||
description: "BWS",
|
||||
amount: { currencyCode: "AUD", value: "-25.00", valueInBaseUnits: -2500 },
|
||||
settledAt: "2026-01-15T00:00:00+11:00",
|
||||
createdAt: "2026-01-15T00:00:00+11:00",
|
||||
category_id: "booze"
|
||||
},
|
||||
up_account: @up_account,
|
||||
category_matcher: matcher
|
||||
).process
|
||||
|
||||
assert_nil entry.transaction.category_id
|
||||
end
|
||||
end
|
||||
|
||||
Reference in New Issue
Block a user