QT/sure
1
0
Fork 0
mirror of https://github.com/we-promise/sure.git synced 2026-05-29 23:39:03 +00:00
Code Issues Packages Projects Releases Wiki Activity

Full .ndjson import / reorganize UI with Financial Tools / Raw Data tabs (#1208)

Browse Source 
* Reorganize import UI with Financial Tools / Raw Data tabs

Split the flat list of import sources into two tabbed sections using
DS::Tabs: "Financial Tools" (Mint, Quicken/QIF, YNAB coming soon) and
"Raw Data" (transactions, investments, accounts, categories, rules,
documents). This prepares for adding more tool-specific importers
without cluttering the list.

https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3

* Fix import controller test to account for YNAB coming soon entry

The new YNAB "coming soon" disabled entry adds a 5th aria-disabled
element to the import dialog.

https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3

* Fix system tests to click Raw Data tab before selecting import type

Transaction, trade, and account imports are now under the Raw Data tab
and need an explicit tab click before the buttons are visible.

https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3

* feat: Add bulk import for NDJSON export files

Implements an import flow that accepts the full all.ndjson file from data exports,
allowing users to restore their complete data including:
- Accounts with accountable types
- Categories with parent relationships
- Tags and merchants
- Transactions with category, merchant, and tag references
- Trades with securities
- Valuations
- Budgets and budget categories
- Rules with conditions and actions (including compound conditions)

Key changes:
- Add BulkImport model extending Import base class
- Add Family::DataImporter to handle NDJSON parsing and import logic
- Update imports controller and views to support NDJSON workflow
- Skip configuration/mapping steps for structured NDJSON imports
- Add i18n translations for bulk import UI
- Add tests for BulkImport and DataImporter

* fix: Fix category import and test query issues

- Add default lucide_icon ("shapes") for categories when not provided
- Fix valuation test to use proper ActiveRecord joins syntax

* Linter errors

* fix: Add default color for tags when not provided in import

* fix: Add default kind for transactions when not provided in import

* Fix test

* Fix tests

* Fix remaining merge conflicts from PR 766 cherry-pick

Resolve conflict markers in test fixtures and clean up BulkImport
entry in new.html.erb to use the _import_option partial consistently.

https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3

* Import Sure `.ndjson`

* Remove `.ndjson` import from raw data

* Fix support for Sure "bulk" import from old branch

* Linter

* Fix CI test

* Fix more CI tests

* Fix tests

* Fix tests / move PDF import to first tab

* Remove redundant title

---------

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Juan José Mata
2026-03-23 14:27:41 +01:00
committed by GitHub
parent 998ea6f7c5
commit 2595885eb7
32 changed files with 1960 additions and 163 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
test/controllers/api/v1/categories_controller_test.rb
View File

@@ -176,7 +176,11 @@ class Api::V1::CategoriesControllerTest < ActionDispatch::IntegrationTest
end
test "should not return category from another family" do
other_family_category = categories(:one) # belongs to :empty family
other_family_category = families(:empty).categories.create!(
name: "Other Family Category",
color: "#FF0000",
classification_unused: "expense"
)
get "/api/v1/categories/#{other_family_category.id}", params: {}, headers: {
"Authorization" => "Bearer #{@access_token.token}"

8
test/controllers/imports_controller_test.rb
View File

@@ -32,10 +32,10 @@ class ImportsControllerTest < ActionDispatch::IntegrationTest
assert_select "button", text: "Import accounts"
assert_select "button", text: "Import transactions", count: 0
assert_select "button", text: "Import investments", count: 0
assert_select "button", text: "Import from Mint", count: 0
assert_select "button", text: "Import from Quicken (QIF)", count: 0
assert_select "span", text: "Import accounts first to unlock this option.", count: 4
assert_select "div[aria-disabled=true]", count: 4
assert_select "button", text: "Import from Mint", count: 1
assert_select "button", text: "Import from Quicken (QIF)", count: 1
assert_select "span", text: "Import accounts first to unlock this option.", count: 2
assert_select "div[aria-disabled=true]", count: 3
end
test "creates import" do

2
test/fixtures/categories.yml vendored
View File

@@ -1,6 +1,6 @@
one:
name: Test
family: empty
family: dylan_family
income:
name: Income

5
test/fixtures/imports.yml vendored
View File

@@ -45,3 +45,8 @@ pdf_with_rows:
category: "Income"
notes: ""
rows_count: 2
sure:
family: dylan_family
type: SureImport
status: pending

2
test/fixtures/merchants.yml vendored
View File

@@ -1,7 +1,7 @@
one:
type: FamilyMerchant
name: Test
family: empty
family: dylan_family
netflix:
type: FamilyMerchant

2
test/fixtures/tags.yml vendored
View File

@@ -8,4 +8,4 @@ two:
three:
name: Test
family: empty
family: dylan_family

574
test/models/family/data_importer_test.rb Normal file
View File

@@ -0,0 +1,574 @@
require "test_helper"
class Family::DataImporterTest < ActiveSupport::TestCase
setup do
@family = families(:empty)
end
test "imports accounts with accountable data" do
ndjson = build_ndjson([
{
type: "Account",
data: {
id: "old-account-1",
name: "Test Checking",
balance: "1500.00",
currency: "USD",
accountable_type: "Depository",
accountable: { subtype: "checking" }
}
}
])
importer = Family::DataImporter.new(@family, ndjson)
result = importer.import!
assert_equal 1, result[:accounts].count
account = result[:accounts].first
assert_equal "Test Checking", account.name
assert_equal 1500.0, account.balance.to_f
assert_equal "USD", account.currency
assert_equal "Depository", account.accountable_type
end
test "imports categories with parent relationships" do
ndjson = build_ndjson([
{
type: "Category",
data: {
id: "cat-parent",
name: "Shopping",
color: "#FF5733",
classification: "expense"
}
},
{
type: "Category",
data: {
id: "cat-child",
name: "Groceries",
color: "#33FF57",
classification: "expense",
parent_id: "cat-parent"
}
}
])
importer = Family::DataImporter.new(@family, ndjson)
importer.import!
parent = @family.categories.find_by(name: "Shopping")
child = @family.categories.find_by(name: "Groceries")
assert_not_nil parent
assert_not_nil child
assert_equal parent.id, child.parent_id
end
test "imports tags" do
ndjson = build_ndjson([
{
type: "Tag",
data: {
id: "tag-1",
name: "Important",
color: "#FF0000"
}
}
])
importer = Family::DataImporter.new(@family, ndjson)
importer.import!
tag = @family.tags.find_by(name: "Important")
assert_not_nil tag
assert_equal "#FF0000", tag.color
end
test "imports merchants" do
ndjson = build_ndjson([
{
type: "Merchant",
data: {
id: "merchant-1",
name: "Amazon"
}
}
])
importer = Family::DataImporter.new(@family, ndjson)
importer.import!
merchant = @family.merchants.find_by(name: "Amazon")
assert_not_nil merchant
end
test "imports transactions with references" do
ndjson = build_ndjson([
{
type: "Account",
data: {
id: "acct-1",
name: "Main Account",
balance: "5000",
currency: "USD",
accountable_type: "Depository"
}
},
{
type: "Category",
data: {
id: "cat-1",
name: "Food",
color: "#FF0000",
classification: "expense"
}
},
{
type: "Tag",
data: {
id: "tag-1",
name: "Essential"
}
},
{
type: "Transaction",
data: {
id: "txn-1",
account_id: "acct-1",
date: "2024-01-15",
amount: "-50.00",
name: "Grocery Store",
currency: "USD",
category_id: "cat-1",
tag_ids: [ "tag-1" ],
notes: "Weekly groceries"
}
}
])
importer = Family::DataImporter.new(@family, ndjson)
result = importer.import!
assert_equal 1, result[:entries].count
transaction = @family.transactions.first
assert_not_nil transaction
assert_equal "Grocery Store", transaction.entry.name
assert_equal -50.0, transaction.entry.amount.to_f
assert_equal "Food", transaction.category.name
assert_equal 1, transaction.tags.count
assert_equal "Essential", transaction.tags.first.name
assert_equal "Weekly groceries", transaction.entry.notes
end
test "imports trades with securities" do
ndjson = build_ndjson([
{
type: "Account",
data: {
id: "inv-acct-1",
name: "Investment Account",
balance: "10000",
currency: "USD",
accountable_type: "Investment"
}
},
{
type: "Trade",
data: {
id: "trade-1",
account_id: "inv-acct-1",
date: "2024-01-15",
ticker: "AAPL",
qty: "10",
price: "150.00",
amount: "-1500.00",
currency: "USD"
}
}
])
importer = Family::DataImporter.new(@family, ndjson)
result = importer.import!
# Account + Opening balance + Trade entry
assert_equal 1, result[:entries].count
trade = @family.trades.first
assert_not_nil trade
assert_equal "AAPL", trade.security.ticker
assert_equal 10.0, trade.qty.to_f
assert_equal 150.0, trade.price.to_f
end
test "imports valuations" do
ndjson = build_ndjson([
{
type: "Account",
data: {
id: "prop-acct-1",
name: "Property",
balance: "500000",
currency: "USD",
accountable_type: "Property"
}
},
{
type: "Valuation",
data: {
id: "val-1",
account_id: "prop-acct-1",
date: "2024-06-15",
amount: "520000",
name: "Updated valuation",
currency: "USD"
}
}
])
importer = Family::DataImporter.new(@family, ndjson)
result = importer.import!
assert_equal 1, result[:entries].count
account = @family.accounts.find_by(name: "Property")
valuation = account.valuations.joins(:entry).find_by(entries: { name: "Updated valuation" })
assert_not_nil valuation
assert_equal 520000.0, valuation.entry.amount.to_f
end
test "imports budgets" do
ndjson = build_ndjson([
{
type: "Budget",
data: {
id: "budget-1",
start_date: "2024-01-01",
end_date: "2024-01-31",
budgeted_spending: "3000.00",
expected_income: "5000.00",
currency: "USD"
}
}
])
importer = Family::DataImporter.new(@family, ndjson)
importer.import!
budget = @family.budgets.first
assert_not_nil budget
assert_equal Date.parse("2024-01-01"), budget.start_date
assert_equal Date.parse("2024-01-31"), budget.end_date
assert_equal 3000.0, budget.budgeted_spending.to_f
assert_equal 5000.0, budget.expected_income.to_f
end
test "imports budget_categories" do
ndjson = build_ndjson([
{
type: "Category",
data: {
id: "cat-groceries",
name: "Groceries",
color: "#00FF00",
classification: "expense"
}
},
{
type: "Budget",
data: {
id: "budget-1",
start_date: "2024-01-01",
end_date: "2024-01-31",
budgeted_spending: "3000.00",
expected_income: "5000.00",
currency: "USD"
}
},
{
type: "BudgetCategory",
data: {
id: "bc-1",
budget_id: "budget-1",
category_id: "cat-groceries",
budgeted_spending: "500.00",
currency: "USD"
}
}
])
importer = Family::DataImporter.new(@family, ndjson)
importer.import!
budget = @family.budgets.first
budget_category = budget.budget_categories.first
assert_not_nil budget_category
assert_equal "Groceries", budget_category.category.name
assert_equal 500.0, budget_category.budgeted_spending.to_f
end
test "imports rules with conditions and actions" do
ndjson = build_ndjson([
{
type: "Rule",
version: 1,
data: {
name: "Categorize Coffee",
resource_type: "transaction",
active: true,
conditions: [
{
condition_type: "transaction_name",
operator: "like",
value: "starbucks"
}
],
actions: [
{
action_type: "set_transaction_category",
value: "Coffee"
}
]
}
}
])
importer = Family::DataImporter.new(@family, ndjson)
importer.import!
rule = @family.rules.find_by(name: "Categorize Coffee")
assert_not_nil rule
assert rule.active
assert_equal "transaction", rule.resource_type
assert_equal 1, rule.conditions.count
condition = rule.conditions.first
assert_equal "transaction_name", condition.condition_type
assert_equal "like", condition.operator
assert_equal "starbucks", condition.value
assert_equal 1, rule.actions.count
action = rule.actions.first
assert_equal "set_transaction_category", action.action_type
# Category should be created
category = @family.categories.find_by(name: "Coffee")
assert_not_nil category
assert_equal category.id, action.value
end
test "imports rules with compound conditions" do
ndjson = build_ndjson([
{
type: "Rule",
version: 1,
data: {
name: "Compound Rule",
resource_type: "transaction",
active: true,
conditions: [
{
condition_type: "compound",
operator: "or",
sub_conditions: [
{
condition_type: "transaction_name",
operator: "like",
value: "walmart"
},
{
condition_type: "transaction_name",
operator: "like",
value: "target"
}
]
}
],
actions: [
{
action_type: "auto_categorize"
}
]
}
}
])
importer = Family::DataImporter.new(@family, ndjson)
importer.import!
rule = @family.rules.find_by(name: "Compound Rule")
assert_not_nil rule
parent_condition = rule.conditions.first
assert_equal "compound", parent_condition.condition_type
assert_equal "or", parent_condition.operator
assert_equal 2, parent_condition.sub_conditions.count
end
test "skips invalid records gracefully" do
ndjson = "not valid json\n" + build_ndjson([
{
type: "Account",
data: {
id: "valid-acct",
name: "Valid Account",
balance: "1000",
currency: "USD",
accountable_type: "Depository"
}
}
])
importer = Family::DataImporter.new(@family, ndjson)
result = importer.import!
assert_equal 1, result[:accounts].count
assert_equal "Valid Account", result[:accounts].first.name
end
test "skips unsupported record types" do
ndjson = build_ndjson([
{
type: "UnsupportedType",
data: { id: "unknown" }
},
{
type: "Account",
data: {
id: "valid-acct",
name: "Known Account",
balance: "1000",
currency: "USD",
accountable_type: "Depository"
}
}
])
importer = Family::DataImporter.new(@family, ndjson)
result = importer.import!
assert_equal 1, result[:accounts].count
end
test "full import scenario with all entity types" do
ndjson = build_ndjson([
# Account
{
type: "Account",
data: {
id: "acct-main",
name: "Main Checking",
balance: "5000",
currency: "USD",
accountable_type: "Depository"
}
},
# Category
{
type: "Category",
data: {
id: "cat-food",
name: "Food",
color: "#FF5733",
classification: "expense"
}
},
# Tag
{
type: "Tag",
data: {
id: "tag-weekly",
name: "Weekly"
}
},
# Merchant
{
type: "Merchant",
data: {
id: "merchant-1",
name: "Local Grocery"
}
},
# Transaction
{
type: "Transaction",
data: {
id: "txn-1",
account_id: "acct-main",
date: "2024-01-15",
amount: "-75.50",
name: "Weekly groceries",
currency: "USD",
category_id: "cat-food",
merchant_id: "merchant-1",
tag_ids: [ "tag-weekly" ]
}
},
# Budget
{
type: "Budget",
data: {
id: "budget-jan",
start_date: "2024-01-01",
end_date: "2024-01-31",
budgeted_spending: "2000",
expected_income: "4000",
currency: "USD"
}
},
# BudgetCategory
{
type: "BudgetCategory",
data: {
id: "bc-food",
budget_id: "budget-jan",
category_id: "cat-food",
budgeted_spending: "500",
currency: "USD"
}
},
# Rule
{
type: "Rule",
version: 1,
data: {
name: "Auto-tag groceries",
resource_type: "transaction",
active: true,
conditions: [
{ condition_type: "transaction_name", operator: "like", value: "grocery" }
],
actions: [
{ action_type: "set_transaction_tags", value: "Weekly" }
]
}
}
])
importer = Family::DataImporter.new(@family, ndjson)
result = importer.import!
# Verify all entities were created
assert_equal 1, result[:accounts].count
assert_equal 1, @family.categories.count
assert_equal 1, @family.tags.count
assert_equal 1, @family.merchants.count
assert_equal 1, @family.transactions.count
assert_equal 1, @family.budgets.count
assert_equal 1, @family.budget_categories.count
assert_equal 1, @family.rules.count
# Verify relationships
transaction = @family.transactions.first
assert_equal "Food", transaction.category.name
assert_equal "Local Grocery", transaction.merchant.name
assert_equal "Weekly", transaction.tags.first.name
end
private
def build_ndjson(records)
records.map(&:to_json).join("\n")
end
end

187
test/models/sure_import_test.rb Normal file
View File

@@ -0,0 +1,187 @@
require "test_helper"
class SureImportTest < ActiveSupport::TestCase
include ActiveJob::TestHelper
setup do
@family = families(:dylan_family)
@import = @family.imports.create!(type: "SureImport")
end
test "dry_run reflects attached ndjson content" do
ndjson = [
{ type: "Account", data: { id: "uuid-1", name: "Test", balance: "1000", currency: "USD", accountable_type: "Depository" } },
{ type: "Transaction", data: { id: "uuid-2" } }
].map(&:to_json).join("\n")
attach_ndjson(ndjson)
dry_run = @import.dry_run
assert_equal 1, dry_run[:accounts]
assert_equal 1, dry_run[:transactions]
end
test "publishable? is false when attached file has no supported records" do
ndjson = { type: "UnknownType", data: {} }.to_json
attach_ndjson(ndjson)
assert @import.uploaded?
assert_not @import.publishable?
end
test "column_keys required_column_keys and mapping_steps are empty" do
assert_equal [], @import.column_keys
assert_equal [], @import.required_column_keys
assert_equal [], @import.mapping_steps
end
test "max_row_count is higher than standard imports" do
assert_equal 100_000, @import.max_row_count
end
test "csv_template returns nil" do
assert_nil @import.csv_template
end
test "uploaded? returns false without ndjson attachment" do
assert_not @import.uploaded?
end
test "uploaded? returns true with valid ndjson attachment" do
attach_ndjson(build_ndjson([
{ type: "Account", data: { id: "uuid-1", name: "Test", balance: "1000", currency: "USD", accountable_type: "Depository" } }
]))
assert @import.uploaded?
end
test "uploaded? returns false with invalid ndjson attachment" do
attach_ndjson("not valid json")
assert_not @import.uploaded?
end
test "configured? and cleaned? follow uploaded?" do
attach_ndjson(build_ndjson([
{ type: "Account", data: { id: "uuid-1", name: "Test", balance: "1000", currency: "USD", accountable_type: "Depository" } }
]))
assert @import.configured?
assert @import.cleaned?
end
test "publishable? returns true when uploaded and valid" do
attach_ndjson(build_ndjson([
{ type: "Account", data: { id: "uuid-1", name: "Test", balance: "1000", currency: "USD", accountable_type: "Depository" } }
]))
assert @import.publishable?
end
test "dry_run returns counts by type" do
attach_ndjson(build_ndjson([
{ type: "Account", data: { id: "uuid-1" } },
{ type: "Account", data: { id: "uuid-2" } },
{ type: "Category", data: { id: "uuid-3" } },
{ type: "Transaction", data: { id: "uuid-4" } },
{ type: "Transaction", data: { id: "uuid-5" } },
{ type: "Transaction", data: { id: "uuid-6" } }
]))
dry_run = @import.dry_run
assert_equal 2, dry_run[:accounts]
assert_equal 1, dry_run[:categories]
assert_equal 3, dry_run[:transactions]
assert_equal 0, dry_run[:tags]
end
test "sync_ndjson_rows_count! sets total row count" do
attach_ndjson(build_ndjson([
{ type: "Account", data: { id: "uuid-1" } },
{ type: "Category", data: { id: "uuid-2" } },
{ type: "Transaction", data: { id: "uuid-3" } }
]))
@import.sync_ndjson_rows_count!
assert_equal 3, @import.rows_count
end
test "publishes import successfully" do
attach_ndjson(build_ndjson([
{ type: "Account", data: {
id: "uuid-1",
name: "Import Test Account",
balance: "1000.00",
currency: "USD",
accountable_type: "Depository",
accountable: { subtype: "checking" }
} }
]))
initial_account_count = @family.accounts.count
@import.publish
assert_equal "complete", @import.status
assert_equal initial_account_count + 1, @family.accounts.count
account = @family.accounts.find_by(name: "Import Test Account")
assert_not_nil account
assert_equal 1000.0, account.balance.to_f
assert_equal "USD", account.currency
assert_equal "Depository", account.accountable_type
end
test "import tracks created accounts for revert" do
attach_ndjson(build_ndjson([
{ type: "Account", data: {
id: "uuid-1",
name: "Revertable Account",
balance: "500.00",
currency: "USD",
accountable_type: "Depository"
} }
]))
@import.publish
assert_equal 1, @import.accounts.count
assert_equal "Revertable Account", @import.accounts.first.name
end
test "publishes later enqueues job" do
attach_ndjson(build_ndjson([
{ type: "Account", data: {
id: "uuid-1",
name: "Async Account",
balance: "100",
currency: "USD",
accountable_type: "Depository"
} }
]))
assert_enqueued_with job: ImportJob, args: [ @import ] do
@import.publish_later
end
assert_equal "importing", @import.status
end
private
def attach_ndjson(ndjson)
@import.ndjson_file.attach(
io: StringIO.new(ndjson),
filename: "all.ndjson",
content_type: "application/x-ndjson"
)
@import.sync_ndjson_rows_count!
end
def build_ndjson(records)
records.map(&:to_json).join("\n")
end
end

5
test/system/imports_test.rb
View File

@@ -13,6 +13,7 @@ class ImportsTest < ApplicationSystemTestCase
test "transaction import" do
visit new_import_path
click_on "Raw Data"
click_on "Import transactions"
within_testid("import-tabs") do
@@ -63,6 +64,7 @@ class ImportsTest < ApplicationSystemTestCase
test "trade import" do
visit new_import_path
click_on "Raw Data"
click_on "Import investments"
within_testid("import-tabs") do
@@ -105,6 +107,7 @@ class ImportsTest < ApplicationSystemTestCase
test "account import" do
visit new_import_path
click_on "Raw Data"
click_on "Import accounts"
within_testid("import-tabs") do
@@ -153,6 +156,8 @@ class ImportsTest < ApplicationSystemTestCase
test "mint import" do
visit new_import_path
# Pending CSV-style imports default the dialog to the Raw Data tab; Mint lives under Financial Tools.
click_on "Financial Tools"
click_on "Import from Mint"
within_testid("import-tabs") do