Files
sure/app/controllers/import/uploads_controller.rb
Juan José Mata 2595885eb7 Full .ndjson import / reorganize UI with Financial Tools / Raw Data tabs (#1208)
* Reorganize import UI with Financial Tools / Raw Data tabs

Split the flat list of import sources into two tabbed sections using
DS::Tabs: "Financial Tools" (Mint, Quicken/QIF, YNAB coming soon) and
"Raw Data" (transactions, investments, accounts, categories, rules,
documents). This prepares for adding more tool-specific importers
without cluttering the list.

https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3

* Fix import controller test to account for YNAB coming soon entry

The new YNAB "coming soon" disabled entry adds a 5th aria-disabled
element to the import dialog.

https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3

* Fix system tests to click Raw Data tab before selecting import type

Transaction, trade, and account imports are now under the Raw Data tab
and need an explicit tab click before the buttons are visible.

https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3

* feat: Add bulk import for NDJSON export files

Implements an import flow that accepts the full all.ndjson file from data exports,
allowing users to restore their complete data including:
- Accounts with accountable types
- Categories with parent relationships
- Tags and merchants
- Transactions with category, merchant, and tag references
- Trades with securities
- Valuations
- Budgets and budget categories
- Rules with conditions and actions (including compound conditions)

Key changes:
- Add BulkImport model extending Import base class
- Add Family::DataImporter to handle NDJSON parsing and import logic
- Update imports controller and views to support NDJSON workflow
- Skip configuration/mapping steps for structured NDJSON imports
- Add i18n translations for bulk import UI
- Add tests for BulkImport and DataImporter

* fix: Fix category import and test query issues

- Add default lucide_icon ("shapes") for categories when not provided
- Fix valuation test to use proper ActiveRecord joins syntax

* Linter errors

* fix: Add default color for tags when not provided in import

* fix: Add default kind for transactions when not provided in import

* Fix test

* Fix tests

* Fix remaining merge conflicts from PR 766 cherry-pick

Resolve conflict markers in test fixtures and clean up BulkImport
entry in new.html.erb to use the _import_option partial consistently.

https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3

* Import Sure `.ndjson`

* Remove `.ndjson` import from raw data

* Fix support for Sure "bulk" import from old branch

* Linter

* Fix CI test

* Fix more CI tests

* Fix tests

* Fix tests / move PDF import to first tab

* Remove redundant title

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-03-23 14:27:41 +01:00

140 lines
4.3 KiB
Ruby

class Import::UploadsController < ApplicationController
layout "imports"
before_action :set_import
def show
end
def sample_csv
send_data @import.csv_template.to_csv,
filename: "#{@import.type.underscore.split('_').first}_sample.csv",
type: "text/csv",
disposition: "attachment"
end
def update
if @import.is_a?(QifImport)
handle_qif_upload
elsif @import.is_a?(SureImport)
update_sure_import_upload
elsif csv_valid?(csv_str)
@import.account = Current.family.accounts.find_by(id: import_account_id)
@import.assign_attributes(raw_file_str: csv_str, col_sep: upload_params[:col_sep])
@import.save!(validate: false)
redirect_to import_configuration_path(@import, template_hint: true), notice: "CSV uploaded successfully."
else
update_csv_import
end
end
private
def update_csv_import
if csv_valid?(csv_str)
@import.account = Current.family.accounts.find_by(id: params.dig(:import, :account_id))
@import.assign_attributes(raw_file_str: csv_str, col_sep: upload_params[:col_sep])
@import.save!(validate: false)
redirect_to import_configuration_path(@import, template_hint: true), notice: t("imports.create.csv_uploaded")
else
flash.now[:alert] = t("import.uploads.show.csv_invalid", default: "Must be valid CSV with headers and at least one row of data")
render :show, status: :unprocessable_entity
end
end
def update_sure_import_upload
uploaded = upload_params[:ndjson_file]
unless uploaded.present?
flash.now[:alert] = t("import.uploads.sure_import.ndjson_invalid", default: "Must be valid NDJSON with at least one record")
render :show, status: :unprocessable_entity
return
end
if uploaded.size > SureImport::MAX_NDJSON_SIZE
flash.now[:alert] = t("imports.create.file_too_large", max_size: SureImport::MAX_NDJSON_SIZE / 1.megabyte)
render :show, status: :unprocessable_entity
return
end
content = uploaded.read
uploaded.rewind
if ndjson_valid?(content)
uploaded.rewind
@import.ndjson_file.attach(uploaded)
@import.sync_ndjson_rows_count!
redirect_to import_path(@import), notice: t("imports.create.ndjson_uploaded")
else
flash.now[:alert] = t("import.uploads.sure_import.ndjson_invalid", default: "Must be valid NDJSON with at least one record")
render :show, status: :unprocessable_entity
end
end
def set_import
@import = Current.family.imports.find(params[:import_id])
end
def handle_qif_upload
unless QifParser.valid?(csv_str)
flash.now[:alert] = "Must be a valid QIF file"
render :show, status: :unprocessable_entity and return
end
unless import_account_id.present?
flash.now[:alert] = "Please select an account for the QIF import"
render :show, status: :unprocessable_entity and return
end
ActiveRecord::Base.transaction do
@import.account = Current.family.accounts.find(import_account_id)
@import.raw_file_str = QifParser.normalize_encoding(csv_str)
@import.save!(validate: false)
@import.generate_rows_from_csv
@import.sync_mappings
end
redirect_to import_qif_category_selection_path(@import), notice: "QIF file uploaded successfully."
end
def csv_str
@csv_str ||= upload_params[:import_file]&.read || upload_params[:raw_file_str]
end
def csv_valid?(str)
begin
csv = Import.parse_csv_str(str, col_sep: upload_params[:col_sep])
return false if csv.headers.empty?
return false if csv.count == 0
true
rescue CSV::MalformedCSVError
false
end
end
def ndjson_valid?(str)
return false if str.blank?
# Check at least first line is valid NDJSON
first_line = str.lines.first&.strip
return false if first_line.blank?
begin
record = JSON.parse(first_line)
record.key?("type") && record.key?("data")
rescue JSON::ParserError
false
end
end
def upload_params
params.require(:import).permit(:raw_file_str, :import_file, :ndjson_file, :col_sep)
end
def import_account_id
params.require(:import).permit(:account_id)[:account_id]
end
end