mirror of
https://github.com/we-promise/sure.git
synced 2026-04-10 07:44:48 +00:00
* Reorganize import UI with Financial Tools / Raw Data tabs Split the flat list of import sources into two tabbed sections using DS::Tabs: "Financial Tools" (Mint, Quicken/QIF, YNAB coming soon) and "Raw Data" (transactions, investments, accounts, categories, rules, documents). This prepares for adding more tool-specific importers without cluttering the list. https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3 * Fix import controller test to account for YNAB coming soon entry The new YNAB "coming soon" disabled entry adds a 5th aria-disabled element to the import dialog. https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3 * Fix system tests to click Raw Data tab before selecting import type Transaction, trade, and account imports are now under the Raw Data tab and need an explicit tab click before the buttons are visible. https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3 * feat: Add bulk import for NDJSON export files Implements an import flow that accepts the full all.ndjson file from data exports, allowing users to restore their complete data including: - Accounts with accountable types - Categories with parent relationships - Tags and merchants - Transactions with category, merchant, and tag references - Trades with securities - Valuations - Budgets and budget categories - Rules with conditions and actions (including compound conditions) Key changes: - Add BulkImport model extending Import base class - Add Family::DataImporter to handle NDJSON parsing and import logic - Update imports controller and views to support NDJSON workflow - Skip configuration/mapping steps for structured NDJSON imports - Add i18n translations for bulk import UI - Add tests for BulkImport and DataImporter * fix: Fix category import and test query issues - Add default lucide_icon ("shapes") for categories when not provided - Fix valuation test to use proper ActiveRecord joins syntax * Linter errors * fix: Add default color for tags when not provided in import * fix: Add default kind for transactions when not provided in import * Fix test * Fix tests * Fix remaining merge conflicts from PR 766 cherry-pick Resolve conflict markers in test fixtures and clean up BulkImport entry in new.html.erb to use the _import_option partial consistently. https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3 * Import Sure `.ndjson` * Remove `.ndjson` import from raw data * Fix support for Sure "bulk" import from old branch * Linter * Fix CI test * Fix more CI tests * Fix tests * Fix tests / move PDF import to first tab * Remove redundant title --------- Co-authored-by: Claude <noreply@anthropic.com>
140 lines
4.3 KiB
Ruby
140 lines
4.3 KiB
Ruby
class Import::UploadsController < ApplicationController
|
|
layout "imports"
|
|
|
|
before_action :set_import
|
|
|
|
def show
|
|
end
|
|
|
|
def sample_csv
|
|
send_data @import.csv_template.to_csv,
|
|
filename: "#{@import.type.underscore.split('_').first}_sample.csv",
|
|
type: "text/csv",
|
|
disposition: "attachment"
|
|
end
|
|
|
|
def update
|
|
if @import.is_a?(QifImport)
|
|
handle_qif_upload
|
|
elsif @import.is_a?(SureImport)
|
|
update_sure_import_upload
|
|
elsif csv_valid?(csv_str)
|
|
@import.account = Current.family.accounts.find_by(id: import_account_id)
|
|
@import.assign_attributes(raw_file_str: csv_str, col_sep: upload_params[:col_sep])
|
|
@import.save!(validate: false)
|
|
|
|
redirect_to import_configuration_path(@import, template_hint: true), notice: "CSV uploaded successfully."
|
|
else
|
|
update_csv_import
|
|
end
|
|
end
|
|
|
|
private
|
|
def update_csv_import
|
|
if csv_valid?(csv_str)
|
|
@import.account = Current.family.accounts.find_by(id: params.dig(:import, :account_id))
|
|
@import.assign_attributes(raw_file_str: csv_str, col_sep: upload_params[:col_sep])
|
|
@import.save!(validate: false)
|
|
|
|
redirect_to import_configuration_path(@import, template_hint: true), notice: t("imports.create.csv_uploaded")
|
|
else
|
|
flash.now[:alert] = t("import.uploads.show.csv_invalid", default: "Must be valid CSV with headers and at least one row of data")
|
|
|
|
render :show, status: :unprocessable_entity
|
|
end
|
|
end
|
|
|
|
def update_sure_import_upload
|
|
uploaded = upload_params[:ndjson_file]
|
|
unless uploaded.present?
|
|
flash.now[:alert] = t("import.uploads.sure_import.ndjson_invalid", default: "Must be valid NDJSON with at least one record")
|
|
render :show, status: :unprocessable_entity
|
|
return
|
|
end
|
|
|
|
if uploaded.size > SureImport::MAX_NDJSON_SIZE
|
|
flash.now[:alert] = t("imports.create.file_too_large", max_size: SureImport::MAX_NDJSON_SIZE / 1.megabyte)
|
|
render :show, status: :unprocessable_entity
|
|
return
|
|
end
|
|
|
|
content = uploaded.read
|
|
uploaded.rewind
|
|
|
|
if ndjson_valid?(content)
|
|
uploaded.rewind
|
|
@import.ndjson_file.attach(uploaded)
|
|
@import.sync_ndjson_rows_count!
|
|
redirect_to import_path(@import), notice: t("imports.create.ndjson_uploaded")
|
|
else
|
|
flash.now[:alert] = t("import.uploads.sure_import.ndjson_invalid", default: "Must be valid NDJSON with at least one record")
|
|
|
|
render :show, status: :unprocessable_entity
|
|
end
|
|
end
|
|
|
|
def set_import
|
|
@import = Current.family.imports.find(params[:import_id])
|
|
end
|
|
|
|
def handle_qif_upload
|
|
unless QifParser.valid?(csv_str)
|
|
flash.now[:alert] = "Must be a valid QIF file"
|
|
render :show, status: :unprocessable_entity and return
|
|
end
|
|
|
|
unless import_account_id.present?
|
|
flash.now[:alert] = "Please select an account for the QIF import"
|
|
render :show, status: :unprocessable_entity and return
|
|
end
|
|
|
|
ActiveRecord::Base.transaction do
|
|
@import.account = Current.family.accounts.find(import_account_id)
|
|
@import.raw_file_str = QifParser.normalize_encoding(csv_str)
|
|
@import.save!(validate: false)
|
|
@import.generate_rows_from_csv
|
|
@import.sync_mappings
|
|
end
|
|
|
|
redirect_to import_qif_category_selection_path(@import), notice: "QIF file uploaded successfully."
|
|
end
|
|
|
|
def csv_str
|
|
@csv_str ||= upload_params[:import_file]&.read || upload_params[:raw_file_str]
|
|
end
|
|
|
|
def csv_valid?(str)
|
|
begin
|
|
csv = Import.parse_csv_str(str, col_sep: upload_params[:col_sep])
|
|
return false if csv.headers.empty?
|
|
return false if csv.count == 0
|
|
true
|
|
rescue CSV::MalformedCSVError
|
|
false
|
|
end
|
|
end
|
|
|
|
def ndjson_valid?(str)
|
|
return false if str.blank?
|
|
|
|
# Check at least first line is valid NDJSON
|
|
first_line = str.lines.first&.strip
|
|
return false if first_line.blank?
|
|
|
|
begin
|
|
record = JSON.parse(first_line)
|
|
record.key?("type") && record.key?("data")
|
|
rescue JSON::ParserError
|
|
false
|
|
end
|
|
end
|
|
|
|
def upload_params
|
|
params.require(:import).permit(:raw_file_str, :import_file, :ndjson_file, :col_sep)
|
|
end
|
|
|
|
def import_account_id
|
|
params.require(:import).permit(:account_id)[:account_id]
|
|
end
|
|
end
|