mirror of
https://github.com/we-promise/sure.git
synced 2026-04-19 03:54:08 +00:00
Full .ndjson import / reorganize UI with Financial Tools / Raw Data tabs (#1208)
* Reorganize import UI with Financial Tools / Raw Data tabs Split the flat list of import sources into two tabbed sections using DS::Tabs: "Financial Tools" (Mint, Quicken/QIF, YNAB coming soon) and "Raw Data" (transactions, investments, accounts, categories, rules, documents). This prepares for adding more tool-specific importers without cluttering the list. https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3 * Fix import controller test to account for YNAB coming soon entry The new YNAB "coming soon" disabled entry adds a 5th aria-disabled element to the import dialog. https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3 * Fix system tests to click Raw Data tab before selecting import type Transaction, trade, and account imports are now under the Raw Data tab and need an explicit tab click before the buttons are visible. https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3 * feat: Add bulk import for NDJSON export files Implements an import flow that accepts the full all.ndjson file from data exports, allowing users to restore their complete data including: - Accounts with accountable types - Categories with parent relationships - Tags and merchants - Transactions with category, merchant, and tag references - Trades with securities - Valuations - Budgets and budget categories - Rules with conditions and actions (including compound conditions) Key changes: - Add BulkImport model extending Import base class - Add Family::DataImporter to handle NDJSON parsing and import logic - Update imports controller and views to support NDJSON workflow - Skip configuration/mapping steps for structured NDJSON imports - Add i18n translations for bulk import UI - Add tests for BulkImport and DataImporter * fix: Fix category import and test query issues - Add default lucide_icon ("shapes") for categories when not provided - Fix valuation test to use proper ActiveRecord joins syntax * Linter errors * fix: Add default color for tags when not provided in import * fix: Add default kind for transactions when not provided in import * Fix test * Fix tests * Fix remaining merge conflicts from PR 766 cherry-pick Resolve conflict markers in test fixtures and clean up BulkImport entry in new.html.erb to use the _import_option partial consistently. https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3 * Import Sure `.ndjson` * Remove `.ndjson` import from raw data * Fix support for Sure "bulk" import from old branch * Linter * Fix CI test * Fix more CI tests * Fix tests * Fix tests / move PDF import to first tab * Remove redundant title --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -49,6 +49,11 @@ class ImportsController < ApplicationController
|
||||
return
|
||||
end
|
||||
|
||||
if file.present? && sure_import_request?
|
||||
create_sure_import(file)
|
||||
return
|
||||
end
|
||||
|
||||
# Handle PDF file uploads - process with AI
|
||||
if file.present? && Import::ALLOWED_PDF_MIME_TYPES.include?(file.content_type)
|
||||
unless valid_pdf_file?(file)
|
||||
@@ -85,6 +90,7 @@ class ImportsController < ApplicationController
|
||||
# Stream reading is not fully applicable here as we store the raw string in the DB,
|
||||
# but we have validated size beforehand to prevent memory exhaustion from massive files.
|
||||
import.update!(raw_file_str: file.read)
|
||||
|
||||
redirect_to import_configuration_path(import), notice: t("imports.create.csv_uploaded")
|
||||
else
|
||||
redirect_to import_upload_path(import)
|
||||
@@ -200,6 +206,40 @@ class ImportsController < ApplicationController
|
||||
params.dig(:import, :type) == "DocumentImport"
|
||||
end
|
||||
|
||||
def sure_import_request?
|
||||
params.dig(:import, :type) == "SureImport"
|
||||
end
|
||||
|
||||
def create_sure_import(file)
|
||||
if file.size > SureImport::MAX_NDJSON_SIZE
|
||||
redirect_to new_import_path, alert: t("imports.create.file_too_large", max_size: SureImport::MAX_NDJSON_SIZE / 1.megabyte)
|
||||
return
|
||||
end
|
||||
|
||||
ext = File.extname(file.original_filename.to_s).downcase
|
||||
unless ext.in?(%w[.ndjson .json])
|
||||
redirect_to new_import_path, alert: t("imports.create.invalid_ndjson_file_type")
|
||||
return
|
||||
end
|
||||
|
||||
content = file.read
|
||||
file.rewind
|
||||
unless SureImport.valid_ndjson_first_line?(content)
|
||||
redirect_to new_import_path, alert: t("imports.create.invalid_ndjson_file_type")
|
||||
return
|
||||
end
|
||||
|
||||
import = Current.family.imports.create!(type: "SureImport")
|
||||
import.ndjson_file.attach(
|
||||
io: StringIO.new(content),
|
||||
filename: file.original_filename,
|
||||
content_type: file.content_type
|
||||
)
|
||||
import.sync_ndjson_rows_count!
|
||||
|
||||
redirect_to import_path(import), notice: t("imports.create.ndjson_uploaded")
|
||||
end
|
||||
|
||||
def valid_pdf_file?(file)
|
||||
header = file.read(5)
|
||||
file.rewind
|
||||
|
||||
Reference in New Issue
Block a user