Full .ndjson import / reorganize UI with Financial Tools / Raw Data tabs (#1208)

* Reorganize import UI with Financial Tools / Raw Data tabs

Split the flat list of import sources into two tabbed sections using
DS::Tabs: "Financial Tools" (Mint, Quicken/QIF, YNAB coming soon) and
"Raw Data" (transactions, investments, accounts, categories, rules,
documents). This prepares for adding more tool-specific importers
without cluttering the list.

https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3

* Fix import controller test to account for YNAB coming soon entry

The new YNAB "coming soon" disabled entry adds a 5th aria-disabled
element to the import dialog.

https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3

* Fix system tests to click Raw Data tab before selecting import type

Transaction, trade, and account imports are now under the Raw Data tab
and need an explicit tab click before the buttons are visible.

https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3

* feat: Add bulk import for NDJSON export files

Implements an import flow that accepts the full all.ndjson file from data exports,
allowing users to restore their complete data including:
- Accounts with accountable types
- Categories with parent relationships
- Tags and merchants
- Transactions with category, merchant, and tag references
- Trades with securities
- Valuations
- Budgets and budget categories
- Rules with conditions and actions (including compound conditions)

Key changes:
- Add BulkImport model extending Import base class
- Add Family::DataImporter to handle NDJSON parsing and import logic
- Update imports controller and views to support NDJSON workflow
- Skip configuration/mapping steps for structured NDJSON imports
- Add i18n translations for bulk import UI
- Add tests for BulkImport and DataImporter

* fix: Fix category import and test query issues

- Add default lucide_icon ("shapes") for categories when not provided
- Fix valuation test to use proper ActiveRecord joins syntax

* Linter errors

* fix: Add default color for tags when not provided in import

* fix: Add default kind for transactions when not provided in import

* Fix test

* Fix tests

* Fix remaining merge conflicts from PR 766 cherry-pick

Resolve conflict markers in test fixtures and clean up BulkImport
entry in new.html.erb to use the _import_option partial consistently.

https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3

* Import Sure `.ndjson`

* Remove `.ndjson` import from raw data

* Fix support for Sure "bulk" import from old branch

* Linter

* Fix CI test

* Fix more CI tests

* Fix tests

* Fix tests / move PDF import to first tab

* Remove redundant title

---------

Co-authored-by: Claude <noreply@anthropic.com>

app/controllers/import/uploads_controller.rb

@@ -16,6 +16,8 @@ class Import::UploadsController < ApplicationController
   def update
     if @import.is_a?(QifImport)
       handle_qif_upload
+    elsif @import.is_a?(SureImport)
+      update_sure_import_upload
     elsif csv_valid?(csv_str)
       @import.account = Current.family.accounts.find_by(id: import_account_id)
       @import.assign_attributes(raw_file_str: csv_str, col_sep: upload_params[:col_sep])
@@ -23,13 +25,54 @@ class Import::UploadsController < ApplicationController
 
       redirect_to import_configuration_path(@import, template_hint: true), notice: "CSV uploaded successfully."
     else
-      flash.now[:alert] = "Must be valid CSV with headers and at least one row of data"
-
-      render :show, status: :unprocessable_entity
+      update_csv_import
     end
   end
 
   private
+    def update_csv_import
+      if csv_valid?(csv_str)
+        @import.account = Current.family.accounts.find_by(id: params.dig(:import, :account_id))
+        @import.assign_attributes(raw_file_str: csv_str, col_sep: upload_params[:col_sep])
+        @import.save!(validate: false)
+
+        redirect_to import_configuration_path(@import, template_hint: true), notice: t("imports.create.csv_uploaded")
+      else
+        flash.now[:alert] = t("import.uploads.show.csv_invalid", default: "Must be valid CSV with headers and at least one row of data")
+
+        render :show, status: :unprocessable_entity
+      end
+    end
+
+    def update_sure_import_upload
+      uploaded = upload_params[:ndjson_file]
+      unless uploaded.present?
+        flash.now[:alert] = t("import.uploads.sure_import.ndjson_invalid", default: "Must be valid NDJSON with at least one record")
+        render :show, status: :unprocessable_entity
+        return
+      end
+
+      if uploaded.size > SureImport::MAX_NDJSON_SIZE
+        flash.now[:alert] = t("imports.create.file_too_large", max_size: SureImport::MAX_NDJSON_SIZE / 1.megabyte)
+        render :show, status: :unprocessable_entity
+        return
+      end
+
+      content = uploaded.read
+      uploaded.rewind
+
+      if ndjson_valid?(content)
+        uploaded.rewind
+        @import.ndjson_file.attach(uploaded)
+        @import.sync_ndjson_rows_count!
+        redirect_to import_path(@import), notice: t("imports.create.ndjson_uploaded")
+      else
+        flash.now[:alert] = t("import.uploads.sure_import.ndjson_invalid", default: "Must be valid NDJSON with at least one record")
+
+        render :show, status: :unprocessable_entity
+      end
+    end
+
     def set_import
       @import = Current.family.imports.find(params[:import_id])
     end
@@ -71,8 +114,23 @@ class Import::UploadsController < ApplicationController
       end
     end
 
+    def ndjson_valid?(str)
+      return false if str.blank?
+
+      # Check at least first line is valid NDJSON
+      first_line = str.lines.first&.strip
+      return false if first_line.blank?
+
+      begin
+        record = JSON.parse(first_line)
+        record.key?("type") && record.key?("data")
+      rescue JSON::ParserError
+        false
+      end
+    end
+
     def upload_params
-      params.require(:import).permit(:raw_file_str, :import_file, :col_sep)
+      params.require(:import).permit(:raw_file_str, :import_file, :ndjson_file, :col_sep)
     end
 
     def import_account_id

app/controllers/imports_controller.rb

@@ -49,6 +49,11 @@ class ImportsController < ApplicationController
       return
     end
 
+    if file.present? && sure_import_request?
+      create_sure_import(file)
+      return
+    end
+
     # Handle PDF file uploads - process with AI
     if file.present? && Import::ALLOWED_PDF_MIME_TYPES.include?(file.content_type)
       unless valid_pdf_file?(file)
@@ -85,6 +90,7 @@ class ImportsController < ApplicationController
       # Stream reading is not fully applicable here as we store the raw string in the DB,
       # but we have validated size beforehand to prevent memory exhaustion from massive files.
       import.update!(raw_file_str: file.read)
+
       redirect_to import_configuration_path(import), notice: t("imports.create.csv_uploaded")
     else
       redirect_to import_upload_path(import)
@@ -200,6 +206,40 @@ class ImportsController < ApplicationController
       params.dig(:import, :type) == "DocumentImport"
     end
 
+    def sure_import_request?
+      params.dig(:import, :type) == "SureImport"
+    end
+
+    def create_sure_import(file)
+      if file.size > SureImport::MAX_NDJSON_SIZE
+        redirect_to new_import_path, alert: t("imports.create.file_too_large", max_size: SureImport::MAX_NDJSON_SIZE / 1.megabyte)
+        return
+      end
+
+      ext = File.extname(file.original_filename.to_s).downcase
+      unless ext.in?(%w[.ndjson .json])
+        redirect_to new_import_path, alert: t("imports.create.invalid_ndjson_file_type")
+        return
+      end
+
+      content = file.read
+      file.rewind
+      unless SureImport.valid_ndjson_first_line?(content)
+        redirect_to new_import_path, alert: t("imports.create.invalid_ndjson_file_type")
+        return
+      end
+
+      import = Current.family.imports.create!(type: "SureImport")
+      import.ndjson_file.attach(
+        io: StringIO.new(content),
+        filename: file.original_filename,
+        content_type: file.content_type
+      )
+      import.sync_ndjson_rows_count!
+
+      redirect_to import_path(import), notice: t("imports.create.ndjson_uploaded")
+    end
+
     def valid_pdf_file?(file)
       header = file.read(5)
       file.rewind