Files
sure/app/controllers/imports_controller.rb
MkDev11 0afdb1d0fd Feature/pdf import transaction rows (#846)
* Add import row generation from PDF extracted data

- Add generate_rows_from_extracted_data method to PdfImport
- Add import! method to create transactions from PDF rows
- Update ProcessPdfJob to generate rows after extraction
- Update configured?, cleaned?, publishable? for PDF workflow
- Add column_keys, required_column_keys, mapping_steps
- Set bank statements to pending status for user review
- Add tests for new functionality

Closes #844

* Add tests for BankStatementExtractor

- Test transaction extraction from PDF content
- Test deduplication across chunk boundaries
- Test amount normalization for various formats
- Test graceful handling of malformed JSON responses
- Test error handling for empty/nil PDF content

* Fix supports_pdf_processing? to validate effective model

The validation was always checking @default_model, but process_pdf
allows overriding the model via parameter. This could cause a
vision-capable override model to be rejected, or a non-vision-capable
override to pass validation only to fail during processing.

Changes:
- supports_pdf_processing? now accepts optional model parameter
- process_pdf passes effective model to validation
- Raise Provider::Openai::Error inside with_provider_response for
  consistent error handling

Addresses review feedback from PR#808

* Fix insert_all! bug: explicitly set import_id

Rails insert_all! on associations does NOT auto-set the foreign key.
Added import_id explicitly and use Import::Row.insert_all! directly.
Also reload rows before counting to ensure accurate count.

* Fix pending status showing as processing for bank statements with rows

When bank statement PDF imports have extracted rows, show a 'Ready for Review'
screen with a link to the confirm path instead of the 'Processing' spinner.

This addresses the PR feedback that users couldn't reach the review flow even
though rows were created.

* Gate publishable? on account.present? to prevent import failure

PDF imports are created without an account, and import! raises if account
is missing. This prevents users from hitting publish and having the job fail.

* Wrap generate_rows_from_extracted_data in transaction for atomicity

- Clear rows and reset count even when no transactions extracted
- Use transaction block to prevent partial updates on failure
- Use mapped_rows.size instead of reload for count

* Localize transactions count string with i18n helper

* Add AccountMapping step for PDF imports when account is nil

PDF imports need account selection before publishing. This adds
Import::AccountMapping to mapping_steps when account is nil,
matching the behavior of TransactionImport and TradeImport.

Addresses PR#846 feedback about account selection for PDF imports.

* Only include CategoryMapping when rows have non-empty categories

PDF extraction doesn't extract categories from bank statements,
so the CategoryMapping step would show empty. Now we only include
CategoryMapping if rows actually have non-empty category values.

This prevents showing an empty mapping step for PDF imports.

* Fix PDF import UI flow and account selection

- Add direct account selection in PDF import UI instead of AccountMapping
- AccountMapping designed for CSV imports with multiple account values
- PDF imports need single account for all transactions
- Add update action and route for imports controller
- Fix controller to handle pdf_import param format from form_with
- Show Publish button when import is publishable (account set)
- Fix stepper nav: Upload/Configure/Clean non-clickable for PDF imports
- Redirect PDF imports from configuration step (auto-configured)
- Improve AI prompt to recognize M-PESA/mobile money as bank statements
- Fix migration ordering for import_rows table columns

* Add guard for invalid account_id in imports#update

Prevents silently clearing account when invalid ID is passed.
Returns error message instead of confusing 'Account saved' notice.

* Localize step names in import nav and add account guard

- Use t() helper for all step names (Upload, Configure, Clean, Map, Confirm)
- Add guard for invalid account_id in imports#update
- Prevents silently clearing account when invalid ID is passed

* Make category column migrations idempotent

Check if columns exist before adding to prevent duplicate column
errors when migrations are re-run with new timestamps.

* Add match_path for PDF import step highlighting

Fixes step detection when path is nil by using separate match_path
for current step highlighting while keeping links disabled.

* Rename category migrations and update to Rails 7.2

- Rename class to EnsureCategoryFieldsOnImportRows to avoid conflicts
- Rename class to EnsureCategoryIconOnImportRows
- Update migration version from 7.1 to 7.2 per guidelines
- Rename files to match class names
- Add match_path for PDF import step highlighting

* Use primary (black) style for Create Account and Save buttons

* Remove match_path from auto-completed PDF steps

Only step 4 (Confirm) needs match_path for active-step detection.
Steps 1-3 are purely informational and always complete.

* Add fallback for document type translation

Handles nil or unexpected document_type values gracefully.
Also removes match_path from auto-completed PDF steps.

* Use index-based step number for mobile indicator

Fixes 'Step 5 of 4' issue when Map step is dynamically removed.

* Fix hostings_controller_test: use blank? instead of nil

Setting returns empty string not nil for unset values.

* Localize step progress label and use design token

* Fix button styling: use design system Tailwind classes

btn--primary and btn--secondary CSS classes don't exist.
Use actual design system classes from DS::Buttonish.

* Fix CRLF line endings in tags_controller_test.rb

---------

Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
2026-02-02 16:27:02 +01:00

146 lines
4.5 KiB
Ruby

class ImportsController < ApplicationController
include SettingsHelper
before_action :set_import, only: %i[show update publish destroy revert apply_template]
def update
# Handle both pdf_import[account_id] and import[account_id] param formats
account_id = params.dig(:pdf_import, :account_id) || params.dig(:import, :account_id)
if account_id.present?
account = Current.family.accounts.find_by(id: account_id)
unless account
redirect_back_or_to import_path(@import), alert: t("imports.update.invalid_account", default: "Account not found.")
return
end
@import.update!(account: account)
end
redirect_to import_path(@import), notice: t("imports.update.account_saved", default: "Account saved.")
end
def publish
@import.publish_later
redirect_to import_path(@import), notice: "Your import has started in the background."
rescue Import::MaxRowCountExceededError
redirect_back_or_to import_path(@import), alert: "Your import exceeds the maximum row count of #{@import.max_row_count}."
end
def index
@pagy, @imports = pagy(Current.family.imports.where(type: Import::TYPES).ordered, limit: safe_per_page)
@breadcrumbs = [
[ t("breadcrumbs.home"), root_path ],
[ t("breadcrumbs.imports"), imports_path ]
]
render layout: "settings"
end
def new
@pending_import = Current.family.imports.ordered.pending.first
end
def create
file = import_params[:import_file]
# Handle PDF file uploads - process with AI
if file.present? && Import::ALLOWED_PDF_MIME_TYPES.include?(file.content_type)
unless valid_pdf_file?(file)
redirect_to new_import_path, alert: t("imports.create.invalid_pdf")
return
end
create_pdf_import(file)
return
end
type = params.dig(:import, :type).to_s
type = "TransactionImport" unless Import::TYPES.include?(type)
account = Current.family.accounts.find_by(id: params.dig(:import, :account_id))
import = Current.family.imports.create!(
type: type,
account: account,
date_format: Current.family.date_format,
)
if file.present?
if file.size > Import::MAX_CSV_SIZE
import.destroy
redirect_to new_import_path, alert: t("imports.create.file_too_large", max_size: Import::MAX_CSV_SIZE / 1.megabyte)
return
end
unless Import::ALLOWED_CSV_MIME_TYPES.include?(file.content_type)
import.destroy
redirect_to new_import_path, alert: t("imports.create.invalid_file_type")
return
end
# Stream reading is not fully applicable here as we store the raw string in the DB,
# but we have validated size beforehand to prevent memory exhaustion from massive files.
import.update!(raw_file_str: file.read)
redirect_to import_configuration_path(import), notice: t("imports.create.csv_uploaded")
else
redirect_to import_upload_path(import)
end
end
def show
return unless @import.requires_csv_workflow?
if !@import.uploaded?
redirect_to import_upload_path(@import), alert: t("imports.show.finalize_upload")
elsif !@import.publishable?
redirect_to import_confirm_path(@import), alert: t("imports.show.finalize_mappings")
end
end
def revert
@import.revert_later
redirect_to imports_path, notice: "Import is reverting in the background."
end
def apply_template
if @import.suggested_template
@import.apply_template!(@import.suggested_template)
redirect_to import_configuration_path(@import), notice: "Template applied."
else
redirect_to import_configuration_path(@import), alert: "No template found, please manually configure your import."
end
end
def destroy
@import.destroy
redirect_to imports_path, notice: "Your import has been deleted."
end
private
def set_import
@import = Current.family.imports.includes(:account).find(params[:id])
end
def import_params
params.require(:import).permit(:import_file)
end
def create_pdf_import(file)
if file.size > Import::MAX_PDF_SIZE
redirect_to new_import_path, alert: t("imports.create.pdf_too_large", max_size: Import::MAX_PDF_SIZE / 1.megabyte)
return
end
pdf_import = Current.family.imports.create!(type: "PdfImport")
pdf_import.pdf_file.attach(file)
pdf_import.process_with_ai_later
redirect_to import_path(pdf_import), notice: t("imports.create.pdf_processing")
end
def valid_pdf_file?(file)
header = file.read(5)
file.rewind
header&.start_with?("%PDF-")
end
end