Files
sure/app/controllers/imports_controller.rb
MkDev11 6f8858b1a6 feat/Add AI-Powered Bank Statement Import (step 1, PDF import & analysis) (#808)
* feat: Add PDF import with AI-powered document analysis

This enhances the import functionality to support PDF files with AI-powered
document analysis. When a PDF is uploaded, it is processed by AI to:
- Identify the document type (bank statement, credit card statement, etc.)
- Generate a summary of the document contents
- Extract key metadata (institution, dates, balances, transaction count)

After processing, an email is sent to the user asking for next steps.

Key changes:
- Add PdfImport model for handling PDF document imports
- Add Provider::Openai::PdfProcessor for AI document analysis
- Add ProcessPdfJob for async PDF processing
- Add PdfImportMailer for user notification emails
- Update imports controller to detect and handle PDF uploads
- Add PDF import option to the new import page
- Add i18n translations for all new strings
- Add comprehensive tests for the new functionality

* Add bank statement import with AI extraction

- Create ImportBankStatement assistant function for MCP
- Add BankStatementExtractor with chunked processing for small context windows
- Register function in assistant configurable
- Make PdfImport#pdf_file_content public for extractor access
- Increase OpenAI request timeout to 600s for slow local models
- Increase DB connection pool to 20 for concurrent operations

Tested with M-Pesa bank statement via remote Ollama (qwen3:8b):
- Successfully extracted 18 transactions
- Generated CSV and created TransactionImport
- Works with 3000 char chunks for small context windows

* Add pdf-reader gem dependency

The BankStatementExtractor uses PDF::Reader to parse bank statement
PDFs, but the gem was not properly declared in the Gemfile. This would
cause NameError in production when processing bank statements.

Added pdf-reader ~> 2.12 to Gemfile dependencies.

* Fix transaction deduplication to preserve legitimate duplicates

The previous deduplication logic removed ALL duplicate transactions based
on [date, amount, name], which would drop legitimate same-day duplicates
like multiple ATM withdrawals or card authorizations.

Changed to only deduplicate transactions that appear in consecutive chunks
(chunking artifacts) while preserving all legitimate duplicates within the
same chunk or non-adjacent chunks.

* Refactor bank statement extraction to use public provider method

Address code review feedback:
- Add public extract_bank_statement method to Provider::Openai
- Remove direct access to private client via send(:client)
- Update ImportBankStatement to use new public method
- Add require 'set' to BankStatementExtractor
- Remove PII-sensitive content from error logs
- Add defensive check for nil response.error
- Handle oversized PDF pages in chunking logic
- Remove unused process_native and process_generic methods
- Update email copy to reflect feature availability
- Add guard for nil document_type in email template
- Document pdf-reader gem rationale in Gemfile

Tested with both OpenAI (gpt-4o) and Ollama (qwen3:8b):
- OpenAI: 49 transactions extracted in 30s
- Ollama: 40 transactions extracted in 368s
- All encapsulation and error handling working correctly

* Update schema.rb with ai_summary and document_type columns

* Address PR #808 review comments

- Rename :csv_file to :import_file across controllers/views/tests
- Add PDF test fixture (sample_bank_statement.pdf)
- Add supports_pdf_processing? method for graceful degradation
- Revert unrelated database.yml pool change (600->3)
- Remove month_start_day schema bleed from other PR
- Fix PdfProcessor: use .strip instead of .strip_heredoc
- Add server-side PDF magic byte validation
- Conditionally show PDF import option when AI provider available
- Fix ProcessPdfJob: sanitize errors, handle update failure
- Move pdf_file attachment from Import to PdfImport
- Document deduplication logic limitations
- Fix ImportBankStatement: catch specific exceptions only
- Remove unnecessary require 'set'
- Remove dead json_schema method from PdfProcessor
- Reduce default OpenAI timeout from 600s to 60s
- Fix nil guard in text mailer template
- Add require 'csv' to ImportBankStatement
- Remove Gemfile pdf-reader comment

* Fix RuboCop indentation in ProcessPdfJob

* Refactor PDF import check to use model predicate method

Replace is_a?(PdfImport) type check with requires_csv_workflow? predicate
that leverages STI inheritance for cleaner controller logic.

* Fix missing 'unknown' locale key and schema version mismatch

- Add 'unknown: Unknown Document' to document_types locale
- Fix schema version to match latest migration (2026_01_24_180211)

* Document OPENAI_REQUEST_TIMEOUT env variable

Added to .env.local.example and docs/hosting/ai.md

* Rename ALLOWED_MIME_TYPES to ALLOWED_CSV_MIME_TYPES for clarity

* Add comment explaining requires_csv_workflow? predicate

* Remove redundant required_column_keys from PdfImport

Base class already returns [] by default

* Add ENV toggle to disable PDF processing for non-vision endpoints

OPENAI_SUPPORTS_PDF_PROCESSING=false can be used for OpenAI-compatible
endpoints (e.g., Ollama) that don't support vision/PDF processing.

* Wire up transaction extraction for PDF bank statements

- Add extracted_data JSONB column to imports
- Add extract_transactions method to PdfImport
- Call extraction in ProcessPdfJob for bank statements
- Store transactions in extracted_data for later review

* Fix ProcessPdfJob retry logic, sanitize and localize errors

- Allow retries after partial success (classification ok, extraction failed)
- Log sanitized error message instead of raw message to avoid data leakage
- Use i18n for user-facing error messages

* Add vision-capable model validation for PDF processing

* Fix drag-and-drop test to use correct field name csv_file

* Schema bleedover from another branch

* Fix drag-drop import form field name to match controller

* Add vision capability guard to process_pdf method

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
2026-01-30 20:44:25 +01:00

130 lines
3.9 KiB
Ruby

class ImportsController < ApplicationController
include SettingsHelper
before_action :set_import, only: %i[show publish destroy revert apply_template]
def publish
@import.publish_later
redirect_to import_path(@import), notice: "Your import has started in the background."
rescue Import::MaxRowCountExceededError
redirect_back_or_to import_path(@import), alert: "Your import exceeds the maximum row count of #{@import.max_row_count}."
end
def index
@pagy, @imports = pagy(Current.family.imports.where(type: Import::TYPES).ordered, limit: safe_per_page)
@breadcrumbs = [
[ t("breadcrumbs.home"), root_path ],
[ t("breadcrumbs.imports"), imports_path ]
]
render layout: "settings"
end
def new
@pending_import = Current.family.imports.ordered.pending.first
end
def create
file = import_params[:import_file]
# Handle PDF file uploads - process with AI
if file.present? && Import::ALLOWED_PDF_MIME_TYPES.include?(file.content_type)
unless valid_pdf_file?(file)
redirect_to new_import_path, alert: t("imports.create.invalid_pdf")
return
end
create_pdf_import(file)
return
end
type = params.dig(:import, :type).to_s
type = "TransactionImport" unless Import::TYPES.include?(type)
account = Current.family.accounts.find_by(id: params.dig(:import, :account_id))
import = Current.family.imports.create!(
type: type,
account: account,
date_format: Current.family.date_format,
)
if file.present?
if file.size > Import::MAX_CSV_SIZE
import.destroy
redirect_to new_import_path, alert: t("imports.create.file_too_large", max_size: Import::MAX_CSV_SIZE / 1.megabyte)
return
end
unless Import::ALLOWED_CSV_MIME_TYPES.include?(file.content_type)
import.destroy
redirect_to new_import_path, alert: t("imports.create.invalid_file_type")
return
end
# Stream reading is not fully applicable here as we store the raw string in the DB,
# but we have validated size beforehand to prevent memory exhaustion from massive files.
import.update!(raw_file_str: file.read)
redirect_to import_configuration_path(import), notice: t("imports.create.csv_uploaded")
else
redirect_to import_upload_path(import)
end
end
def show
return unless @import.requires_csv_workflow?
if !@import.uploaded?
redirect_to import_upload_path(@import), alert: t("imports.show.finalize_upload")
elsif !@import.publishable?
redirect_to import_confirm_path(@import), alert: t("imports.show.finalize_mappings")
end
end
def revert
@import.revert_later
redirect_to imports_path, notice: "Import is reverting in the background."
end
def apply_template
if @import.suggested_template
@import.apply_template!(@import.suggested_template)
redirect_to import_configuration_path(@import), notice: "Template applied."
else
redirect_to import_configuration_path(@import), alert: "No template found, please manually configure your import."
end
end
def destroy
@import.destroy
redirect_to imports_path, notice: "Your import has been deleted."
end
private
def set_import
@import = Current.family.imports.includes(:account).find(params[:id])
end
def import_params
params.require(:import).permit(:import_file)
end
def create_pdf_import(file)
if file.size > Import::MAX_PDF_SIZE
redirect_to new_import_path, alert: t("imports.create.pdf_too_large", max_size: Import::MAX_PDF_SIZE / 1.megabyte)
return
end
pdf_import = Current.family.imports.create!(type: "PdfImport")
pdf_import.pdf_file.attach(file)
pdf_import.process_with_ai_later
redirect_to import_path(pdf_import), notice: t("imports.create.pdf_processing")
end
def valid_pdf_file?(file)
header = file.read(5)
file.rewind
header&.start_with?("%PDF-")
end
end