* feat: Add PDF import with AI-powered document analysis
This enhances the import functionality to support PDF files with AI-powered
document analysis. When a PDF is uploaded, it is processed by AI to:
- Identify the document type (bank statement, credit card statement, etc.)
- Generate a summary of the document contents
- Extract key metadata (institution, dates, balances, transaction count)
After processing, an email is sent to the user asking for next steps.
Key changes:
- Add PdfImport model for handling PDF document imports
- Add Provider::Openai::PdfProcessor for AI document analysis
- Add ProcessPdfJob for async PDF processing
- Add PdfImportMailer for user notification emails
- Update imports controller to detect and handle PDF uploads
- Add PDF import option to the new import page
- Add i18n translations for all new strings
- Add comprehensive tests for the new functionality
* Add bank statement import with AI extraction
- Create ImportBankStatement assistant function for MCP
- Add BankStatementExtractor with chunked processing for small context windows
- Register function in assistant configurable
- Make PdfImport#pdf_file_content public for extractor access
- Increase OpenAI request timeout to 600s for slow local models
- Increase DB connection pool to 20 for concurrent operations
Tested with M-Pesa bank statement via remote Ollama (qwen3:8b):
- Successfully extracted 18 transactions
- Generated CSV and created TransactionImport
- Works with 3000 char chunks for small context windows
* Add pdf-reader gem dependency
The BankStatementExtractor uses PDF::Reader to parse bank statement
PDFs, but the gem was not properly declared in the Gemfile. This would
cause NameError in production when processing bank statements.
Added pdf-reader ~> 2.12 to Gemfile dependencies.
* Fix transaction deduplication to preserve legitimate duplicates
The previous deduplication logic removed ALL duplicate transactions based
on [date, amount, name], which would drop legitimate same-day duplicates
like multiple ATM withdrawals or card authorizations.
Changed to only deduplicate transactions that appear in consecutive chunks
(chunking artifacts) while preserving all legitimate duplicates within the
same chunk or non-adjacent chunks.
* Refactor bank statement extraction to use public provider method
Address code review feedback:
- Add public extract_bank_statement method to Provider::Openai
- Remove direct access to private client via send(:client)
- Update ImportBankStatement to use new public method
- Add require 'set' to BankStatementExtractor
- Remove PII-sensitive content from error logs
- Add defensive check for nil response.error
- Handle oversized PDF pages in chunking logic
- Remove unused process_native and process_generic methods
- Update email copy to reflect feature availability
- Add guard for nil document_type in email template
- Document pdf-reader gem rationale in Gemfile
Tested with both OpenAI (gpt-4o) and Ollama (qwen3:8b):
- OpenAI: 49 transactions extracted in 30s
- Ollama: 40 transactions extracted in 368s
- All encapsulation and error handling working correctly
* Update schema.rb with ai_summary and document_type columns
* Address PR #808 review comments
- Rename :csv_file to :import_file across controllers/views/tests
- Add PDF test fixture (sample_bank_statement.pdf)
- Add supports_pdf_processing? method for graceful degradation
- Revert unrelated database.yml pool change (600->3)
- Remove month_start_day schema bleed from other PR
- Fix PdfProcessor: use .strip instead of .strip_heredoc
- Add server-side PDF magic byte validation
- Conditionally show PDF import option when AI provider available
- Fix ProcessPdfJob: sanitize errors, handle update failure
- Move pdf_file attachment from Import to PdfImport
- Document deduplication logic limitations
- Fix ImportBankStatement: catch specific exceptions only
- Remove unnecessary require 'set'
- Remove dead json_schema method from PdfProcessor
- Reduce default OpenAI timeout from 600s to 60s
- Fix nil guard in text mailer template
- Add require 'csv' to ImportBankStatement
- Remove Gemfile pdf-reader comment
* Fix RuboCop indentation in ProcessPdfJob
* Refactor PDF import check to use model predicate method
Replace is_a?(PdfImport) type check with requires_csv_workflow? predicate
that leverages STI inheritance for cleaner controller logic.
* Fix missing 'unknown' locale key and schema version mismatch
- Add 'unknown: Unknown Document' to document_types locale
- Fix schema version to match latest migration (2026_01_24_180211)
* Document OPENAI_REQUEST_TIMEOUT env variable
Added to .env.local.example and docs/hosting/ai.md
* Rename ALLOWED_MIME_TYPES to ALLOWED_CSV_MIME_TYPES for clarity
* Add comment explaining requires_csv_workflow? predicate
* Remove redundant required_column_keys from PdfImport
Base class already returns [] by default
* Add ENV toggle to disable PDF processing for non-vision endpoints
OPENAI_SUPPORTS_PDF_PROCESSING=false can be used for OpenAI-compatible
endpoints (e.g., Ollama) that don't support vision/PDF processing.
* Wire up transaction extraction for PDF bank statements
- Add extracted_data JSONB column to imports
- Add extract_transactions method to PdfImport
- Call extraction in ProcessPdfJob for bank statements
- Store transactions in extracted_data for later review
* Fix ProcessPdfJob retry logic, sanitize and localize errors
- Allow retries after partial success (classification ok, extraction failed)
- Log sanitized error message instead of raw message to avoid data leakage
- Use i18n for user-facing error messages
* Add vision-capable model validation for PDF processing
* Fix drag-and-drop test to use correct field name csv_file
* Schema bleedover from another branch
* Fix drag-drop import form field name to match controller
* Add vision capability guard to process_pdf method
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
* Remove stale 1.0 import logic and model
* Fresh start
* Checkpoint before removing nav
* First working prototype
* Add trade, account, and mint import flows
* Basic working version with tests
* System tests for each import type
* Clean up mappings flow
* Clean up PR, refactor stale code, tests
* Add back row validations
* Row validations
* Fix import job test
* Fix import navigation
* Fix mint import configuration form
* Currency preset for new accounts
* Rough draft of issue system
* Simplify design
* Remove stale files from merge conflicts
* STI for issues
* Cleanup
* Improve Synth api key flow
* Stub api key for test
* Add .tool-versions to gitignore
* Add dropzone js for drag and drop file uploads
* UI for csv file uploads for import
* dropzone controller and use lucide_icon instead of svg
* Preview for file chosen
* File upload
* Remove dropzone
* Normalize I18n keys and fix lint issues
* Add system tests
* Cleanup
* Remove unwanted
* Add sync model
* Fresh fixtures for sync tests
* Sync tests overhaul
* Fix entry tests
* Complete remaining model test updates
* Update system tests
* Update demo data task
* Add system tests back to PR checks
* More simplifications, add empty family to fixtures for easier testing
* Initial entryable models
* Update transfer and tests
* Update transaction controllers and tests
* Update sync process to use new entries model
* Get dashboard working again
* Update transfers, imports, and accounts to use Account::Entry
* Update system tests
* Consolidate transaction management into entries controller
* Add permitted partial key helper
* Move account transactions list to entries controller
* Delegate transaction entries search
* Move transfer relation to entry
* Update bulk transaction management flows to use entries
* Remove test code
* Test fix attempt
* Update demo data script
* Consolidate remaining transaction partials to entries
* Consolidate valuations controller to entries controller
* Lint fix
* Remove unused files, additional cleanup
* Add back valuation creation
* Make migrations fully reversible
* Stale routes cleanup
* Migrations reversible fix
* Move types to entryable concern
* Fix search when no entries found
* Remove more unused code
* Add transfer model and clean up family snapshot fixtures
* Ignore transfers in income and expense snapshots
* Add transfer validations
* Implement basic transfer matching UI
* Fix merge conflicts
* Add missing translations
* Tweak selection states for transfer types
* Add missing i18n translation
* Introduce ActiveStorage
* Add active storage related service gems
* Update storage.yml
* Install image processing gem
- sudo apt-get install libvips (required dependency)
* Set default active storage service
* Add profile image to user model
* Amend form to allow profile images to be saved, introduce stimulus controller.
* Purge image when form is blank
* Update markup/stimulus controller
* Add test for profile image uplaods
* Add profile image validation
* Use rails guide gem versions
* Use correct ERB syntax and make all storage options configurable
* Ensure form submits when user clears profile image
* Add profile image thumbnail method
* Extract profile image to a partial
* Updates env.example and storage.yml
* Fix bug with double form save
* Add profile image to the sidenav
* Update production config
* Fix ERB formatting
* normalize en.yml
* Handle non-square images
* Use pre-processing on thumbnail variant
* Resovle gemfile.lock issues
* Rubocop style changes
---------
Signed-off-by: Christian <47796704+crobbo@users.noreply.github.com>
Co-authored-by: Christian Robinson <christian@robbo.dev>
