* Reorganize import UI with Financial Tools / Raw Data tabs
Split the flat list of import sources into two tabbed sections using
DS::Tabs: "Financial Tools" (Mint, Quicken/QIF, YNAB coming soon) and
"Raw Data" (transactions, investments, accounts, categories, rules,
documents). This prepares for adding more tool-specific importers
without cluttering the list.
https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3
* Fix import controller test to account for YNAB coming soon entry
The new YNAB "coming soon" disabled entry adds a 5th aria-disabled
element to the import dialog.
https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3
* Fix system tests to click Raw Data tab before selecting import type
Transaction, trade, and account imports are now under the Raw Data tab
and need an explicit tab click before the buttons are visible.
https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3
* feat: Add bulk import for NDJSON export files
Implements an import flow that accepts the full all.ndjson file from data exports,
allowing users to restore their complete data including:
- Accounts with accountable types
- Categories with parent relationships
- Tags and merchants
- Transactions with category, merchant, and tag references
- Trades with securities
- Valuations
- Budgets and budget categories
- Rules with conditions and actions (including compound conditions)
Key changes:
- Add BulkImport model extending Import base class
- Add Family::DataImporter to handle NDJSON parsing and import logic
- Update imports controller and views to support NDJSON workflow
- Skip configuration/mapping steps for structured NDJSON imports
- Add i18n translations for bulk import UI
- Add tests for BulkImport and DataImporter
* fix: Fix category import and test query issues
- Add default lucide_icon ("shapes") for categories when not provided
- Fix valuation test to use proper ActiveRecord joins syntax
* Linter errors
* fix: Add default color for tags when not provided in import
* fix: Add default kind for transactions when not provided in import
* Fix test
* Fix tests
* Fix remaining merge conflicts from PR 766 cherry-pick
Resolve conflict markers in test fixtures and clean up BulkImport
entry in new.html.erb to use the _import_option partial consistently.
https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3
* Import Sure `.ndjson`
* Remove `.ndjson` import from raw data
* Fix support for Sure "bulk" import from old branch
* Linter
* Fix CI test
* Fix more CI tests
* Fix tests
* Fix tests / move PDF import to first tab
* Remove redundant title
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Feat: Add QIF (Quicken Interchange Format) import functionality
- Add the ability to import QIF files for users coming from Quicken
- Includes categories and tags
- Comprehensive tests for QifImport, including parsing, row generation, and import functionality.
- Ensure handling of hierarchical categories (ex "Home:Home Improvement" is imported as Parent:Child)
* Fix QIF import issues raised in code review
- Fix two-digit year windowing in QIF date parser (e.g. '99 → 1999, not 2099)
- Fix ArgumentError from invalid `undef: :raise` encoding option
- Nil-safe `leaf_category_name` with blank guard and `.to_s` coercion
- Memoize `qif_account_type` to avoid re-parsing the full QIF file
- Add strong parameters (`selection_params`) to QifCategorySelectionsController
- Wrap all mutations in DB transactions in uploads and category-selections controllers
- Skip unchanged tag rows (only write rows where tags actually differ)
- Replace hardcoded strings with i18n keys across QIF views and nav
- Fix potentially colliding checkbox/label IDs in category selection view
- Improve keyboard accessibility: use semantic `<label>` for file picker area
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix QIF import test count and Brakeman mass assignment warning
- Update ImportsControllerTest to expect 4 disabled import options (was 3),
accounting for the new QIF import type added in this branch
- Remove :account_id from upload_params permit list; it was never accessed
through strong params (always via params.dig with Current.family scope),
so this resolves the Brakeman high-confidence mass assignment warning
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: QIF import security, safety, and i18n issues raised in code review
- Added french, spanish and german translations for newly added i18n keys
- Replace params.dig(:import, :account_id) with a proper strong-params
accessor (import_account_id) in UploadsController to satisfy Rails
parameter filtering requirements
- Guard ImportsController#show against QIF imports reaching the publish
screen before a file has been uploaded, preventing an unrescued error
on publish
- Gate the QIF "Clean" nav step link on import.uploaded? to prevent
routing to CleansController with an unconfigured import (which would
raise "Unknown import type: QifImport" via ImportsHelper)
- Replace hard-coded "txn" pluralize calls in the category/tag selection
view with t(".txn_count") and add pluralization keys to the locale file
- Localize all hard-coded strings in the QIF upload section of
uploads/show.html.erb and add corresponding en.yml keys
- Convert the CSV upload drop zone from a clickable <div> (JS-only) to
a semantic <label> element, making it keyboard-accessible without
JavaScript
* Fix: missing translations keys
* Add icon mapping and random color assignment to new categories
* fix a lint issue
* Add a warning about splits and some plumbing for future support.
Updated locales.
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat: Add PDF import with AI-powered document analysis
This enhances the import functionality to support PDF files with AI-powered
document analysis. When a PDF is uploaded, it is processed by AI to:
- Identify the document type (bank statement, credit card statement, etc.)
- Generate a summary of the document contents
- Extract key metadata (institution, dates, balances, transaction count)
After processing, an email is sent to the user asking for next steps.
Key changes:
- Add PdfImport model for handling PDF document imports
- Add Provider::Openai::PdfProcessor for AI document analysis
- Add ProcessPdfJob for async PDF processing
- Add PdfImportMailer for user notification emails
- Update imports controller to detect and handle PDF uploads
- Add PDF import option to the new import page
- Add i18n translations for all new strings
- Add comprehensive tests for the new functionality
* Add bank statement import with AI extraction
- Create ImportBankStatement assistant function for MCP
- Add BankStatementExtractor with chunked processing for small context windows
- Register function in assistant configurable
- Make PdfImport#pdf_file_content public for extractor access
- Increase OpenAI request timeout to 600s for slow local models
- Increase DB connection pool to 20 for concurrent operations
Tested with M-Pesa bank statement via remote Ollama (qwen3:8b):
- Successfully extracted 18 transactions
- Generated CSV and created TransactionImport
- Works with 3000 char chunks for small context windows
* Add pdf-reader gem dependency
The BankStatementExtractor uses PDF::Reader to parse bank statement
PDFs, but the gem was not properly declared in the Gemfile. This would
cause NameError in production when processing bank statements.
Added pdf-reader ~> 2.12 to Gemfile dependencies.
* Fix transaction deduplication to preserve legitimate duplicates
The previous deduplication logic removed ALL duplicate transactions based
on [date, amount, name], which would drop legitimate same-day duplicates
like multiple ATM withdrawals or card authorizations.
Changed to only deduplicate transactions that appear in consecutive chunks
(chunking artifacts) while preserving all legitimate duplicates within the
same chunk or non-adjacent chunks.
* Refactor bank statement extraction to use public provider method
Address code review feedback:
- Add public extract_bank_statement method to Provider::Openai
- Remove direct access to private client via send(:client)
- Update ImportBankStatement to use new public method
- Add require 'set' to BankStatementExtractor
- Remove PII-sensitive content from error logs
- Add defensive check for nil response.error
- Handle oversized PDF pages in chunking logic
- Remove unused process_native and process_generic methods
- Update email copy to reflect feature availability
- Add guard for nil document_type in email template
- Document pdf-reader gem rationale in Gemfile
Tested with both OpenAI (gpt-4o) and Ollama (qwen3:8b):
- OpenAI: 49 transactions extracted in 30s
- Ollama: 40 transactions extracted in 368s
- All encapsulation and error handling working correctly
* Update schema.rb with ai_summary and document_type columns
* Address PR #808 review comments
- Rename :csv_file to :import_file across controllers/views/tests
- Add PDF test fixture (sample_bank_statement.pdf)
- Add supports_pdf_processing? method for graceful degradation
- Revert unrelated database.yml pool change (600->3)
- Remove month_start_day schema bleed from other PR
- Fix PdfProcessor: use .strip instead of .strip_heredoc
- Add server-side PDF magic byte validation
- Conditionally show PDF import option when AI provider available
- Fix ProcessPdfJob: sanitize errors, handle update failure
- Move pdf_file attachment from Import to PdfImport
- Document deduplication logic limitations
- Fix ImportBankStatement: catch specific exceptions only
- Remove unnecessary require 'set'
- Remove dead json_schema method from PdfProcessor
- Reduce default OpenAI timeout from 600s to 60s
- Fix nil guard in text mailer template
- Add require 'csv' to ImportBankStatement
- Remove Gemfile pdf-reader comment
* Fix RuboCop indentation in ProcessPdfJob
* Refactor PDF import check to use model predicate method
Replace is_a?(PdfImport) type check with requires_csv_workflow? predicate
that leverages STI inheritance for cleaner controller logic.
* Fix missing 'unknown' locale key and schema version mismatch
- Add 'unknown: Unknown Document' to document_types locale
- Fix schema version to match latest migration (2026_01_24_180211)
* Document OPENAI_REQUEST_TIMEOUT env variable
Added to .env.local.example and docs/hosting/ai.md
* Rename ALLOWED_MIME_TYPES to ALLOWED_CSV_MIME_TYPES for clarity
* Add comment explaining requires_csv_workflow? predicate
* Remove redundant required_column_keys from PdfImport
Base class already returns [] by default
* Add ENV toggle to disable PDF processing for non-vision endpoints
OPENAI_SUPPORTS_PDF_PROCESSING=false can be used for OpenAI-compatible
endpoints (e.g., Ollama) that don't support vision/PDF processing.
* Wire up transaction extraction for PDF bank statements
- Add extracted_data JSONB column to imports
- Add extract_transactions method to PdfImport
- Call extraction in ProcessPdfJob for bank statements
- Store transactions in extracted_data for later review
* Fix ProcessPdfJob retry logic, sanitize and localize errors
- Allow retries after partial success (classification ok, extraction failed)
- Log sanitized error message instead of raw message to avoid data leakage
- Use i18n for user-facing error messages
* Add vision-capable model validation for PDF processing
* Fix drag-and-drop test to use correct field name csv_file
* Schema bleedover from another branch
* Fix drag-drop import form field name to match controller
* Add vision capability guard to process_pdf method
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
* Implement API v1 Imports controller
- Add Api::V1::ImportsController with index, show, and create actions
- Add Jbuilder views for index and show
- Add integration tests
- Implement row generation logic in create action
- Update routes
* Validate import account belongs to family
- Add validation to Import model to ensure account belongs to the same family
- Add regression test case in Api::V1::ImportsControllerTest
* updating docs to be more detailed
* Rescue StandardError instead of bare rescue in ImportsController
* Optimize Imports API and fix documentation
- Implement rows_count counter cache for Imports
- Preload rows in Api::V1::ImportsController#show
- Update documentation to show correct OAuth scopes
* Fix formatting in ImportsControllerTest
* Permit all import parameters and fix unknown attribute error
* Restore API routes for auth, chats, and messages
* removing pr summary
* Fix trailing whitespace and configured? test failure
- Update Import#configured? to use rows_count for performance and consistency
- Mock rows_count in TransactionImportTest
- Fix trailing whitespace in migration
* Harden security and fix mass assignment in ImportsController
- Handle type and account_id explicitly in create action
- Rename import_params to import_config_params for clarity
- Validate type against Import::TYPES
* Fix MintImport rows_count update and migration whitespace
- Update MintImport#generate_rows_from_csv to update rows_count counter cache
- Fix trailing whitespace and final newline in AddRowsCountToImports migration
* Implement full-screen Drag and Drop CSV import on Transactions page
- Add DragAndDropImport Stimulus controller listening on document
- Add full-screen overlay with icon and text to Transactions index
- Update ImportsController to handle direct file uploads via create action
- Add system test for drag and drop functionality
* Implement Drag and Drop CSV upload on Import Upload page
- Add drag-and-drop-import controller to import/uploads/show
- Add full-screen overlay to import/uploads/show
- Annotate upload form and input with drag-and-drop targets
- Add PR_SUMMARY.md
* removing pr summary
* Add file validation to ImportsController
- Validate file size (max 10MB) and MIME type in create action
- Prevent memory exhaustion and invalid file processing
- Defined MAX_CSV_SIZE and ALLOWED_MIME_TYPES in Import model
* Refactor dragLeave logic with counter pattern to prevent flickering
* Extract shared drag-and-drop overlay partial
- Create app/views/imports/_drag_drop_overlay.html.erb
- Update transactions/index and import/uploads/show to use the partial
- Reduce code duplication in views
* Update Brakeman and harden ImportsController security
- Update brakeman to 7.1.2
- Explicitly handle type assignment in ImportsController#create to avoid mass assignment
- Remove :type from permitted import parameters
* Fix trailing whitespace in DragAndDropImportTest
* Don't commit LLM comments as file
* FIX add api validation
---------
Co-authored-by: Carlos Adames <cj@Carloss-MacBook-Air.local>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: sokie <sokysrm@gmail.com>
* Add full import/export support for rules with versioned JSON schema
This commit implements comprehensive import/export functionality for rules,
allowing users to back up and restore their rule definitions.
Key features:
- Export rules to both CSV and NDJSON formats with versioned schema (v1)
- Import rules from CSV with full support for nested conditions and actions
- UUID to name mapping for categories and merchants for portability
- Support for compound conditions with sub-conditions
- Comprehensive test coverage for export and import functionality
- UI integration for rules import in the imports interface
Technical details:
- Extended Family::DataExporter to generate rules.csv and include rules in all.ndjson
- Created RuleImport model following the existing Import STI pattern
- Added migration for rule-specific columns in import_rows table
- Implemented serialization helpers to map UUIDs to human-readable names
- Added i18n support for the new import option
- Included versioning in NDJSON export to support future schema evolution
The implementation ensures rules can be safely exported from one family
and imported into another, even when category/merchant IDs differ,
by mapping between names and IDs during export/import.
* Fix AR migration version
* Mention support for rules export
* Rabbit suggestion
* Fix tests
* Missed schema.rb
* Fix sample CSV download for rule import
* Fix parsing in Rules import
* Fix tests
* Rule import message i18n
* Export tag names, not UUIDs
* Make sure tags are created if needed at import
* Avoid test errors when running in parallel
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Allow category imports to set icons
* Linter
* Update category import description in English locale
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
* Rename icon in CSV header to `lucide-icon`
* Make sure we export the icon while we're at this
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
* Allow CSV imports to be configured to a single account or multiple accounts
* Initialize import directly from accounts page
* Fix brakeman warnings
* Fix schema
* Fix Synth check
* feat(import): add currency and number format support for CSV imports
* feat(import): add currency and format for mint and trade
* fix(imports): remove currency field in favor of currency csv col
* fix(imports): remove validate column from import model
* test(import): add some tests for imports
* test(import): add some tests for generate_rows_from_csv
* fix: change method name for import model
* fix: change before validation
---------
Co-authored-by: danestves <danestves@users.noreply.github.com>
* Remove stale 1.0 import logic and model
* Fresh start
* Checkpoint before removing nav
* First working prototype
* Add trade, account, and mint import flows
* Basic working version with tests
* System tests for each import type
* Clean up mappings flow
* Clean up PR, refactor stale code, tests
* Add back row validations
* Row validations
* Fix import job test
* Fix import navigation
* Fix mint import configuration form
* Currency preset for new accounts
* add col_sep to import model
* add validation for col_sep column
* add col_sep option to csv import model
* make use of col_sep option in import model
* add column separator field to new/edit action of an import
* add col_sep parameter to create/update action
* fix spacing between fields
Co-authored-by: Zach Gollwitzer <zach.gollwitzer@gmail.com>
Signed-off-by: Alexander Schrot <alexander@axs-labs.com>
---------
Signed-off-by: Alexander Schrot <alexander@axs-labs.com>
Co-authored-by: Zach Gollwitzer <zach.gollwitzer@gmail.com>
* Initial entryable models
* Update transfer and tests
* Update transaction controllers and tests
* Update sync process to use new entries model
* Get dashboard working again
* Update transfers, imports, and accounts to use Account::Entry
* Update system tests
* Consolidate transaction management into entries controller
* Add permitted partial key helper
* Move account transactions list to entries controller
* Delegate transaction entries search
* Move transfer relation to entry
* Update bulk transaction management flows to use entries
* Remove test code
* Test fix attempt
* Update demo data script
* Consolidate remaining transaction partials to entries
* Consolidate valuations controller to entries controller
* Lint fix
* Remove unused files, additional cleanup
* Add back valuation creation
* Make migrations fully reversible
* Stale routes cleanup
* Migrations reversible fix
* Move types to entryable concern
* Fix search when no entries found
* Remove more unused code
The get_selected_header_for_field method was incorrectly using the entire field object instead of the field key to dig into the column_mappings hash. This caused an error when trying to retrieve the selected header for a field.
Introduces a basic CSV import module for bulk-importing account transactions.
Changes include:
- User can load a CSV
- User can configure the column mappings for a CSV
- Imported CSV shows invalid cells
- User can clean up their data directly in the UI
- User can see a preview of the import rows and confirm import
- Layout refactor + Import nav stepper
- System test stability improvements
