* feat: Add PDF import with AI-powered document analysis
This enhances the import functionality to support PDF files with AI-powered
document analysis. When a PDF is uploaded, it is processed by AI to:
- Identify the document type (bank statement, credit card statement, etc.)
- Generate a summary of the document contents
- Extract key metadata (institution, dates, balances, transaction count)
After processing, an email is sent to the user asking for next steps.
Key changes:
- Add PdfImport model for handling PDF document imports
- Add Provider::Openai::PdfProcessor for AI document analysis
- Add ProcessPdfJob for async PDF processing
- Add PdfImportMailer for user notification emails
- Update imports controller to detect and handle PDF uploads
- Add PDF import option to the new import page
- Add i18n translations for all new strings
- Add comprehensive tests for the new functionality
* Add bank statement import with AI extraction
- Create ImportBankStatement assistant function for MCP
- Add BankStatementExtractor with chunked processing for small context windows
- Register function in assistant configurable
- Make PdfImport#pdf_file_content public for extractor access
- Increase OpenAI request timeout to 600s for slow local models
- Increase DB connection pool to 20 for concurrent operations
Tested with M-Pesa bank statement via remote Ollama (qwen3:8b):
- Successfully extracted 18 transactions
- Generated CSV and created TransactionImport
- Works with 3000 char chunks for small context windows
* Add pdf-reader gem dependency
The BankStatementExtractor uses PDF::Reader to parse bank statement
PDFs, but the gem was not properly declared in the Gemfile. This would
cause NameError in production when processing bank statements.
Added pdf-reader ~> 2.12 to Gemfile dependencies.
* Fix transaction deduplication to preserve legitimate duplicates
The previous deduplication logic removed ALL duplicate transactions based
on [date, amount, name], which would drop legitimate same-day duplicates
like multiple ATM withdrawals or card authorizations.
Changed to only deduplicate transactions that appear in consecutive chunks
(chunking artifacts) while preserving all legitimate duplicates within the
same chunk or non-adjacent chunks.
* Refactor bank statement extraction to use public provider method
Address code review feedback:
- Add public extract_bank_statement method to Provider::Openai
- Remove direct access to private client via send(:client)
- Update ImportBankStatement to use new public method
- Add require 'set' to BankStatementExtractor
- Remove PII-sensitive content from error logs
- Add defensive check for nil response.error
- Handle oversized PDF pages in chunking logic
- Remove unused process_native and process_generic methods
- Update email copy to reflect feature availability
- Add guard for nil document_type in email template
- Document pdf-reader gem rationale in Gemfile
Tested with both OpenAI (gpt-4o) and Ollama (qwen3:8b):
- OpenAI: 49 transactions extracted in 30s
- Ollama: 40 transactions extracted in 368s
- All encapsulation and error handling working correctly
* Update schema.rb with ai_summary and document_type columns
* Address PR #808 review comments
- Rename :csv_file to :import_file across controllers/views/tests
- Add PDF test fixture (sample_bank_statement.pdf)
- Add supports_pdf_processing? method for graceful degradation
- Revert unrelated database.yml pool change (600->3)
- Remove month_start_day schema bleed from other PR
- Fix PdfProcessor: use .strip instead of .strip_heredoc
- Add server-side PDF magic byte validation
- Conditionally show PDF import option when AI provider available
- Fix ProcessPdfJob: sanitize errors, handle update failure
- Move pdf_file attachment from Import to PdfImport
- Document deduplication logic limitations
- Fix ImportBankStatement: catch specific exceptions only
- Remove unnecessary require 'set'
- Remove dead json_schema method from PdfProcessor
- Reduce default OpenAI timeout from 600s to 60s
- Fix nil guard in text mailer template
- Add require 'csv' to ImportBankStatement
- Remove Gemfile pdf-reader comment
* Fix RuboCop indentation in ProcessPdfJob
* Refactor PDF import check to use model predicate method
Replace is_a?(PdfImport) type check with requires_csv_workflow? predicate
that leverages STI inheritance for cleaner controller logic.
* Fix missing 'unknown' locale key and schema version mismatch
- Add 'unknown: Unknown Document' to document_types locale
- Fix schema version to match latest migration (2026_01_24_180211)
* Document OPENAI_REQUEST_TIMEOUT env variable
Added to .env.local.example and docs/hosting/ai.md
* Rename ALLOWED_MIME_TYPES to ALLOWED_CSV_MIME_TYPES for clarity
* Add comment explaining requires_csv_workflow? predicate
* Remove redundant required_column_keys from PdfImport
Base class already returns [] by default
* Add ENV toggle to disable PDF processing for non-vision endpoints
OPENAI_SUPPORTS_PDF_PROCESSING=false can be used for OpenAI-compatible
endpoints (e.g., Ollama) that don't support vision/PDF processing.
* Wire up transaction extraction for PDF bank statements
- Add extracted_data JSONB column to imports
- Add extract_transactions method to PdfImport
- Call extraction in ProcessPdfJob for bank statements
- Store transactions in extracted_data for later review
* Fix ProcessPdfJob retry logic, sanitize and localize errors
- Allow retries after partial success (classification ok, extraction failed)
- Log sanitized error message instead of raw message to avoid data leakage
- Use i18n for user-facing error messages
* Add vision-capable model validation for PDF processing
* Fix drag-and-drop test to use correct field name csv_file
* Schema bleedover from another branch
* Fix drag-drop import form field name to match controller
* Add vision capability guard to process_pdf method
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
* feat: add valuations API endpoints for managing account reconciliations
* refactor: formatting
* fix: make account extraction clearer
* feat: validation and error handling improvements
* feat: transaction
* feat: error handling
* Add API documentation LLM context
* Make it easier for people
* feat: transaction in creation
* feat: add OpenAPI spec for Valuations API
* fix: update notes validation to check for key presence
* Prevent double render
* All other docs use `apiKeyAuth`
* More `apiKeyAuth`
* Remove testing assertions from API doc specs
* fix: correct valuation entry references
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Add files via upload
Signed-off-by: Jose <39016041+jospaquim@users.noreply.github.com>
* Add merchants and tags resources to routes
Signed-off-by: Jose <39016041+jospaquim@users.noreply.github.com>
* update
* update spaces
* fix: Apply CodeRabbit suggestions and add YARD documentation
* docs: Add API documentation for merchants and tags endpoints
* fix: Address CodeRabbit feedback on documentation
* fix: Use authorize_scope! instead of ensure_read_scope
* test(api): Add request specs for merchants and tags endpoints
* test(api): Add request specs for merchants and tags endpoints
* test(api): Convert specs to Minitest format in test/
* fix: Correct indentation for private methods
* fix: merchant and tag test
* Enhance tag tests for family scope and access
Added tests to ensure tags from other families are not returned and that attempts to access them return 404.
Signed-off-by: Jose <39016041+jospaquim@users.noreply.github.com>
* Enhance merchants controller tests for family scope
Added tests to ensure that merchants from other families are not returned in the index action and that accessing a merchant from another family returns a 404 error.
Signed-off-by: Jose <39016041+jospaquim@users.noreply.github.com>
* Fix test/implementation
* Remove old token test code
* Improve test
---------
Signed-off-by: Jose <39016041+jospaquim@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Implement dynamic role assignment for new family creators.
Introduced `User.role_for_new_family_creator` to assign `super_admin` to the first user of an instance and a configurable fallback role (e.g., `admin`) to subsequent users. Updated controllers and tests accordingly.
* Update default fallback role for family creators to admin.
---------
Co-authored-by: luckyPipewrench <luckypipewrench@proton.me>
* Add files via upload
Signed-off-by: Jose <39016041+jospaquim@users.noreply.github.com>
* Add merchants and tags resources to routes
Signed-off-by: Jose <39016041+jospaquim@users.noreply.github.com>
* update
* update spaces
* fix: Apply CodeRabbit suggestions and add YARD documentation
* docs: Add API documentation for merchants and tags endpoints
* fix: Address CodeRabbit feedback on documentation
* fix: Use authorize_scope! instead of ensure_read_scope
---------
Signed-off-by: Jose <39016041+jospaquim@users.noreply.github.com>
Backend fixes:
- Fix duplicate AssistantResponseJob triggering causing duplicate AI responses
- UserMessage model already handles job triggering via after_create_commit callback
- Remove redundant job enqueue in chats_controller and messages_controller
Mobile app features:
- Implement complete AI chat interface and conversation management
- Add Chat, Message, and ToolCall data models
- Add ChatProvider for state management with polling mechanism
- Add ChatService to handle all chat-related API requests
- Add chat list screen (ChatListScreen)
- Add conversation detail screen (ChatConversationScreen)
- Refactor navigation structure with bottom navigation bar (MainNavigationScreen)
- Add settings screen (SettingsScreen)
- Optimize TransactionsProvider to support account filtering
Technical details:
- Implement message polling mechanism for real-time AI responses
- Support chat creation, deletion, retry and other operations
- Integrate Material Design 3 design language
- Improve user experience and error handling
Co-authored-by: dwvwdv <dwvwdv@protonmail.com>
* Implement API v1 Imports controller
- Add Api::V1::ImportsController with index, show, and create actions
- Add Jbuilder views for index and show
- Add integration tests
- Implement row generation logic in create action
- Update routes
* Validate import account belongs to family
- Add validation to Import model to ensure account belongs to the same family
- Add regression test case in Api::V1::ImportsControllerTest
* updating docs to be more detailed
* Rescue StandardError instead of bare rescue in ImportsController
* Optimize Imports API and fix documentation
- Implement rows_count counter cache for Imports
- Preload rows in Api::V1::ImportsController#show
- Update documentation to show correct OAuth scopes
* Fix formatting in ImportsControllerTest
* Permit all import parameters and fix unknown attribute error
* Restore API routes for auth, chats, and messages
* removing pr summary
* Fix trailing whitespace and configured? test failure
- Update Import#configured? to use rows_count for performance and consistency
- Mock rows_count in TransactionImportTest
- Fix trailing whitespace in migration
* Harden security and fix mass assignment in ImportsController
- Handle type and account_id explicitly in create action
- Rename import_params to import_config_params for clarity
- Validate type against Import::TYPES
* Fix MintImport rows_count update and migration whitespace
- Update MintImport#generate_rows_from_csv to update rows_count counter cache
- Fix trailing whitespace and final newline in AddRowsCountToImports migration
* Implement full-screen Drag and Drop CSV import on Transactions page
- Add DragAndDropImport Stimulus controller listening on document
- Add full-screen overlay with icon and text to Transactions index
- Update ImportsController to handle direct file uploads via create action
- Add system test for drag and drop functionality
* Implement Drag and Drop CSV upload on Import Upload page
- Add drag-and-drop-import controller to import/uploads/show
- Add full-screen overlay to import/uploads/show
- Annotate upload form and input with drag-and-drop targets
- Add PR_SUMMARY.md
* removing pr summary
* Add file validation to ImportsController
- Validate file size (max 10MB) and MIME type in create action
- Prevent memory exhaustion and invalid file processing
- Defined MAX_CSV_SIZE and ALLOWED_MIME_TYPES in Import model
* Refactor dragLeave logic with counter pattern to prevent flickering
* Extract shared drag-and-drop overlay partial
- Create app/views/imports/_drag_drop_overlay.html.erb
- Update transactions/index and import/uploads/show to use the partial
- Reduce code duplication in views
* Update Brakeman and harden ImportsController security
- Update brakeman to 7.1.2
- Explicitly handle type assignment in ImportsController#create to avoid mass assignment
- Remove :type from permitted import parameters
* Fix trailing whitespace in DragAndDropImportTest
* Don't commit LLM comments as file
* FIX add api validation
---------
Co-authored-by: Carlos Adames <cj@Carloss-MacBook-Air.local>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: sokie <sokysrm@gmail.com>
* Add API endpoint for triggering family sync
Introduces Api::V1::SyncController with a create action to queue a family sync, applying all active rules and syncing accounts. Adds corresponding route, JSON response view, and comprehensive controller tests for authorization and response validation.
* Rename started_at to syncing_at in sync API response
Updated the sync create JSON response to use 'syncing_at' instead of 'started_at'. Adjusted related controller test to check for 'syncing_at'. Also updated API authentication header in test to use 'X-Api-Key' instead of Bearer token.
* Update app/controllers/api/v1/sync_controller.rb
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Mark Hendriksen <hendriksen-mark@hotmail.com>
---------
Signed-off-by: Mark Hendriksen <hendriksen-mark@hotmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Add comprehensive AI/LLM configuration documentation
* Fix Chat.start! to use default model when model is nil or empty
* Ensure all controllers use Chat.default_model for consistency
* Move AI doc inside `hosting/`
* Probably too much error handling
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
- Introduced NoopApiRateLimiter to effectively disable API rate limiting for self-hosted mode.
- Updated ApiRateLimiter to delegate to NoopApiRateLimiter when running self-hosted.
- Increased Rack::Attack throttle limits significantly for self-hosted deployments.
- Added tests for NoopApiRateLimiter to ensure correct behavior.
- This allows self-hosted users to make more API requests without restriction, while keeping stricter limits for SaaS deployments.
- Remove trailing whitespace in auth controller and mobile device model
- Update API key test to expect new validation message with source
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Replace API keys with OAuth2 tokens for mobile apps
- Add device tracking and management for mobile sessions
- Implement 30-day token expiration with refresh tokens
- Add MFA/2FA support for mobile login
- Create dedicated auth endpoints (signup/login/refresh)
- Skip CSRF protection for API endpoints
- Return plaintext tokens (not hashed) in responses
- Track devices with unique IDs and metadata
- Enable seamless native mobile experience without OAuth redirects
This provides enterprise-grade security for the iOS/Android apps while maintaining a completely native authentication flow.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add chats#index and chats#show endpoints to list and view AI conversations
- Add messages#create endpoint to send messages to AI chats
- Include API documentation for chat endpoints
- Add controller tests for new endpoints
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* OAuth
* Add API test routes and update Doorkeeper token handling for test environment
- Introduced API namespace with test routes for controller testing in the test environment.
- Updated Doorkeeper configuration to allow fallback to plain tokens in the test environment for easier testing.
- Modified schema to change resource_owner_id type from bigint to string.
* Implement API key authentication and enhance access control
- Replaced Doorkeeper OAuth authentication with a custom method supporting both OAuth and API keys in the BaseController.
- Added methods for API key authentication, including validation and logging.
- Introduced scope-based authorization for API keys in the TestController.
- Updated routes to include API key management endpoints.
- Enhanced logging for API access to include authentication method details.
- Added tests for API key functionality, including validation, scope checks, and access control enforcement.
* Add API key rate limiting and usage tracking
- Implemented rate limiting for API key authentication in BaseController.
- Added methods to check rate limits, render appropriate responses, and include rate limit headers in responses.
- Updated routes to include a new usage resource for tracking API usage.
- Enhanced tests to verify rate limit functionality, including exceeding limits and per-key tracking.
- Cleaned up Redis data in tests to ensure isolation between test cases.
* Add Jbuilder for JSON rendering and refactor AccountsController
- Added Jbuilder gem for improved JSON response handling.
- Refactored index action in AccountsController to utilize Jbuilder for rendering JSON.
- Removed manual serialization of accounts and streamlined response structure.
- Implemented a before_action in BaseController to enforce JSON format for all API requests.
* Add transactions resource to API routes
- Added routes for transactions, allowing index, show, create, update, and destroy actions.
- This enhancement supports comprehensive transaction management within the API.
* Enhance API authentication and onboarding handling
- Updated BaseController to skip onboarding requirements for API endpoints and added manual token verification for OAuth authentication.
- Improved error handling and logging for invalid access tokens.
- Introduced a method to set up the current context for API requests, ensuring compatibility with session-like behavior.
- Excluded API paths from onboarding redirects in the Onboardable concern.
- Updated database schema to change resource_owner_id type from bigint to string for OAuth access grants.
* Fix rubocop offenses
- Fix indentation and spacing issues
- Convert single quotes to double quotes
- Add spaces inside array brackets
- Fix comment alignment
- Add missing trailing newlines
- Correct else/end alignment
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Fix API test failures and improve test reliability
- Fix ApiRateLimiterTest by removing mock users method and using fixtures
- Fix UsageControllerTest by removing mock users method and using fixtures
- Fix BaseControllerTest by using different users for multiple API keys
- Use unique display_key values with SecureRandom to avoid conflicts
- Fix double render issue in UsageController by returning after authorize_scope\!
- Specify controller name in routes for usage resource
- Remove trailing whitespace and empty lines per Rubocop
All tests now pass and linting is clean.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
* Add API transactions controller warning to brakeman ignore
The account_id parameter in the API transactions controller is properly
validated on line 79: family.accounts.find(transaction_params[:account_id])
This ensures users can only create transactions in accounts belonging to
their family, making this a false positive.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Signed-off-by: Josh Pigford <josh@joshpigford.com>
Co-authored-by: Claude <noreply@anthropic.com>
