* feat: Add PDF import with AI-powered document analysis
This enhances the import functionality to support PDF files with AI-powered
document analysis. When a PDF is uploaded, it is processed by AI to:
- Identify the document type (bank statement, credit card statement, etc.)
- Generate a summary of the document contents
- Extract key metadata (institution, dates, balances, transaction count)
After processing, an email is sent to the user asking for next steps.
Key changes:
- Add PdfImport model for handling PDF document imports
- Add Provider::Openai::PdfProcessor for AI document analysis
- Add ProcessPdfJob for async PDF processing
- Add PdfImportMailer for user notification emails
- Update imports controller to detect and handle PDF uploads
- Add PDF import option to the new import page
- Add i18n translations for all new strings
- Add comprehensive tests for the new functionality
* Add bank statement import with AI extraction
- Create ImportBankStatement assistant function for MCP
- Add BankStatementExtractor with chunked processing for small context windows
- Register function in assistant configurable
- Make PdfImport#pdf_file_content public for extractor access
- Increase OpenAI request timeout to 600s for slow local models
- Increase DB connection pool to 20 for concurrent operations
Tested with M-Pesa bank statement via remote Ollama (qwen3:8b):
- Successfully extracted 18 transactions
- Generated CSV and created TransactionImport
- Works with 3000 char chunks for small context windows
* Add pdf-reader gem dependency
The BankStatementExtractor uses PDF::Reader to parse bank statement
PDFs, but the gem was not properly declared in the Gemfile. This would
cause NameError in production when processing bank statements.
Added pdf-reader ~> 2.12 to Gemfile dependencies.
* Fix transaction deduplication to preserve legitimate duplicates
The previous deduplication logic removed ALL duplicate transactions based
on [date, amount, name], which would drop legitimate same-day duplicates
like multiple ATM withdrawals or card authorizations.
Changed to only deduplicate transactions that appear in consecutive chunks
(chunking artifacts) while preserving all legitimate duplicates within the
same chunk or non-adjacent chunks.
* Refactor bank statement extraction to use public provider method
Address code review feedback:
- Add public extract_bank_statement method to Provider::Openai
- Remove direct access to private client via send(:client)
- Update ImportBankStatement to use new public method
- Add require 'set' to BankStatementExtractor
- Remove PII-sensitive content from error logs
- Add defensive check for nil response.error
- Handle oversized PDF pages in chunking logic
- Remove unused process_native and process_generic methods
- Update email copy to reflect feature availability
- Add guard for nil document_type in email template
- Document pdf-reader gem rationale in Gemfile
Tested with both OpenAI (gpt-4o) and Ollama (qwen3:8b):
- OpenAI: 49 transactions extracted in 30s
- Ollama: 40 transactions extracted in 368s
- All encapsulation and error handling working correctly
* Update schema.rb with ai_summary and document_type columns
* Address PR #808 review comments
- Rename :csv_file to :import_file across controllers/views/tests
- Add PDF test fixture (sample_bank_statement.pdf)
- Add supports_pdf_processing? method for graceful degradation
- Revert unrelated database.yml pool change (600->3)
- Remove month_start_day schema bleed from other PR
- Fix PdfProcessor: use .strip instead of .strip_heredoc
- Add server-side PDF magic byte validation
- Conditionally show PDF import option when AI provider available
- Fix ProcessPdfJob: sanitize errors, handle update failure
- Move pdf_file attachment from Import to PdfImport
- Document deduplication logic limitations
- Fix ImportBankStatement: catch specific exceptions only
- Remove unnecessary require 'set'
- Remove dead json_schema method from PdfProcessor
- Reduce default OpenAI timeout from 600s to 60s
- Fix nil guard in text mailer template
- Add require 'csv' to ImportBankStatement
- Remove Gemfile pdf-reader comment
* Fix RuboCop indentation in ProcessPdfJob
* Refactor PDF import check to use model predicate method
Replace is_a?(PdfImport) type check with requires_csv_workflow? predicate
that leverages STI inheritance for cleaner controller logic.
* Fix missing 'unknown' locale key and schema version mismatch
- Add 'unknown: Unknown Document' to document_types locale
- Fix schema version to match latest migration (2026_01_24_180211)
* Document OPENAI_REQUEST_TIMEOUT env variable
Added to .env.local.example and docs/hosting/ai.md
* Rename ALLOWED_MIME_TYPES to ALLOWED_CSV_MIME_TYPES for clarity
* Add comment explaining requires_csv_workflow? predicate
* Remove redundant required_column_keys from PdfImport
Base class already returns [] by default
* Add ENV toggle to disable PDF processing for non-vision endpoints
OPENAI_SUPPORTS_PDF_PROCESSING=false can be used for OpenAI-compatible
endpoints (e.g., Ollama) that don't support vision/PDF processing.
* Wire up transaction extraction for PDF bank statements
- Add extracted_data JSONB column to imports
- Add extract_transactions method to PdfImport
- Call extraction in ProcessPdfJob for bank statements
- Store transactions in extracted_data for later review
* Fix ProcessPdfJob retry logic, sanitize and localize errors
- Allow retries after partial success (classification ok, extraction failed)
- Log sanitized error message instead of raw message to avoid data leakage
- Use i18n for user-facing error messages
* Add vision-capable model validation for PDF processing
* Fix drag-and-drop test to use correct field name csv_file
* Schema bleedover from another branch
* Fix drag-drop import form field name to match controller
* Add vision capability guard to process_pdf method
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
* feat: add valuations API endpoints for managing account reconciliations
* refactor: formatting
* fix: make account extraction clearer
* feat: validation and error handling improvements
* feat: transaction
* feat: error handling
* Add API documentation LLM context
* Make it easier for people
* feat: transaction in creation
* feat: add OpenAPI spec for Valuations API
* fix: update notes validation to check for key presence
* Prevent double render
* All other docs use `apiKeyAuth`
* More `apiKeyAuth`
* Remove testing assertions from API doc specs
* fix: correct valuation entry references
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Use Accept-Language for unauthenticated locale
* Add per-user locale overrides
* Fix test
* Use more than the top `accept-language` entry
* Localization of string
* feat: add protection indicator to entries and unlock functionality
- Introduced protection indicator component rendering on hover and in detail views.
- Added support to unlock entries, clearing protection flags (`user_modified`, `import_locked`, and locked attributes).
- Updated routes, controllers, and models to enable unlock functionality for trades and transactions.
- Refactored views and localized content to support the new feature.
- Added relevant tests for unlocking functionality and attribute handling.
* feat: improve sync protection and turbo stream updates for entries
- Added tests for turbo stream updates reflecting protection indicators.
- Ensured user-modified entries lock specific attributes to prevent overwrites.
- Updated controllers to mark entries as user-modified and reload for accurate rendering.
- Enhanced protection indicator rendering using turbo frames.
- Applied consistent lock state handling across trades and transactions.
* Address PR review comments for protection indicator
---------
Co-authored-by: luckyPipewrench <luckypipewrench@proton.me>
* Hide payment contribution options from demo and manually created users
Demo data users and manually created users don't have stripe_customer_id
set on their family, so they should not see payment/contribution options.
Changes:
- Add can_manage_subscription? method to Family::Subscribeable that checks
for presence of stripe_customer_id
- Guard Settings::PaymentsController to return 403 for users without
stripe_customer_id
- Guard SubscriptionsController#show action (Stripe portal redirect) for
users without stripe_customer_id
- Update settings navigation to hide the payment link when
stripe_customer_id is not present
- Add tests for the new behavior
* Fix broken test
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Add files via upload
Signed-off-by: Jose <39016041+jospaquim@users.noreply.github.com>
* Add merchants and tags resources to routes
Signed-off-by: Jose <39016041+jospaquim@users.noreply.github.com>
* update
* update spaces
* fix: Apply CodeRabbit suggestions and add YARD documentation
* docs: Add API documentation for merchants and tags endpoints
* fix: Address CodeRabbit feedback on documentation
* fix: Use authorize_scope! instead of ensure_read_scope
* test(api): Add request specs for merchants and tags endpoints
* test(api): Add request specs for merchants and tags endpoints
* test(api): Convert specs to Minitest format in test/
* fix: Correct indentation for private methods
* fix: merchant and tag test
* Enhance tag tests for family scope and access
Added tests to ensure tags from other families are not returned and that attempts to access them return 404.
Signed-off-by: Jose <39016041+jospaquim@users.noreply.github.com>
* Enhance merchants controller tests for family scope
Added tests to ensure that merchants from other families are not returned in the index action and that accessing a merchant from another family returns a 404 error.
Signed-off-by: Jose <39016041+jospaquim@users.noreply.github.com>
* Fix test/implementation
* Remove old token test code
* Improve test
---------
Signed-off-by: Jose <39016041+jospaquim@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* **Add Coinbase integration with item and account management**
- Creates migrations for `coinbase_items` and `coinbase_accounts`.
- Adds models, controllers, views, and background tasks to support account linking, syncing, and transaction handling.
- Implements Coinbase API client and adapter for seamless integration.
- Supports ActiveRecord encryption for secure credential storage.
- Adds UI components for provider setup, account management, and synchronization.
* Localize Coinbase-related UI strings, refine account linking for security, and add timeouts to Coinbase API requests.
* Localize Coinbase account handling to support native currencies (USD, EUR, GBP, etc.) across balances, trades, holdings, and transactions.
* Improve Coinbase processing with timezone-safe parsing, native currency support, and immediate holdings updates.
* Improve trend percentage formatting and enhance race condition handling for Coinbase account linking.
* Fix log message wording for orphan cleanup
* Ensure `selected_accounts` parameter is sanitized by rejecting blank entries.
* Add tests for Coinbase integration: account, item, and controller coverage
- Adds unit tests for `CoinbaseAccount` and `CoinbaseItem` models.
- Adds integration tests for `CoinbaseItemsController`.
- Introduces Stimulus `select-all` controller for UI checkbox handling.
- Localizes UI strings and logging for Coinbase integration.
* Update test fixtures to use consistent placeholder API keys and secrets
* Refine `coinbase_item` tests to ensure deterministic ordering and improve scope assertions.
* Integrate `SyncStats::Collector` into Coinbase syncer to streamline statistics collection and enhance consistency.
* Localize Coinbase sync status messages and improve sync summary test coverage.
* Update `CoinbaseItem` encryption: use deterministic encryption for `api_key` and standard for `api_secret`.
* fix schema drift
* Beta labels to lower expectations
---------
Co-authored-by: luckyPipewrench <luckypipewrench@proton.me>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Rename billing to payment throughout the codebase
This change updates terminology from "billing" to "payment" to better
reflect that these are contributions/payments rather than bills.
Changes include:
- Rename BillingsController to PaymentsController
- Rename billing_email to payment_email
- Rename next_billing_date to next_payment_date
- Rename create_billing_portal_session_url to create_payment_portal_session_url
- Update routes from billing to payment
- Update all 12 locale files with new terminology
- Update views, helpers, and tests
* Update app/views/subscriptions/upgrade.html.erb
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
---------
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* feat(settings): split imports and exports
* feat(security): sanitize pagination params to prevent abuse
* fix(settings): fix syntax in settings nav
* feat(settings): internationalize family_exports and imports UI strings
* fix(settings): fix coderabbit review
* fix(settings): fix coderabbit review
* fix(settings): fix coderabbit review
* Change default per_page value from 20 to 10
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
* Add `/family_export` to navigation
* Consistency with old defaults
* Align `safe_per_page` even if not DRY
---------
Signed-off-by: Julien Orain <julien.orain@gmail.com>
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: JulienOrain <your-github-email@example.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* feat: Add subcategory breakdown to Cash Flow and Reports
Implements Discussion #546 - adds hierarchical category/subcategory
visualization to both the Sankey chart and Reports breakdown tables.
Sankey chart changes:
- Income: subcategory → parent category → Cash Flow
- Expense: Cash Flow → parent category → subcategory
- Extracted process_category_totals helper to DRY up income/expense logic
Reports breakdown changes:
- Subcategories display nested under parent categories
- Smaller dots and indented rows for visual hierarchy
- Extracted _breakdown_table partial to eliminate duplication
* fix: Dynamic node padding for Sankey chart with many nodes
- Add dynamic nodePadding calculation to prevent padding from dominating
chart height when there are many subcategory nodes
- Extract magic numbers to static constants for configuration
- Decompose monolithic #draw() into focused methods
- Consolidate duplicate tooltip/currency formatting code
- Modernize syntax with spread operators and optional chaining
* fix: Hide overlapping Sankey labels, show on hover
- Add label overlap detection by grouping nodes by column depth
- Hide labels that would overlap with adjacent nodes
- Show hidden labels on hover (node rectangle or connected links)
- Add hover events to node rectangles (not just text)
* fix: Use deterministic fallback colors for categories
- Replace Category::COLORS.sample with Category::UNCATEGORIZED_COLOR
for income categories in Sankey chart (was producing different colors
on each page load)
- Add nil color fallback in reports_controller for parent and root
categories
Addresses CodeRabbit review feedback.
* fix: Expand CSS variable map for d3 color manipulation
Add hex mappings for commonly used CSS variables so d3 can manipulate
opacity for gradients and hover effects:
- var(--color-destructive) -> #EC2222
- var(--color-gray-400) -> #9E9E9E
- var(--color-gray-500) -> #737373
* test: Add tests for subcategory breakdown in dashboard and reports
- Test dashboard renders Sankey chart with parent/subcategory transactions
- Test reports groups transactions by parent and subcategories
- Test reports handles categories with nil colors
- Use EntriesTestHelper#create_transaction for cleaner test setup
* Fix lint: use Number.NEGATIVE_INFINITY
* Remove obsolete nil color test
Category model now validates color presence, so nil color categories
cannot exist. The fallback handling in reports_controller is still in
place but the scenario is unreachable.
* Update reports_controller.rb
* FIX trade category
---------
Co-authored-by: sokie <sokysrm@gmail.com>
* fix: Allow locale preview on onboarding preferences page
When a user selects a different language on /onboarding/preferences,
the page now immediately displays in the selected language. This is
achieved by checking for a valid locale URL parameter before falling
back to the family's saved locale setting.
* fix: Harden locale param handling and restore locale in tests
- Add type check to ensure params[:locale] is a String before calling
.to_sym, preventing 500 errors from array/hash injection attacks
- Add teardown to tests to restore original locale, preventing test
pollution
* fix: Reload family in teardown to handle update_column
* fix: Remove edge case test that used update_column with nil locale
* fix: Simplify localize tests - rely on fixture defaults and transactional isolation
* fix: Update system test to expect Spanish button text after locale preview
* refactor: Use I18n.t for button text in system test instead of hardcoded string
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Implement dynamic role assignment for new family creators.
Introduced `User.role_for_new_family_creator` to assign `super_admin` to the first user of an instance and a configurable fallback role (e.g., `admin`) to subsequent users. Updated controllers and tests accordingly.
* Update default fallback role for family creators to admin.
---------
Co-authored-by: luckyPipewrench <luckypipewrench@proton.me>
* Add UI and functionality for new user registration via OIDC integration
* Add tests and localization for new user registration via OIDC
---------
Co-authored-by: luckyPipewrench <luckypipewrench@proton.me>
- Introduced `update_access_url!` method to reuse existing SimpleFIN items during reconnections, preserving account linkages.
- Refactored `SimplefinConnectionUpdateJob` to update access URLs in place without creating new items or transferring accounts.
- Adjusted sync logic to leverage `repair_stale_linkages` for seamless reconnections.
- Enhanced `SimplefinItem::Importer` to auto-recover the `good` status if no auth errors are found during sync.
- Updated tests to validate in-place updates and preserved account relationships.
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
* Add cost basis tracking and management to holdings
- Added migration to introduce `cost_basis_source` and `cost_basis_locked` fields to `holdings`.
- Implemented backfill for existing holdings to set `cost_basis_source` based on heuristics.
- Introduced `Holding::CostBasisReconciler` to manage cost basis resolution logic.
- Added user interface components for editing and locking cost basis in holdings.
- Updated `materializer` to integrate reconciliation logic and respect locked holdings.
- Extended tests for cost basis-related workflows to ensure accuracy and reliability.
* Fix cost basis calculation in holdings controller
- Ensure `cost_basis` is converted to decimal for accurate arithmetic.
- Fix conditional check to properly validate positive `cost_basis`.
* Improve cost basis validation and error handling in holdings controller
- Allow zero as a valid cost basis for gifted/inherited shares.
- Add error handling with user feedback for invalid cost basis values.
---------
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
* Implement API v1 Imports controller
- Add Api::V1::ImportsController with index, show, and create actions
- Add Jbuilder views for index and show
- Add integration tests
- Implement row generation logic in create action
- Update routes
* Validate import account belongs to family
- Add validation to Import model to ensure account belongs to the same family
- Add regression test case in Api::V1::ImportsControllerTest
* updating docs to be more detailed
* Rescue StandardError instead of bare rescue in ImportsController
* Optimize Imports API and fix documentation
- Implement rows_count counter cache for Imports
- Preload rows in Api::V1::ImportsController#show
- Update documentation to show correct OAuth scopes
* Fix formatting in ImportsControllerTest
* Permit all import parameters and fix unknown attribute error
* Restore API routes for auth, chats, and messages
* removing pr summary
* Fix trailing whitespace and configured? test failure
- Update Import#configured? to use rows_count for performance and consistency
- Mock rows_count in TransactionImportTest
- Fix trailing whitespace in migration
* Harden security and fix mass assignment in ImportsController
- Handle type and account_id explicitly in create action
- Rename import_params to import_config_params for clarity
- Validate type against Import::TYPES
* Fix MintImport rows_count update and migration whitespace
- Update MintImport#generate_rows_from_csv to update rows_count counter cache
- Fix trailing whitespace and final newline in AddRowsCountToImports migration
* Implement full-screen Drag and Drop CSV import on Transactions page
- Add DragAndDropImport Stimulus controller listening on document
- Add full-screen overlay with icon and text to Transactions index
- Update ImportsController to handle direct file uploads via create action
- Add system test for drag and drop functionality
* Implement Drag and Drop CSV upload on Import Upload page
- Add drag-and-drop-import controller to import/uploads/show
- Add full-screen overlay to import/uploads/show
- Annotate upload form and input with drag-and-drop targets
- Add PR_SUMMARY.md
* removing pr summary
* Add file validation to ImportsController
- Validate file size (max 10MB) and MIME type in create action
- Prevent memory exhaustion and invalid file processing
- Defined MAX_CSV_SIZE and ALLOWED_MIME_TYPES in Import model
* Refactor dragLeave logic with counter pattern to prevent flickering
* Extract shared drag-and-drop overlay partial
- Create app/views/imports/_drag_drop_overlay.html.erb
- Update transactions/index and import/uploads/show to use the partial
- Reduce code duplication in views
* Update Brakeman and harden ImportsController security
- Update brakeman to 7.1.2
- Explicitly handle type assignment in ImportsController#create to avoid mass assignment
- Remove :type from permitted import parameters
* Fix trailing whitespace in DragAndDropImportTest
* Don't commit LLM comments as file
* FIX add api validation
---------
Co-authored-by: Carlos Adames <cj@Carloss-MacBook-Air.local>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: sokie <sokysrm@gmail.com>
* Add stale account detection and handling in SimpleFin setup
- Introduced UI for managing stale accounts during SimpleFin setup.
- Added logic to detect accounts no longer provided by SimpleFin.
- Implemented actions to delete, move transactions, or skip stale accounts.
- Updated `simplefin_items_controller` with stale account processing and handling.
- Enhanced tests to validate stale account scenarios, including detection, deletion, moving transactions, and skipping.
* Update SimpleFin to SimpleFIN in locale file
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
* Silly changes break things ...
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
* Refactor stale account processing and UI handling
- Moved `target_account.sync_later` to execute after commit for proper recalculation of balances.
- Added additional safeguard in JavaScript to check for `moveRadioTarget` before updating target visibility.
* More silly capitalization changes
* Enhance stale account action handling in SimpleFIN setup
- Introduced `permitted_stale_account_actions` to validate and permit nested `stale_account_actions` parameters.
- Updated `complete_account_setup` to use the new method for safer processing.
- Corrected capitalization in SimpleFIN update success and error messages.
* Add error tracking and UI feedback for stale account actions
- Updated `process_stale_account_actions` to track errors for delete and move actions.
- Enhanced UI to display success and error messages for stale account processing.
- Implemented destruction of conflicting transfers during account move to maintain data integrity.
* Refactor transfer destruction and improve SimpleFIN account setup messages
- Updated `simplefin_items_controller` to use `find_each(&:destroy!)` for transfer deletions, ensuring callbacks are invoked.
- Enhanced localization for success messages in account creation to handle singular and plural cases.
---------
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
* Feat(CoinStats): Scaffold implementation, not yet functional
* Feat(CoinStats): Implement crypto wallet balance and transactions
* Feat(CoinStats): Add tests, Minor improvements
* Feat(CoinStats): Utilize bulk fetch API endpoints
* Feat(CoinStats): Migrate strings to i8n
* Feat(CoinStats): Fix error handling in wallet link modal
* Feat(CoinStats): Implement hourly provider sync job
* Feat(CoinStats): Generate docstrings
* Fix(CoinStats): Validate API Key on provider update
* Fix(Providers): Safely handle race condition in merchance creation
* Fix(CoinStats): Don't catch system signals in account processor
* Fix(CoinStats): Preload before iterating accounts
* Fix(CoinStats): Add no opener / referrer to API dashboard link
* Fix(CoinStats): Use strict matching for symbols
* Fix(CoinStats): Remove dead code in transactions importer
* Fix(CoinStats): Avoid transaction fallback ID collisions
* Fix(CoinStats): Improve Blockchains fetch error handling
* Fix(CoinStats): Enforce NOT NULL constraint for API Key schema
* Fix(CoinStats): Migrate sync status strings to i8n
* Fix(CoinStats): Use class name rather than hardcoded string
* Fix(CoinStats): Use account currency rather than hardcoded USD
* Fix(CoinStats): Migrate from standalone to Provider class
* Fix(CoinStats): Fix test failures due to string changes
* Add tests and enhance logic for SimpleFin account synchronization and reconciliation
- Added retry logic with exponential backoff for network errors in `Provider::Simplefin`.
- Introduced tests to verify retry functionality and error handling for rate-limit, server errors, and stale data.
- Updated `SimplefinItem` to detect stale sync status and reconciliation issues.
- Enhanced UI to display stale sync warnings and data integrity notices.
- Improved SimpleFin account matching during updates with multi-tier strategy (ID, fingerprint, fuzzy match).
- Added transaction reconciliation logic to detect data gaps, transaction count drops, and duplicate transaction IDs.
* Introduce `SimplefinConnectionUpdateJob` for asynchronous SimpleFin connection updates
- Moved SimpleFin connection update logic to `SimplefinConnectionUpdateJob` to improve response times by offloading network retries, data fetching, and reconciliation tasks.
- Enhanced SimpleFin account matching with a multi-tier strategy (ID, fingerprint, fuzzy name match).
- Added retry logic and bounded latency for token claim requests in `Provider::Simplefin`.
- Updated tests to cover the new job flow and ensure correct account reconciliation during updates.
* Remove unused SimpleFin account matching logic and improve error handling in `SimplefinConnectionUpdateJob`
- Deleted the multi-tier account matching logic from `SimplefinItemsController` as it is no longer used.
- Enhanced error handling in `SimplefinConnectionUpdateJob` to gracefully handle import failures, ensuring orphaned items can be manually resolved.
- Updated job flow to conditionally set item status based on the success of import operations.
* Fix SimpleFin sync: check both legacy FK and AccountProvider for linked accounts
* Add crypto, checking, savings, and cash account detection; refine subtype selection and linking
- Enhanced `Simplefin::AccountTypeMapper` to include detection for crypto, checking, savings, and standalone cash accounts.
- Improved subtype selection UI with validation and warning indicators for missing selections.
- Updated SimpleFin account linking to handle both legacy FK and `AccountProvider` associations consistently.
- Refined job flow and importer logic for better handling of linked accounts and subtype inference.
* Improve `SimplefinConnectionUpdateJob` and holdings processing logic
- Fixed race condition in `SimplefinConnectionUpdateJob` by moving `destroy_later` calls outside of transactions.
- Updated fuzzy name match logic to use Levenshtein distance for better accuracy.
- Enhanced synthetic ticker generation in holdings processor with hash suffix for uniqueness.
* Refine SimpleFin entry processing logic and ensure `extra` data persistence
- Simplified pending flag determination to rely solely on provider-supplied values.
- Fixed potential stale values in `extra` by ensuring deep merge overwrite with `entry.transaction.save!`.
* Replace hardcoded fallback transaction description with localized string
* Refine pending flag logic in SimpleFin processor tests
- Adjust test to prevent falsely inferring pending status from missing posted dates.
- Ensure provider explicitly sets pending flag for transactions.
* Add `has_many :holdings` association to `AccountProvider` with `dependent: :nullify`
---------
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
Multi-provider SSO support:
- Database-backed SSO provider management with admin UI
- Support for OpenID Connect, Google OAuth2, GitHub, and SAML 2.0
- Flipper feature flag (db_sso_providers) for dynamic provider loading
- ProviderLoader service for YAML or database configuration
Admin functionality:
- Admin::SsoProvidersController for CRUD operations
- Admin::UsersController for super_admin role management
- Pundit policies for authorization
- Test connection endpoint for validating provider config
User provisioning improvements:
- JIT (just-in-time) account creation with configurable default role
- Changed default JIT role from admin to member (security)
- User attribute sync on each SSO login
- Group/role mapping from IdP claims
SSO identity management:
- Settings::SsoIdentitiesController for users to manage connected accounts
- Issuer validation for OIDC identities
- Unlink protection when no password set
Audit logging:
- SsoAuditLog model tracking login, logout, link, unlink, JIT creation
- Captures IP address, user agent, and metadata
Advanced OIDC features:
- Custom scopes per provider
- Configurable prompt parameter (login, consent, select_account, none)
- RP-initiated logout (federated logout to IdP)
- id_token storage for logout
SAML 2.0 support:
- omniauth-saml gem integration
- IdP metadata URL or manual configuration
- Certificate and fingerprint validation
- NameID format configuration
- Add institution name & domain, to allow fetching logos when no provider is configured
- Add free-form textarea for storing misc. notes (eg. sort codes, account numbers)
- Update account settings form to support these new fields
* Add configuration and logic for dynamic SSO provider support and stricter JIT account creation
- Introduced `config/auth.yml` for centralized auth configuration and documentation.
- Added support for multiple SSO providers, including Google, GitHub, and OpenID Connect.
- Implemented stricter JIT SSO account creation modes (`create_and_link` vs `link_only`).
- Enabled optional restriction of JIT creation by allowed email domains.
- Enhanced OmniAuth initializer for dynamic provider setup and better configurability.
- Refined login UI to handle local login disabling and emergency super-admin override.
- Updated account creation flow to respect JIT mode and domain checks.
- Added tests for SSO account creation, login form visibility, and emergency overrides.
# Conflicts:
# app/controllers/sessions_controller.rb
* remove non-translation
* Refactor authentication views to use translation keys and update locale files
- Extracted hardcoded strings in `oidc_accounts/link.html.erb` and `sessions/new.html.erb` into translation keys for better localization support.
- Added missing translations for English and Spanish in `sessions` and `oidc_accounts` locale files.
* Enhance OmniAuth provider configuration and refine local login override logic
- Updated OmniAuth initializer to support dynamic provider configuration with `name` and scoped parameters for Google and GitHub.
- Improved local login logic to enforce stricter handling of super-admin override when local login is disabled.
- Added test for invalid super-admin override credentials.
* Document Google sign-in configuration for local development and self-hosted environments
---------
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
* Update SimpleFIN relinking flow and enhance duplicate account handling
- Updated logic to allow relinking of SimpleFIN accounts while preserving legacy mappings.
- Introduced clean-up logic to hide orphaned duplicate accounts after relinking.
- Enhanced UI to display current mappings for linked accounts.
- Improved test coverage for relinking scenarios and SimpleFIN account visibility.
* Localize SimpleFIN account selection messages and remove hardcoded text
- Added translations for user-facing messages in `select_existing_account` flow (`pt-BR` and `en` locales).
- Replaced hardcoded strings in the view with localized keys.
* Localize Enable Banking and SimpleFIN account linking messages; add support for investment accounts.
- Added translations for Enable Banking and SimpleFIN account linking flows.
- Updated views and controllers to replace hardcoded strings with localized keys.
- Introduced support for investment accounts in `Provider::LunchflowAdapter`.
- Enhanced relinking logic for SimpleFIN accounts and improved test coverage for related scenarios.
---------
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
* Address remaining CodeRabbit comments from PR #267
This commit addresses the remaining unresolved code review comments:
1. Fix down migration in drop_was_merged_from_transactions.rb
- Add null: false, default: false constraints to match original column
- Ensures proper rollback compatibility
2. Fix bare rescue in maps_helper.rb compute_duplicate_only_flag
- Replace bare rescue with rescue StandardError => e
- Add proper logging for debugging
- Follows Ruby best practices by being explicit about exception handling
These changes improve code quality and follow Rails/Ruby best practices.
* Refactor `SimplefinItemsController` and add tests for balances sync and account relinking behavior
- Replaced direct sync execution with `SyncJob` for asynchronous handling of balances sync.
- Updated account relinking logic to prevent disabling accounts with other active provider links.
- Removed unused `compute_relink_candidates` method.
- Added tests to verify `balances` action enqueues `SyncJob` and relinking respects account-provider relationships.
* Refactor balances sync to use runtime-only `balances_only` flag
- Replaced persistent `sync_stats` usage with runtime `balances_only?` predicate via `define_singleton_method`.
- Updated `SimplefinItemsController` `balances` action to pass `balances_only` flag to `SyncJob`.
- Enhanced `SyncJob` to attach transient `balances_only?` flag for execution.
- Adjusted `SimplefinItem::Syncer` logic to rely on the runtime `balances_only?` method.
- Updated controller tests to validate runtime flag usage in `SyncJob`.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
* Add API endpoint for triggering family sync
Introduces Api::V1::SyncController with a create action to queue a family sync, applying all active rules and syncing accounts. Adds corresponding route, JSON response view, and comprehensive controller tests for authorization and response validation.
* Rename started_at to syncing_at in sync API response
Updated the sync create JSON response to use 'syncing_at' instead of 'started_at'. Adjusted related controller test to check for 'syncing_at'. Also updated API authentication header in test to use 'X-Api-Key' instead of Bearer token.
* Update app/controllers/api/v1/sync_controller.rb
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Mark Hendriksen <hendriksen-mark@hotmail.com>
---------
Signed-off-by: Mark Hendriksen <hendriksen-mark@hotmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Move provider config to family
* remove global settings
* Remove turbo auto submit
* Fix flash location
* Fix mssing syncer for lunchflow
* Update schema.rb
* FIX tests and encryption config
* FIX make rabbit happy
* FIX run migration in SQL
* FIX turbo frame modal
* Branding fixes
* FIX rabbit
* OCD with product names
* More OCD
* No other console.log|warn in codebase
---------
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* SimpleFin: metadata + merge fixes; holdings (incl. crypto) + Day Change; Sync Summary; ops rakes; lint
# Conflicts:
# db/schema.rb
# Conflicts:
# app/controllers/simplefin_items_controller.rb
* fix testing
* fix linting
* xfix linting x2
* Review PR #267 on we-promise/sure (SimpleFin enhancements v2). Address all 15 actionable CodeRabbit comments: Add UUID validations in rakes (e.g., simplefin_unlink), swap Ruby pattern matching/loops for efficient DB queries (e.g., where LOWER(name) LIKE ?), generate docstrings for low-coverage areas (31%), consolidate routes for simplefin_items, move view logic to helpers (e.g., format_transaction_extra), strengthen tests with exact assertions/fixtures for dedup/relink failures. Also, check for overlaps with merged #262 (merchants fix): Ensure merchant creation in simplefin_entry/processor.rb aligns with new payee-based flow and MD5 IDs; add tests for edge cases like empty payees or over-merging pendings. Prioritize security (PII redaction in logs, no hardcoded secrets).
* SimpleFin: address CodeRabbit comments (batch 1)
- Consolidate simplefin_items routes under a single resources block; keep URLs stable
- Replace inline JS with Stimulus auto-relink controller; auto-load relink modal via global modal frame
- Improve a11y in relink modal by wrapping rows in labels
- Harden unlink rake: default dry_run=true, UUID validation, redact PII in outputs, clearer errors
- Backfill rake: default dry_run=true, UUID validation; groundwork for per-SFA counters
- Fix-was-merged rake: default dry_run=true, UUID validation; clearer outputs
- Idempotent transfer auto-match (find_or_create_by! + RecordNotUnique rescue)
- Extract SimpleFin error tooltip assembly into helper and use it in view
RuboCop: maintain 2-space indentation, spaces inside array brackets, spaces after commas, and no redundant returns
* Linter noise
* removed filed commited by mistake.
* manual relink flow and tighten composite matching
* enforce manual relink UI; fix adapter keywords; guarantee extra.simplefin hash
* refactor(simplefin): extract relink service; enforce manual relink UI; tighten composite match; migration 7.2
* add provider date parser; refactor rake; move view queries; partial resilience
* run balances-only import in background job. make update flow enqueue balances-only job
* persists across all update redirects and initialize
used_manual_ids to prevent NameError in relink candidate computation.
* SimpleFin: metadata + merge fixes; holdings (incl. crypto) + Day Change; Sync Summary; ops rakes; lint
* Fixed failed test after rebase.
* scan_ruby fix
* Calming the rabbit:
Fix AccountProvider linking when accounts change
Drop the legacy unique index instead of duplicating it
Fix dynamic constant assignment
Use fixtures consistently; avoid rescue for control flow.
Replace bare rescue with explicit exception class.
Move business logic out of the view.
Critical: Transaction boundary excludes recompute phase, risking data loss.
Inconsistency between documentation and implementation for zero-error case.
Refactor to use the compute_unlinked_count helper for consistency.
Fix cleanup task default: it deletes by default.
Move sync stats computation to controller to avoid N+1 queries.
Consolidate duplicate sync query.
Clarify the intent of setting flash notice on the error path.
Fix Date/Time comparison in should_be_inactive?.
Move stats retrieval logic to controller.
Remove duplicate Sync summary section.
Remove the unnecessary sleep statement; use Capybara's built-in waiting.
Add label wrappers for accessibility and consistency.
* FIX SimpleFIN new account modal
Now new account properly loads as a Modal, instead of new page.
Fixes also form showing dashboard instead of settings page.
* Remove SimpleFin legacy UI components, migrate schema, and refine linking behavior.
# Conflicts:
# app/helpers/settings_helper.rb
* Extract SimpleFin-related logic to `prepare_show_context` helper and refactor for consistency. Adjust conditional checks and ensure controller variables are properly initialized.
* Remove unused SimpleFin maps from prepare_show_context; select IDs to avoid N+1
Replace Tailwind bg-green-500 with semantic bg-success in _simplefin_panel/_provider_form
Add f.label :setup_token in simplefin_items/new for a11y
Remove duplicate require in AccountsControllerSimplefinCtaTest
* Remove unnecessary blank lines
* Reduce unnecessary changes
This reduces the diff against main
* Simplefin Account Setup: Display in modal
This fixes an issue with the `X` dismiss button in the top right corner
* Removed unnecessary comment.
* removed unnecessary function.
* fixed broken links
* Removed unnecessary file
* changed to database query
* set to use UTC and gaurd against null
* set dry_run=true
* Fixed comment
* Changed to use a database-level query
* matched test name to test behavior.
* Eliminate code duplication and Time.zone dependency
* make final summary surface failures
* lint fix
* Revised timezone comment. better handle missing selectors.
* sanitized LIKE wildcards
* Fixed SimpleFin import to avoid “Currency can’t be blank” validation failures when providers return an empty currency string.
* Added helper methods for admin and self-hosted checks
* Specify exception types in rescue clauses.
* Refined logic to determine transaction dates for credit accounts.
* Refined stats calculation for `total_accounts` to track the maximum unique accounts per run instead of accumulating totals.
* Moved `unlink_all!` logic to `SimplefinItem::Unlinking` concern and deprecated `SimplefinItem::Unlinker`. Updated related references.
* Refined legacy unlinking logic, improved `current_holdings` formatting, and added ENV-based overrides for self-hosted checks.
* Enhanced `unlink_all!` with explicit error handling, improved transaction safety, and refined ENV-based self-hosted checks. Adjusted exception types and cleaned up private method handling.
* Improved currency assignment logic by adding fallback to `current_account` and `family` currencies.
* Enhanced error tracking during SimpleFin account imports by adding categorized error buckets, limiting stored errors to the last 5, and improving `stats` calculations.
* typo fix
* Didn't realize rabbit was still mad...
Refactored SimpleFin error handling and CTA logic: centralized duplicate detection and relink visibility into controller, improved task counters, adjusted redirect notices, and fixed form indexing.
* Dang rabbit never stops... Centralized SimpleFin maps logic into `MapsHelper` concern and integrated it into relevant controllers and rake tasks. Optimized queries, reduced redundancy, and improved unlinked counts and manual account checks with batch processing. Adjusted task arguments for clarity.
* Persistent rabbit. Optimized SimpleFin maps logic by implementing batch queries for manual account and unlinked count checks, reducing N+1 issues. Improved clarity of rake task argument descriptions and error messages for better usability.
* Lost a commit somehow, resolved here. Refactored transaction extra details logic by introducing `build_transaction_extra_details` helper to improve clarity, reusability, and reduce view complexity. Enhanced rake tasks with strict dry-run validation and better error handling. Updated schema to allow nullable `merchant_id` and added conditional unique indexes for recurring transactions.
* Refactored sensitive data redaction in `simplefin_unlink` task for recursive handling, optimized SQL sanitization in `simplefin_holdings_backfill`, improved error handling in `transactions_helper`, and streamlined day change calculation logic in `Holding` model.
* Lint fix
* Removed per PR comments.
* Also removing per PR comment.
* git commit -m "SimpleFIN polish: preserve #manual-accounts wrapper, unify \"manual\" scope, and correct unlinked counts
- Preserve #manual-accounts wrapper: switch non-empty updates to turbo_stream.update and background broadcast_update_to; keep empty-path replace to render <div id=\"manual-accounts\"></div>
- Unify definition of manual accounts via Account.visible_manual (visible + legacy-nil + no AccountProvider); reuse in controllers, jobs, and helper
- Correct setup/unlinked counts: SimplefinItem::Syncer#finalize_setup_counts and maps now consider AccountProvider links (legacy account AND provider must be absent)
Deleted:
- app/models/simplefin_item/relink_service.rb
- app/controllers/concerns/simplefin_items/relink_helpers.rb
- app/javascript/controllers/auto_relink_controller.js
- app/views/simplefin_items/_relink_modal.html.erb
- app/views/simplefin_items/manual_relink.html.erb
- app/views/simplefin_items/relink.html.erb
- test/services/simplefin_item/relink_service_test.rb
Refs: PR #318 unified link/unlink; PR #267 SimpleFIN; follow-up to fix wrapper ID loss and counting drift."
* Extend unlinked account check to include "Investment" type
* set SimpleFIN item for `balances`, remove redundant unpacking, and improve holdings task error
* SimpleFIN: add `errors` action + modal; do not reintroduce legacy relink actions; removed dead helper
* FIX simpleFIN linking
* Add delay back, tests benefit from it
* Put cache back in
* Remove empty `rake` task
* Small spelling fixes.
---------
Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: sokie <sokysrm@gmail.com>
Co-authored-by: Dylan Corrales <deathcamel58@gmail.com>
* Add support to unlink lunch flow accounts
* add support to link and unlink to any provider
* Fix tests and query
* Let's keep Amr happy about his brand
* Wrap unlink operations in a transaction and add error handling.
* Fix tests
---------
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* First reporting version
* Fixes for all tabs
* Transactions table
* Budget section re-design
* FIX exports
Fix transactions table aggregation
* Add support for google sheets
Remove pdf and xlsx for now
* Multiple fixes
- Trends & Insights now follows top filter
- Transactions Breakdown removed filters, implemented sort by amount.
- The entire section follows top filters.
- Export to CSV adds per month breakdown
* Linter and tests
* Fix amounts
- Correctly handle amounts across the views and controller.
- Pass proper values to do calculation on, and not loose precision
* Update Gemfile.lock
* Add support for api-key on reports
Also fix custom date filter
* Review fixes
* Move budget status calculations out of the view.
* fix ensures that quarterly reports end at the quarter boundary
* Fix bugdet days remaining
Fix raw css style
* Fix test
* Implement google sheets properly with hotwire
* Improve UX on period comparison
* FIX csv export for non API key auth
* FIX Read-Modify-Write issue with dynamic fields
Ruby caching + queueing updates might cause some dynamic fields to not be updated.
* Small fix for true dynamic fields
* Add suite of tests for new settings page
* Treat nil values as deletions to keep the hash clean
* Test fix
* Implement Yahoo Finance
* Added tests
* Updated hosting controller to check for managed app_mode instead of env_override
* Suggestions from CodeRabbit and Fixes on tests
* Remove Css changes
* Fix yahoo finance impl and i18n
* Updated view to use healthy method
* remove usage
* Updated env example
* keep usage on class just to keep same format
* Ci test
* Remove some useless validations
* Remove logs
* Linter fixes
* Broke this in my conflict merge
* Wrong indentation level
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
