Commit Graph

117 Commits

Author SHA1 Message Date
David Gil
ba442d5f26 Implement Indexa Capital provider with real API integration (#933)
* Add Indexa Capital provider scaffold

Generate Indexa Capital provider scaffolding and align credential fields with the API authentication requirements.

* Fix PR 926 lint and schema CI failures

* Implement Indexa Capital provider with real API integration

- Rewrite all broken view templates (were meta-ERB from code generator)
- Create missing select_accounts.html.erb template
- Implement real API calls: list_accounts via /users/me, get_holdings
  via /accounts/{number}/fiscal-results, get_account_balance via
  /accounts/{number}/performance
- Add API token auth support (stored token > env token > credentials)
- Add api_token column with encryption support
- Redesign settings panel: API token prominent, credentials collapsible
- Fix account balances display using performance endpoint portfolios
- Fix accounts index empty-state guard missing indexa_capital_items
- Simplify activities fetch job (no activities API endpoint exists)
- Fix i18n interpolation (%%{ -> %{) throughout locale file

* Add tests for Indexa Capital provider integration

- IndexaCapitalItemTest: validations, credentials, scopes, sync status
- IndexaCapitalAccountTest: upsert, holdings, account provider linking
- Provider::IndexaCapitalTest: auth modes, API stubs, error handling
- IndexaCapitalItemsControllerTest: CRUD, setup, linking, authorization
- Fixtures for items (token + credentials) and accounts (mutual + pension)

52 tests, 98 assertions, 0 failures

* Address code review feedback from PR #933

- Fix zero balance bug: use `nil?` instead of `present?` so 0 is stored
- Fix has_indexa_capital_credentials? to check api_token (was ignored)
- Fix build_provider to delegate to Provided concern (was ignoring token)
- Fix IndexaCapital section outside encryption_error guard in settings
- Add account_number sanitization to prevent path traversal in API URLs
- Replace all skipped processor tests with real working tests
- Add zero-balance and path-traversal test coverage

61 tests, 107 assertions, 0 failures

* Address code review round 2: credentials validation, RuboCop, test quality

- Fix RuboCop SpaceInsideArrayLiteralBrackets in credentials check
- Chain where.not calls so all three username/document/password must be present
- Require all three credentials (||) instead of any one (&&) in validate_configuration!
- Move attr_reader to private to avoid exposing credentials publicly
- Parse dates with Date.parse in extract_balance for robustness
- Remove stale TODO and Crypto from supported_account_types
- Order build_provider query deterministically by created_at
- Replace no-op holdings assertion with meaningful assert_difference

* Address code review round 3: JSON parse safety and test precision

- Rescue JSON::ParserError on 2xx responses for clearer error messages
- Fix weak balance assertion: set balance to 0 before processing, assert
  expected value (27093.01 = sum of holdings amounts)

* Include Indexa Capital in automatic family sync

Add indexa_capital_items to Family::Syncer#child_syncables so balances
and holdings refresh on daily auto-sync and login sync, not only on
manual sync button clicks.

---------

Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2026-02-08 18:19:37 +01:00
BitToby
ba6e286b41 feat: add SSL_CA_FILE and SSL_VERIFY environment variables to support… (#894)
* feat: add SSL_CA_FILE and SSL_VERIFY environment variables to support self-signed certificates in self-hosted environments

* fix: NoMethodError by defining SSL helper methods before configure block executes

* refactor: Refactor SessionsController to use shared SslConfigurable module and simplify SSL initializer redundant checks

* refactor: improve SSL configuration robustness and error detection accuracy

* fix:HTTParty SSL options, add file validation guards, prevent Tempfile GC, and redact URLs in error logs

* fix:  Fix SSL concern indentation and stub Simplefin POST correctly in tests

* fix: normalize ssl_verify to always return boolean instead of nil

* fix: solve failing SimpleFin test

* refactor:  trim unused error-handling code from SslConfigurable, replace Tempfile with fixed-path CA bundle, fix namespace pollution in initializers, and add unit tests for core SSL configuration and Langfuse CRL callback.

* fix: added require ileutils in the initializer and require ostruct in the test file.

* fix: solve autoload conflict that broke provider loading, validate all certs in PEM bundles, and add missing requires.
2026-02-06 18:04:03 +01:00
MkDev11
0afdb1d0fd Feature/pdf import transaction rows (#846)
* Add import row generation from PDF extracted data

- Add generate_rows_from_extracted_data method to PdfImport
- Add import! method to create transactions from PDF rows
- Update ProcessPdfJob to generate rows after extraction
- Update configured?, cleaned?, publishable? for PDF workflow
- Add column_keys, required_column_keys, mapping_steps
- Set bank statements to pending status for user review
- Add tests for new functionality

Closes #844

* Add tests for BankStatementExtractor

- Test transaction extraction from PDF content
- Test deduplication across chunk boundaries
- Test amount normalization for various formats
- Test graceful handling of malformed JSON responses
- Test error handling for empty/nil PDF content

* Fix supports_pdf_processing? to validate effective model

The validation was always checking @default_model, but process_pdf
allows overriding the model via parameter. This could cause a
vision-capable override model to be rejected, or a non-vision-capable
override to pass validation only to fail during processing.

Changes:
- supports_pdf_processing? now accepts optional model parameter
- process_pdf passes effective model to validation
- Raise Provider::Openai::Error inside with_provider_response for
  consistent error handling

Addresses review feedback from PR#808

* Fix insert_all! bug: explicitly set import_id

Rails insert_all! on associations does NOT auto-set the foreign key.
Added import_id explicitly and use Import::Row.insert_all! directly.
Also reload rows before counting to ensure accurate count.

* Fix pending status showing as processing for bank statements with rows

When bank statement PDF imports have extracted rows, show a 'Ready for Review'
screen with a link to the confirm path instead of the 'Processing' spinner.

This addresses the PR feedback that users couldn't reach the review flow even
though rows were created.

* Gate publishable? on account.present? to prevent import failure

PDF imports are created without an account, and import! raises if account
is missing. This prevents users from hitting publish and having the job fail.

* Wrap generate_rows_from_extracted_data in transaction for atomicity

- Clear rows and reset count even when no transactions extracted
- Use transaction block to prevent partial updates on failure
- Use mapped_rows.size instead of reload for count

* Localize transactions count string with i18n helper

* Add AccountMapping step for PDF imports when account is nil

PDF imports need account selection before publishing. This adds
Import::AccountMapping to mapping_steps when account is nil,
matching the behavior of TransactionImport and TradeImport.

Addresses PR#846 feedback about account selection for PDF imports.

* Only include CategoryMapping when rows have non-empty categories

PDF extraction doesn't extract categories from bank statements,
so the CategoryMapping step would show empty. Now we only include
CategoryMapping if rows actually have non-empty category values.

This prevents showing an empty mapping step for PDF imports.

* Fix PDF import UI flow and account selection

- Add direct account selection in PDF import UI instead of AccountMapping
- AccountMapping designed for CSV imports with multiple account values
- PDF imports need single account for all transactions
- Add update action and route for imports controller
- Fix controller to handle pdf_import param format from form_with
- Show Publish button when import is publishable (account set)
- Fix stepper nav: Upload/Configure/Clean non-clickable for PDF imports
- Redirect PDF imports from configuration step (auto-configured)
- Improve AI prompt to recognize M-PESA/mobile money as bank statements
- Fix migration ordering for import_rows table columns

* Add guard for invalid account_id in imports#update

Prevents silently clearing account when invalid ID is passed.
Returns error message instead of confusing 'Account saved' notice.

* Localize step names in import nav and add account guard

- Use t() helper for all step names (Upload, Configure, Clean, Map, Confirm)
- Add guard for invalid account_id in imports#update
- Prevents silently clearing account when invalid ID is passed

* Make category column migrations idempotent

Check if columns exist before adding to prevent duplicate column
errors when migrations are re-run with new timestamps.

* Add match_path for PDF import step highlighting

Fixes step detection when path is nil by using separate match_path
for current step highlighting while keeping links disabled.

* Rename category migrations and update to Rails 7.2

- Rename class to EnsureCategoryFieldsOnImportRows to avoid conflicts
- Rename class to EnsureCategoryIconOnImportRows
- Update migration version from 7.1 to 7.2 per guidelines
- Rename files to match class names
- Add match_path for PDF import step highlighting

* Use primary (black) style for Create Account and Save buttons

* Remove match_path from auto-completed PDF steps

Only step 4 (Confirm) needs match_path for active-step detection.
Steps 1-3 are purely informational and always complete.

* Add fallback for document type translation

Handles nil or unexpected document_type values gracefully.
Also removes match_path from auto-completed PDF steps.

* Use index-based step number for mobile indicator

Fixes 'Step 5 of 4' issue when Map step is dynamically removed.

* Fix hostings_controller_test: use blank? instead of nil

Setting returns empty string not nil for unset values.

* Localize step progress label and use design token

* Fix button styling: use design system Tailwind classes

btn--primary and btn--secondary CSS classes don't exist.
Use actual design system classes from DS::Buttonish.

* Fix CRLF line endings in tags_controller_test.rb

---------

Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
2026-02-02 16:27:02 +01:00
MkDev11
6f8858b1a6 feat/Add AI-Powered Bank Statement Import (step 1, PDF import & analysis) (#808)
* feat: Add PDF import with AI-powered document analysis

This enhances the import functionality to support PDF files with AI-powered
document analysis. When a PDF is uploaded, it is processed by AI to:
- Identify the document type (bank statement, credit card statement, etc.)
- Generate a summary of the document contents
- Extract key metadata (institution, dates, balances, transaction count)

After processing, an email is sent to the user asking for next steps.

Key changes:
- Add PdfImport model for handling PDF document imports
- Add Provider::Openai::PdfProcessor for AI document analysis
- Add ProcessPdfJob for async PDF processing
- Add PdfImportMailer for user notification emails
- Update imports controller to detect and handle PDF uploads
- Add PDF import option to the new import page
- Add i18n translations for all new strings
- Add comprehensive tests for the new functionality

* Add bank statement import with AI extraction

- Create ImportBankStatement assistant function for MCP
- Add BankStatementExtractor with chunked processing for small context windows
- Register function in assistant configurable
- Make PdfImport#pdf_file_content public for extractor access
- Increase OpenAI request timeout to 600s for slow local models
- Increase DB connection pool to 20 for concurrent operations

Tested with M-Pesa bank statement via remote Ollama (qwen3:8b):
- Successfully extracted 18 transactions
- Generated CSV and created TransactionImport
- Works with 3000 char chunks for small context windows

* Add pdf-reader gem dependency

The BankStatementExtractor uses PDF::Reader to parse bank statement
PDFs, but the gem was not properly declared in the Gemfile. This would
cause NameError in production when processing bank statements.

Added pdf-reader ~> 2.12 to Gemfile dependencies.

* Fix transaction deduplication to preserve legitimate duplicates

The previous deduplication logic removed ALL duplicate transactions based
on [date, amount, name], which would drop legitimate same-day duplicates
like multiple ATM withdrawals or card authorizations.

Changed to only deduplicate transactions that appear in consecutive chunks
(chunking artifacts) while preserving all legitimate duplicates within the
same chunk or non-adjacent chunks.

* Refactor bank statement extraction to use public provider method

Address code review feedback:
- Add public extract_bank_statement method to Provider::Openai
- Remove direct access to private client via send(:client)
- Update ImportBankStatement to use new public method
- Add require 'set' to BankStatementExtractor
- Remove PII-sensitive content from error logs
- Add defensive check for nil response.error
- Handle oversized PDF pages in chunking logic
- Remove unused process_native and process_generic methods
- Update email copy to reflect feature availability
- Add guard for nil document_type in email template
- Document pdf-reader gem rationale in Gemfile

Tested with both OpenAI (gpt-4o) and Ollama (qwen3:8b):
- OpenAI: 49 transactions extracted in 30s
- Ollama: 40 transactions extracted in 368s
- All encapsulation and error handling working correctly

* Update schema.rb with ai_summary and document_type columns

* Address PR #808 review comments

- Rename :csv_file to :import_file across controllers/views/tests
- Add PDF test fixture (sample_bank_statement.pdf)
- Add supports_pdf_processing? method for graceful degradation
- Revert unrelated database.yml pool change (600->3)
- Remove month_start_day schema bleed from other PR
- Fix PdfProcessor: use .strip instead of .strip_heredoc
- Add server-side PDF magic byte validation
- Conditionally show PDF import option when AI provider available
- Fix ProcessPdfJob: sanitize errors, handle update failure
- Move pdf_file attachment from Import to PdfImport
- Document deduplication logic limitations
- Fix ImportBankStatement: catch specific exceptions only
- Remove unnecessary require 'set'
- Remove dead json_schema method from PdfProcessor
- Reduce default OpenAI timeout from 600s to 60s
- Fix nil guard in text mailer template
- Add require 'csv' to ImportBankStatement
- Remove Gemfile pdf-reader comment

* Fix RuboCop indentation in ProcessPdfJob

* Refactor PDF import check to use model predicate method

Replace is_a?(PdfImport) type check with requires_csv_workflow? predicate
that leverages STI inheritance for cleaner controller logic.

* Fix missing 'unknown' locale key and schema version mismatch

- Add 'unknown: Unknown Document' to document_types locale
- Fix schema version to match latest migration (2026_01_24_180211)

* Document OPENAI_REQUEST_TIMEOUT env variable

Added to .env.local.example and docs/hosting/ai.md

* Rename ALLOWED_MIME_TYPES to ALLOWED_CSV_MIME_TYPES for clarity

* Add comment explaining requires_csv_workflow? predicate

* Remove redundant required_column_keys from PdfImport

Base class already returns [] by default

* Add ENV toggle to disable PDF processing for non-vision endpoints

OPENAI_SUPPORTS_PDF_PROCESSING=false can be used for OpenAI-compatible
endpoints (e.g., Ollama) that don't support vision/PDF processing.

* Wire up transaction extraction for PDF bank statements

- Add extracted_data JSONB column to imports
- Add extract_transactions method to PdfImport
- Call extraction in ProcessPdfJob for bank statements
- Store transactions in extracted_data for later review

* Fix ProcessPdfJob retry logic, sanitize and localize errors

- Allow retries after partial success (classification ok, extraction failed)
- Log sanitized error message instead of raw message to avoid data leakage
- Use i18n for user-facing error messages

* Add vision-capable model validation for PDF processing

* Fix drag-and-drop test to use correct field name csv_file

* Schema bleedover from another branch

* Fix drag-drop import form field name to match controller

* Add vision capability guard to process_pdf method

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
2026-01-30 20:44:25 +01:00
MkDev11
eeff4edbea Add warning for TwelveData plan-restricted tickers (#803)
* Add warning for TwelveData plan-restricted tickers

Fixes #800

- Add Security::PlanRestrictionTracker concern using Rails cache
- Detect plan upgrade errors during Security::Price::Importer sync
- Display amber warning on /settings/hosting with affected tickers
- Include unit tests for the new functionality

* Scope plan restriction cache by provider

Addresses review feedback:
- Cache key now includes provider name to support multiple data providers
- Methods now require provider parameter for proper scoping
- Added tests for provider-scoped restrictions
- Added documentation explaining instance-level API key architecture

* Fix RuboCop array bracket spacing

* Fix empty array bracket spacing

* Move plan upgrade detection to Provider::TwelveData

* Fix provider scoping tests to use direct cache writes

---------

Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
2026-01-27 15:45:50 +01:00
Juan José Mata
6b5a5b1877 fix: Show cancellation message when subscription is pending cancellation (#752)
* fix: Show cancellation message when subscription is pending cancellation

When a subscription is cancelled via Stripe, the UI incorrectly showed
"Your contribution continues on..." instead of reflecting the cancellation
status. This fix adds tracking of `cancel_at_period_end` from Stripe webhooks
and displays "Your contribution ends on..." when a subscription has been
cancelled but is still active until the billing period ends.

https://claude.ai/code/session_01Y8ELTdK1k9o315iSq43TRN

* chore: Update schema.rb with cancel_at_period_end column

https://claude.ai/code/session_01Y8ELTdK1k9o315iSq43TRN

* Schema version

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-01-23 18:55:51 +01:00
AdamWHY2K
3f5fff27ea feat: process pending transactions from lunchflow (#731)
* feat(config): add Lunchflow runtime configuration flags

* feat(api): add include_pending parameter to Lunchflow API

* feat(processor): add pending metadata support to Lunchflow processor

* feat(processor): generate temporary IDs for pending transactions

* feat(importer): integrate pending transaction support in sync

* fix(importer): improve deduplication for transactions without IDs

* feat(model): add Lunchflow pending support to Transaction scopes

* test: add Lunchflow processor pending metadata tests

* docs: update AGENTS.md for Lunchflow pending support

* chore: remove unused variable

* fix: simplify key check

* fix: dotenv-linter key order

* fix: avoid collapsing distinct pending transactions

* fix: prevent unbounded raw payload growth for blank IDs
2026-01-23 00:53:24 +01:00
LPW
a83f70425f Add SnapTrade brokerage integration with full trade history support (#737)
* Introduce SnapTrade integration with models, migrations, views, and activity processing logic.

* Refactor SnapTrade activities processing: improve activity fetching flow, handle pending states, and update UI elements for enhanced user feedback.

* Update Brakeman ignore file to include intentional redirect for SnapTrade OAuth portal.

* Refactor SnapTrade models, views, and processing logic: add currency extraction helper, improve pending state handling, optimize migration checks, and enhance user feedback in UI.

* Remove encryption for SnapTrade `snaptrade_user_id`, as it is an identifier, not a secret.

* Introduce `SnaptradeConnectionCleanupJob` to asynchronously handle SnapTrade connection cleanup and improve i18n for SnapTrade item status messages.

* Update SnapTrade encryption: make `snaptrade_user_secret` non-deterministic to enhance security.

---------

Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: luckyPipewrench <luckypipewrench@proton.me>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2026-01-22 20:52:49 +01:00
soky srm
179552657c Mercury integration (#723)
* Initial mercury impl

* FIX both mercury and generator class

* Finish mercury integration and provider generator

* Fix schema

* Fix linter and tags

* Update routes.rb

* Avoid schema drift

---------

Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2026-01-22 20:37:07 +01:00
LPW
dd991fa339 Add Coinbase exchange integration with CDP API support (#704)
* **Add Coinbase integration with item and account management**
- Creates migrations for `coinbase_items` and `coinbase_accounts`.
- Adds models, controllers, views, and background tasks to support account linking, syncing, and transaction handling.
- Implements Coinbase API client and adapter for seamless integration.
- Supports ActiveRecord encryption for secure credential storage.
- Adds UI components for provider setup, account management, and synchronization.

* Localize Coinbase-related UI strings, refine account linking for security, and add timeouts to Coinbase API requests.

* Localize Coinbase account handling to support native currencies (USD, EUR, GBP, etc.) across balances, trades, holdings, and transactions.

* Improve Coinbase processing with timezone-safe parsing, native currency support, and immediate holdings updates.

* Improve trend percentage formatting and enhance race condition handling for Coinbase account linking.

* Fix log message wording for orphan cleanup

* Ensure `selected_accounts` parameter is sanitized by rejecting blank entries.

* Add tests for Coinbase integration: account, item, and controller coverage

- Adds unit tests for `CoinbaseAccount` and `CoinbaseItem` models.
- Adds integration tests for `CoinbaseItemsController`.
- Introduces Stimulus `select-all` controller for UI checkbox handling.
- Localizes UI strings and logging for Coinbase integration.

* Update test fixtures to use consistent placeholder API keys and secrets

* Refine `coinbase_item` tests to ensure deterministic ordering and improve scope assertions.

* Integrate `SyncStats::Collector` into Coinbase syncer to streamline statistics collection and enhance consistency.

* Localize Coinbase sync status messages and improve sync summary test coverage.

* Update `CoinbaseItem` encryption: use deterministic encryption for `api_key` and standard for `api_secret`.

* fix schema drift

* Beta labels to lower expectations

---------

Co-authored-by: luckyPipewrench <luckypipewrench@proton.me>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2026-01-21 22:56:39 +01:00
Juan José Mata
8e36c8e736 Rename billing to payment throughout the codebase (#726)
* Rename billing to payment throughout the codebase

This change updates terminology from "billing" to "payment" to better
reflect that these are contributions/payments rather than bills.

Changes include:
- Rename BillingsController to PaymentsController
- Rename billing_email to payment_email
- Rename next_billing_date to next_payment_date
- Rename create_billing_portal_session_url to create_payment_portal_session_url
- Update routes from billing to payment
- Update all 12 locale files with new terminology
- Update views, helpers, and tests

* Update app/views/subscriptions/upgrade.html.erb

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Juan José Mata <jjmata@jjmata.com>

---------

Signed-off-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-01-21 19:06:00 +01:00
Tobi Demeco
c0bc695ee0 fix: 🐛 properly return cached securities info wrapped in Provider::Response object 2026-01-17 12:24:31 -03:00
soky srm
0e22c60286 FIX Yahoo issues (#636)
* FIX Yahoo issues

* Update yahoo_finance_test.rb

* FIX proper cookie access
2026-01-13 16:46:07 +01:00
Ethan
d354ce48e1 Implement Requested Fixes for CoinStats Integration (#621)
* Fix(CoinStats): Add provider network exception handling

* Fix(CoinStats): Don't expose HTTP response to user

* Fix(CoinStats): Migrate syncer strings to locale files
2026-01-12 09:27:00 +01:00
soky srm
a5bccaf2a1 Generic LLMs improvements (#605)
* Generic LLMs improvements

  1. app/models/provider/openai/auto_categorizer.rb:535-540
  - If description is blank, now falls back to merchant name

  2. app/models/provider/openai/auto_merchant_detector.rb:492-498
  - Now includes merchant first, then description (joined with " - ")

* FIX linter
2026-01-11 10:32:03 +01:00
soky srm
5750e69acf Provider investment fixes (#600)
* FIX issue with stock price retrieval on weekend

* make weekend provisional and increase lookback

* FIX query error

* fix gap fill

The bug: When a price is provisional but the provider doesn't return a new value (weekends), we fall back to the existing DB value instead of gap-filling from Friday's correct price.

* Update importer.rb

Align provider fetch to use PROVISIONAL_LOOKBACK_DAYS for consistency. In the DB fallback, derive currency from provider_prices or db_prices and filter the query accordingly.

* Update 20260110122603_mark_suspicious_prices_provisional.rb

* Delete db/migrate/20260110122603_mark_suspicious_prices_provisional.rb

Signed-off-by: soky srm <sokysrm@gmail.com>

* Update importer.rb

* FIX tests

* FIX last tests

* Update importer_test.rb

The test doesn't properly force effective_start_date to skip old dates because there are many missing dates between the old date and recent dates. Let me fix it to properly test the subset processing scenario.

---------

Signed-off-by: soky srm <sokysrm@gmail.com>
2026-01-10 15:43:07 +01:00
soky srm
ca4fb7995c Implement holdings for lunch flow (#590)
* Implement holdings for lunch flow

* Implement holdings function call
2026-01-09 13:14:14 +01:00
Ethan
3b4ab735b0 Add (beta) CoinStats Crypto Wallet Integration with Balance and Transaction Syncing (#512)
* Feat(CoinStats): Scaffold implementation, not yet functional

* Feat(CoinStats): Implement crypto wallet balance and transactions

* Feat(CoinStats): Add tests, Minor improvements

* Feat(CoinStats): Utilize bulk fetch API endpoints

* Feat(CoinStats): Migrate strings to i8n

* Feat(CoinStats): Fix error handling in wallet link modal

* Feat(CoinStats): Implement hourly provider sync job

* Feat(CoinStats): Generate docstrings

* Fix(CoinStats): Validate API Key on provider update

* Fix(Providers): Safely handle race condition in merchance creation

* Fix(CoinStats): Don't catch system signals in account processor

* Fix(CoinStats): Preload before iterating accounts

* Fix(CoinStats): Add no opener / referrer to API dashboard link

* Fix(CoinStats): Use strict matching for symbols

* Fix(CoinStats): Remove dead code in transactions importer

* Fix(CoinStats): Avoid transaction fallback ID collisions

* Fix(CoinStats): Improve Blockchains fetch error handling

* Fix(CoinStats): Enforce NOT NULL constraint for API Key schema

* Fix(CoinStats): Migrate sync status strings to i8n

* Fix(CoinStats): Use class name rather than hardcoded string

* Fix(CoinStats): Use account currency rather than hardcoded USD

* Fix(CoinStats): Migrate from standalone to Provider class

* Fix(CoinStats): Fix test failures due to string changes
2026-01-07 15:59:04 +01:00
LPW
c12c585a0e Harden SimpleFin sync: retries, safer imports, manual relinking, and data-quality reconciliation (#544)
* Add tests and enhance logic for SimpleFin account synchronization and reconciliation

- Added retry logic with exponential backoff for network errors in `Provider::Simplefin`.
- Introduced tests to verify retry functionality and error handling for rate-limit, server errors, and stale data.
- Updated `SimplefinItem` to detect stale sync status and reconciliation issues.
- Enhanced UI to display stale sync warnings and data integrity notices.
- Improved SimpleFin account matching during updates with multi-tier strategy (ID, fingerprint, fuzzy match).
- Added transaction reconciliation logic to detect data gaps, transaction count drops, and duplicate transaction IDs.

* Introduce `SimplefinConnectionUpdateJob` for asynchronous SimpleFin connection updates

- Moved SimpleFin connection update logic to `SimplefinConnectionUpdateJob` to improve response times by offloading network retries, data fetching, and reconciliation tasks.
- Enhanced SimpleFin account matching with a multi-tier strategy (ID, fingerprint, fuzzy name match).
- Added retry logic and bounded latency for token claim requests in `Provider::Simplefin`.
- Updated tests to cover the new job flow and ensure correct account reconciliation during updates.

* Remove unused SimpleFin account matching logic and improve error handling in `SimplefinConnectionUpdateJob`

- Deleted the multi-tier account matching logic from `SimplefinItemsController` as it is no longer used.
- Enhanced error handling in `SimplefinConnectionUpdateJob` to gracefully handle import failures, ensuring orphaned items can be manually resolved.
- Updated job flow to conditionally set item status based on the success of import operations.

* Fix SimpleFin sync: check both legacy FK and AccountProvider for linked accounts

* Add crypto, checking, savings, and cash account detection; refine subtype selection and linking

- Enhanced `Simplefin::AccountTypeMapper` to include detection for crypto, checking, savings, and standalone cash accounts.
- Improved subtype selection UI with validation and warning indicators for missing selections.
- Updated SimpleFin account linking to handle both legacy FK and `AccountProvider` associations consistently.
- Refined job flow and importer logic for better handling of linked accounts and subtype inference.

* Improve `SimplefinConnectionUpdateJob` and holdings processing logic

- Fixed race condition in `SimplefinConnectionUpdateJob` by moving `destroy_later` calls outside of transactions.
- Updated fuzzy name match logic to use Levenshtein distance for better accuracy.
- Enhanced synthetic ticker generation in holdings processor with hash suffix for uniqueness.

* Refine SimpleFin entry processing logic and ensure `extra` data persistence

- Simplified pending flag determination to rely solely on provider-supplied values.
- Fixed potential stale values in `extra` by ensuring deep merge overwrite with `entry.transaction.save!`.

* Replace hardcoded fallback transaction description with localized string

* Refine pending flag logic in SimpleFin processor tests

- Adjust test to prevent falsely inferring pending status from missing posted dates.
- Ensure provider explicitly sets pending flag for transactions.

* Add `has_many :holdings` association to `AccountProvider` with `dependent: :nullify`

---------

Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
2026-01-05 22:11:47 +01:00
Juan José Mata
7af16340eb Add environment to Langfuse trace 2025-12-20 20:24:32 +00:00
LPW
e9dbf5f4e7 Fix Broken Account Re-linking Feature (#469)
* Update SimpleFIN relinking flow and enhance duplicate account handling

- Updated logic to allow relinking of SimpleFIN accounts while preserving legacy mappings.
- Introduced clean-up logic to hide orphaned duplicate accounts after relinking.
- Enhanced UI to display current mappings for linked accounts.
- Improved test coverage for relinking scenarios and SimpleFIN account visibility.

* Localize SimpleFIN account selection messages and remove hardcoded text

- Added translations for user-facing messages in `select_existing_account` flow (`pt-BR` and `en` locales).
- Replaced hardcoded strings in the view with localized keys.

* Localize Enable Banking and SimpleFIN account linking messages; add support for investment accounts.

- Added translations for Enable Banking and SimpleFIN account linking flows.
- Updated views and controllers to replace hardcoded strings with localized keys.
- Introduced support for investment accounts in `Provider::LunchflowAdapter`.
- Enhanced relinking logic for SimpleFIN accounts and improved test coverage for related scenarios.

---------

Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
2025-12-20 21:18:55 +01:00
LPW
664c6c2b7c Pending detection, FX metadata, Pending UI badge. (#374)
* - Add support for `SIMPLEFIN_INCLUDE_PENDING` to control pending behavior via ENV.
- Enhance debug logging for SimpleFin API requests and raw payloads.
- Refine pending flag handling in `SimplefinEntry::Processor` based on provider data and inferred conditions.
- Improve FX metadata processing for transactions with currency mismatches.
- Add new tests for pending detection, FX metadata, and edge cases involving `posted` values.
- Add pending indicator UI to transaction view.

* Document pending transaction detection, storage, and UI behavior for SimpleFIN and Plaid integrations. Add debug flags for troubleshooting.

* Add `pending?` method to `Transaction` model, refactor UI indicator, and centralize SimpleFIN configuration

- Introduced `pending?` method in `Transaction` for unified pending state detection.
- Refactored transaction pending indicator in the UI to use `pending?` method.
- Centralized SimpleFIN configuration in initializer with ENV-backed toggles.
- Updated tests for `pending?` behavior and clarified docs for pending detection logic

* Add SimpleFIN debug and runtime flags to `.env.local.example` and `.env.test.example`

- Introduced `SIMPLEFIN_INCLUDE_PENDING` and `SIMPLEFIN_DEBUG_RAW` flags for controlling pending behavior and debugging.
- Updated example environment files with descriptions for new configuration options.

* Normalize formatting for `SIMPLEFIN_INCLUDE_PENDING` and `SIMPLEFIN_DEBUG_RAW` flags in `.env.local.example` and `.env.test.example`.

---------

Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
2025-12-19 23:24:48 +01:00
soky srm
0d52566cc1 Fix api call for cloudflare API (#467) 2025-12-19 13:18:43 +01:00
Alessio Cappa
dd461faf84 feat: Allow account linking for Enable Banking accounts (#428)
* feat: Allow account linking for Enable Banking accounts

* fix: Typo in function name

* fix: naming issue

* fix: Add missing Enable Banking route

* feat: Add ability to link Enable Banking when adding a new account

* Mispelling

* fix: typo in method call

* fix: typo in column name

* Review suggestions

* Linter noise

* Small copy changes to avoid mobile UI blowout

* Provider generator (#364)

* Move provider config to family

* Update schema.rb

* Add provier generator

* Add table creation also

* FIX generator namespace

* Add support for global providers also

* Remove over-engineered stuff

* FIX parser

* FIX linter

* Some generator fixes

* Update generator with fixes

* Update item_model.rb.tt

* Add missing linkable concern

* Add missing routes

* Update adapter.rb.tt

* Update connectable_concern.rb.tt

* Update unlinking_concern.rb.tt

* Update family_generator.rb

* Update family_generator.rb

* Delete .claude/settings.local.json

Signed-off-by: soky srm <sokysrm@gmail.com>

* Move docs under API related folder

* Rename Rails generator doc

* Light edits to LLM generated doc

* Small Lunch Flow config panel regressions.

---------

Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>

* Skip generators autoloading (#430)

* Include Enable Banking items in Syncer (#434)

* feat: Include Enable Banking items in Syncer

* feat: include only active Enable Banking accounts

* Fix budgets page UI (#427)

* fix: Budget UI improvements

* feat: Reduce padding for sub-categories

* fix: Adjust padding for sub-category arrow

* Revert "feat: Reduce padding for sub-categories"

This reverts commit 7516c5a8e0.

* Revert "fix: Adjust padding for sub-category arrow"

This reverts commit ebc82542cf.

* fix: adjust padding for sub-categories

* fix: Add padding to uncategorized budget

* fix: Remove unnecessary HTML tag

* feat: Add translation keys for budgeted/actual

* feat(lang): add all brazilian portuguese translations (#416)

* feat(lang): add all brazilian portuguese translations

* feat: update pt-BR errors on translation

* fix: atualizar fix base

* feat: add reports translations

* feat: finish translation to brazilian portuguese

* fix: add to supported locales

* fix: number of translations

* fix: errors on translations

* fix: error on rubocop lint

---------

Co-authored-by: Leonardo Ralph <theleoralph@gmail.com>

* Add exclude transaction rule action (#437)

* Initial plan

* Add ExcludeTransaction rule action executor with tests

Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com>

* Copy clarification

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>

* Preparing for v0.6.6-alpha.3

Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>

* fix: remove account_id clearing for Enable Banking accounts

* fix: Remove unexisting available_balance attribute and rename variable for consistency

---------

Signed-off-by: soky srm <sokysrm@gmail.com>
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: soky srm <sokysrm@gmail.com>
Co-authored-by: Marcon Neves <marconwillian@icloud.com>
Co-authored-by: Leonardo Ralph <theleoralph@gmail.com>
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com>
2025-12-12 11:19:50 +01:00
soky srm
88952e4714 Small llms improvements (#400)
* Initial implementation

* FIX keys

* Add langfuse evals support

* FIX trace upload

* Delete .claude/settings.local.json

Signed-off-by: soky srm <sokysrm@gmail.com>

* Update client.rb

* Small LLMs improvements

* Keep batch size normal

* Update categorizer

* FIX json mode

* Add reasonable alternative to matching

* FIX thinking blocks for llms

* Implement json mode support with AUTO mode

* Make auto default for everyone

* FIX linter

* Address review

* Allow export manual categories

* FIX user export

* FIX oneshot example pollution

* Update categorization_golden_v1.yml

* Update categorization_golden_v1.yml

* Trim to 100 items

* Update auto_categorizer.rb

* FIX for auto retry in auto mode

* Separate the Eval Logic from the Auto-Categorizer

The expected_null_count parameter conflates eval-specific logic with production categorization logic.

* Force json mode on evals

* Introduce a more mixed dataset

150 items, performance from a local model:

By Difficulty:
  easy: 93.22% accuracy (55/59)
  medium: 93.33% accuracy (42/45)
  hard: 92.86% accuracy (26/28)
  edge_case: 100.0% accuracy (18/18)

* Improve datasets

Remove Data leakage from prompts

* Create eval runs as "pending"

---------

Signed-off-by: soky srm <sokysrm@gmail.com>
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2025-12-07 18:11:34 +01:00
Alessio Cappa
4dbe1d8df6 fix: retrieve only accounted transactions to avoid duplicates 2025-12-01 19:37:52 +01:00
soky srm
4a29d030af Initial enable banking implementation (#382)
* Initial enable banking implementation

* Handle multiple connections

* Amount fixes

* Account type mapping

* Add option to skip accounts

* Update schema.rb

* Transaction fixes

* Provider fixes

* FIX account identifier

* FIX support unlinking

* UI style fixes

* FIX safe redirect and brakeman issue

* FIX

- pagination max fix
- wrap crud in transaction logic

* FIX api uid access

- The Enable Banking API expects the UUID (uid from the API response) to fetch balances/transactions, not the identification_hash

* FIX add new connection

* FIX erb code

* Alert/notice box overflow protection

* Give alert/notification boxes room to grow (3 lines max)

* Add "Enable Banking (beta)" to `/settings/bank_sync`

* Make Enable Banking section collapsible like all others

* Add callback hint to error message

---------

Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2025-11-29 13:31:08 +01:00
soky srm
91a91c3834 Improvements (#379)
* Improvements

- Fix button visibility in reports on light theme
- Unify logic for provider syncs
- Add default option is to skip accounts linking ( no op default )

* Stability fixes and UX improvements

* FIX add unlinking when deleting lunch flow connection as well

* Wrap updates in transaction

* Some more improvements

* FIX proper provider setup check

* Make provider section collapsible

* Fix balance calculation

* Restore focus ring

* Use browser default focus

* Fix lunch flow balance for credit cards
2025-11-25 20:21:29 +01:00
soky srm
226207e2f7 Provider optimisation (#375)
* Implement a provider configured account type

* Fix for SimpleFIN

* FIX tests and linter
2025-11-24 19:52:34 +01:00
Juan José Mata
f491916411 Track failed LLM API calls in llm_usages table (#360)
* Track failed LLM API calls in llm_usages table

This commit adds comprehensive error tracking for failed LLM API calls:

- Updated LlmUsage model with helper methods to identify failed calls
  and retrieve error details (failed?, http_status_code, error_message)

- Modified Provider::Openai to record failed API calls with error metadata
  including HTTP status codes and error messages in both native and
  generic chat response methods

- Enhanced UsageRecorder concern with record_usage_error method to support
  error tracking for auto-categorization and auto-merchant detection

- Updated LLM usage UI to display failed calls with:
  - Red background highlighting for failed rows
  - Error indicator icon with "Failed" label
  - Interactive tooltip on hover showing error message and HTTP status code

Failed calls are now tracked with zero tokens and null cost, storing
error details in the metadata JSONB column for visibility and debugging.

* Dark mode fixes

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-11-22 02:15:20 +01:00
soky srm
be0b20dfd9 Lunchflow settings family (#363)
* Move provider config to family

* remove global settings

* Remove turbo auto  submit

* Fix flash location

* Fix mssing syncer for lunchflow

* Update schema.rb

* FIX tests and encryption config

* FIX make rabbit happy

* FIX run migration in SQL

* FIX turbo frame modal

* Branding fixes

* FIX rabbit

* OCD with product names

* More OCD

* No other console.log|warn in codebase

---------

Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2025-11-22 02:14:29 +01:00
soky srm
0192d0638c Merge pull request #356 from jakubkottnauer/cross-rates-twelve-date
Use TwelveData cross API for exotic exchange rates
2025-11-21 15:58:11 +01:00
Jakub Kottnauer
153e7184fb Use TwelveData cross API for exotic currency pairs 2025-11-19 22:26:21 +01:00
soky srm
1234fd5568 FIX AI categorization & Twelvedata currency (#354)
* FIX AI categorization

* FIX twelve data API

Looking at api docs we were using the wrong field, currency is in meta tag.
2025-11-19 19:30:28 +01:00
soky srm
0290ed636f FIX holdings and exchange rate (#345)
* FIX holdings and exchange rate

- Fix holdings not showing total return instead showing day change %
- Fix exchange rate saving 0 for holidays, use LOCF - Last Observation Carried Forward

* Fix failing test
2025-11-17 15:18:20 +01:00
Matthew Kilpatrick
ff43e71dec Fix Yahoo minor unit handling (#335)
Some security prices get returned in non-standard minor units (eg. `GBp` for Great British Pence, instead of `GBP` for Great British Pounds), leading to incorrect handling of the returned price data.

This commit adds conversion logic for the two currencies I've been able to find examples of this affecting.
2025-11-15 18:34:22 +01:00
soky srm
e8f935bc6f Remove plaid initialiser (#317)
* Remove plaid initialiser

The initializer can be safely removed because:
  - Config is lazily loaded via Provider::Registry
  - reload_configuration is called after settings updates
  - All calling code handles nil configs gracefully
  - Initial nil state is fine - config loads on first use

* Fix for missing config

* Actually don't pollute application.rb

* Add currency loading for balances

* Fix race condition on lazy load

* Allow loans to be imported in lunch flow also

* Fix currency processor
2025-11-12 16:01:19 +01:00
soky srm
fad241c416 Fixes & Improvements (#316)
* Some improvements

- Fix issue with lunch flow accounts that were imported
- Remove the period comparison section from reports

* Add cleanup migration

* FIX for dynamic config

* Fix linter

* FIX settings setter

Reuse the base class’ atomic setter to leverage its locking and cache invalidation.

* Make upsert atomic

* Remove migration file

Signed-off-by: soky srm <sokysrm@gmail.com>

* Delete db/migrate/20251111094448_migrate_dynamic_fields_to_individual_entries.rb

Signed-off-by: soky srm <sokysrm@gmail.com>

* Fix cache reset

* Revert "Remove migration file"

This reverts commit 1f2a21ef58.

* Revert "Delete db/migrate/20251111094448_migrate_dynamic_fields_to_individual_entries.rb"

This reverts commit 29dcaaafb2.

* Fix Plaid initialiser

---------

Signed-off-by: soky srm <sokysrm@gmail.com>
2025-11-11 19:51:07 +01:00
soky srm
c6771ebaab Lunchflow fix (#307)
* Fix lunch flow pre-loading and UX

* Small UX fixes

- Proper closing of modal on cancel
- Preload on new account already

* Review comments

* Fix json error

* Delete .claude/settings.local.json

Signed-off-by: soky srm <sokysrm@gmail.com>

* Lunch Flow brand (again :-)

* FIX process only linked accounts

* FIX disable accounts with no name

* Fix string normalization

---------

Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2025-11-10 21:32:55 +01:00
Juan José Mata
0327552a9a Avoid auto-categorizing without user categories (#261) 2025-11-01 10:15:56 +01:00
soky srm
5eadfaad98 Lunchflow integration (#259)
* First pass lunch flow

* Fixes

- Fix apikey not being saved properly due to provider no reload support
- Fix proper messages if we try to link existing accounts.

* Fix better error handling

* Filter existing transactions and skip duplicates

* FIX messaging

* Branding :)

* Fix XSS and linter

* FIX provider concern

- also fix code duplication

* FIX md5 digest

* Updated determine_sync_start_date to be account-aware

* Review fixes

* Broaden error catch to not crash UI

* Fix buttons styling

* FIX process account error handling

* FIX account cap and url parsing

* Lunch Flow brand

* Found orphan i18n strings

* Remove per conversation with @sokie

---------

Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2025-10-30 14:07:16 +01:00
soky srm
96713ee8b4 Add support for dynamic config UI (#256)
* Add support for dynamic config UI

* Add support for section description

* Better dynamic class settings

Added dynamic_fields hash field - Stores all undeclared settings
[] method - Checks declared fields first, then falls back to dynamic hash
[]= method - Updates declared fields normally, stores others in hash
No runtime field declaration - Fields are never dynamically created on the class

* FIX proper lookup for provider keys

- Also validate configurable values properly.
- Change Provider factory to use Rails autoloading (Zeitwerk)

* Fix factory

The derive_adapter_name method relies on string manipulation ("PlaidAccount".sub(/Account$/, "") + "Adapter" → "PlaidAdapter"), but we already have explicit registration in place.

* Make updates atomic, field-aware, and handle blanks explicitly

* Small UX detail

* Add support for PlaidEU in UI also

- This looks like partial support atm
2025-10-29 13:11:04 +01:00
João Felipe
9fefe57de5 Feature/yahoo finance (#123)
* Implement Yahoo Finance

* Added tests

* Updated hosting controller to check for managed app_mode instead of env_override

* Suggestions from CodeRabbit and Fixes on tests

* Remove Css changes

* Fix yahoo finance impl and i18n

* Updated view to use healthy method

* remove usage

* Updated env example

* keep usage on class just to keep same format

* Ci test

* Remove some useless validations

* Remove logs

* Linter fixes

* Broke this in my conflict merge

* Wrong indentation level

---------

Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2025-10-28 23:15:14 +01:00
soky srm
4fb0a3856e Providers factory (#250)
* Implement providers factory

* Multiple providers sync support

- Proper Multi-Provider Syncing: When you click sync on an account with multiple providers (e.g., both Plaid and SimpleFin), all provider items are synced
- Better API: The existing account.providers method already returns all providers, and account.provider returns the first one for backward compatibility
- Correct Holdings Deletion Logic: Holdings can only be deleted if ALL providers allow it, preventing accidental deletions that would be recreated on next sync
TODO: validate this is the way we want to go? We would need to check holdings belong to which account, and then check provider allows deletion. More complex
- Database Constraints: The existing validations ensure an account can have at most one provider of each type (one PlaidAccount, one SimplefinAccount, etc.)

* Add generic provider_import_adapter

* Finish unified import strategy

* Update app/models/plaid_account.rb

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: soky srm <sokysrm@gmail.com>

* Update app/models/provider/factory.rb

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: soky srm <sokysrm@gmail.com>

* Fix account linked by plaid_id instead of external_id

* Parse numerics to BigDecimal

Parse numerics to BigDecimal before computing amount; guard nils.

Avoid String * String and float drift; also normalize date.

* Fix incorrect usage of assert_raises.

* Fix linter

* Fix processor test.

* Update current_balance_manager.rb

* Test fixes

* Fix plaid linked account test

* Add support for holding per account_provider

* Fix proper account access

Also fix account deletion for simpefin too

* FIX match tests for consistency

* Some more factory updates

* Fix account schema for multipe providers

  Can do:
  - Account #1 → PlaidAccount + SimplefinAccount (multiple different providers)
  - Account #2 → PlaidAccount only
  - Account #3 → SimplefinAccount only

  Cannot do:
  - Account #1 → PlaidAccount + PlaidAccount (duplicate provider type)
  - PlaidAccount #123 → Account #1 + Account #2 (provider linked to multiple accounts)

* Fix account setup

- An account CAN have multiple providers (the schema shows account_providers with unique index on [account_id, provider_type])
  - Each provider should maintain its own separate entries
  - We should NOT update one provider's entry when another provider syncs

* Fix linter and guard migration

* FIX linter issues.

* Fixes

- Remove duplicated index
- Pass account_provider_id
- Guard holdings call to avoid NoMethodError

* Update schema and provider import fix

* Plaid doesn't allow holdings deletion

* Use ClimateControl for proper env setup

* No need for this in .git

---------

Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2025-10-28 19:32:27 +01:00
Copilot
d51ba515c9 Fix Twelve Data API parsing errors causing "Current Market Price Unknown" (#224)
* Add tests and fix for Twelve Data API parsing errors
* Fix search_securities to handle nil data key

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2025-10-24 13:33:06 +02:00
Copilot
a8f318c3f9 Fix "Messages is invalid" error for Ollama/custom LLM providers and add comprehensive AI documentation (#225)
* Add comprehensive AI/LLM configuration documentation
* Fix Chat.start! to use default model when model is nil or empty
* Ensure all controllers use Chat.default_model for consistency
* Move AI doc inside `hosting/`
* Probably too much error handling

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2025-10-24 12:04:19 +02:00
soky srm
bb364fab38 LLM cost estimation (#223)
* Password reset back button also after confirmation

Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>

* Implement a filter for category (#215)

- Also implement an is empty/is null condition.

* Implement an LLM cost estimation page

Track costs across all the cost categories: auto categorization, auto merchant detection and chat.
Show warning with estimated cost when running a rule that contains AI.

* Update pricing

* Add google pricing

and fix inferred model everywhere.

* Update app/models/llm_usage.rb

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: soky srm <sokysrm@gmail.com>

* FIX address review

* Linter

* Address review

- Lowered log level
- extracted the duplicated record_usage method into a shared concern

* Update app/controllers/settings/llm_usages_controller.rb

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: soky srm <sokysrm@gmail.com>

* Moved attr_reader out of private

---------

Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-10-24 00:08:59 +02:00
Himmelschmidt
4cd737b5d9 Fix SimpleFin investment holdings and comprehensive integration improvements (#104)
* Remove skipped account functionality from SimpleFin

- Remove "Skip - don't add" option from account setup
- Simplify account setup flow to require all accounts be assigned types
- Update controller logic to handle all accounts without skipping
- Redirect to accounts page instead of SimpleFin items page
- Add I18N message support with t(".success")

* Simplify SimpleFin sync logic by removing skipped accounts

- Remove skipped account filtering from syncer
- All unlinked accounts now block sync until setup is complete
- Remove skipped account UI elements from setup view
- Streamline sync flow without complex skipped/non-skipped logic

* Fix cash balance calculation for SimpleFin investment accounts

- Update SimplefinAccount::Processor to recalculate balances during sync
- Properly calculate cash_balance for investment accounts using BalanceCalculator
- Handle negative balances for credit cards and loans correctly
- Ensure account balance and cash balance are updated from latest SimpleFin data

* Add I18N translations and edit view for SimpleFin

- Add comprehensive English translations for SimpleFin UI
- Create edit view for SimpleFin re-authentication
- Support status messages, errors, and user feedback
- Match translation structure with Plaid integration

* Add specialized SimpleFin data processors

- Add investment processors for transactions, holdings, and balance calculation
- Add liability processors for credit cards and loans
- Add transaction processor for standard account transactions
- Add account importer for SimpleFin account data
- Organize processors by account type for maintainable architecture

* Add loading button controller for SimpleFin forms

- Add Stimulus controller to show loading state during form submission
- Disable button and show loading text to prevent double submissions
- Improve user experience during SimpleFin account setup

* Add SimpleFin edit and update routes

- Add edit and update actions to SimpleFin items routes
- Enable re-authentication flow for expired SimpleFin connections
- Match route structure with Plaid items for consistency

* Add institution metadata fields to SimpleFin items

- Add institution_id, institution_name, institution_domain fields
- Add institution_url, institution_color for UI customization
- Add raw_institution_payload for complete institution data storage
- Enable better institution organization and display

* Enhance SimpleFin item with institution support and metadata

- Add institution summary and connected institutions methods
- Store and retrieve institution metadata from SimpleFin API
- Add institution-aware import functionality
- Support multiple institutions per SimpleFin connection

* Fix account creation and Plaid provider issues

- Fix cash balance calculation in Account.create_from_simplefin_account
- Add nil check for plaid_provider in remove_plaid_item method
- Ensure proper balance handling for investment accounts during creation

* Improve sync parent relationship handling

- Add parent sync assignment for existing syncs when parent_sync is provided
- Ensure sync hierarchy is maintained when expanding sync windows
- Fix sync relationship consistency in nested sync operations

* Update SimpleFin item view with enhanced UI

- Improve SimpleFin connection display and status information
- Add better visual styling and user feedback
- Match UI consistency with Plaid item views

* Update database schema with institution fields

- Add institution metadata fields to simplefin_items table
- Support institution tracking and organization features

* Update SimpleFin tests for new functionality

- Update controller tests for edit/update actions and removed skip functionality
- Update model tests for institution metadata and enhanced features
- Ensure test coverage for SimpleFin improvements

* Add migration to remove old institution fields

* Fix linting issues

* Apply suggestion from @coderabbitai[bot]

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: Himmelschmidt <46351743+Himmelschmidt@users.noreply.github.com>

* Fix 2 failing tests

* Wrap SimpleFin account transfer in database transaction

* Make loading button controller more reusable

- Add loadingText Stimulus value for configurable loading text
- Remove unused originalText variable
- Update view to pass loading text via data attribute

* Remove unused SimplefinAccount::Importer class

This class was added in the PR but is never called anywhere in the codebase.
The actual SimpleFin account processing is handled by SimplefinAccount::Processor
and its specialized sub-processors.

* Fix SimpleFin account transfer bug during token updates

- Call import_latest_simplefin_data before account transfer to ensure
  new SimpleFin accounts exist with account_id populated
- Prevents silent failure where accounts become orphaned when
  refreshing expired SimpleFin tokens

* Fix SimpleFin error handling to render correct template and use i18n

- Update render_error method to accept context parameter for template selection
- Fix update action to render :edit template instead of :new on errors
- Replace hardcoded error messages with localized strings using t() calls
- Add comprehensive error message keys to SimpleFin locale file

* Improve loading button accessibility and HTML semantics

- Add aria-disabled and aria-busy attributes for screen readers
- Use semantic span elements instead of divs for button content
- Add aria-hidden to decorative spinner element

* Fix SimpleFin SSL verification to use OpenSSL constant

* Remove HTTParty streaming to prevent empty response body and PII logging

* Use BigDecimal zero for consistent numeric types in balance calculator

* Add investment account guard to holdings processor

* Remove duplicate balance normalization from SimpleFin loan processor

* Fix critical account deletion bug in SimpleFin token update

* Fix linting issues in SimpleFin controller tests

* Replace hardcoded colors with design system tokens in SimpleFin views

* Gate investment processors to Investment accounts only

Prevents investment processors from running on non-Investment account types,
matching the pattern used by liability processors.

* Localize hardcoded strings in SimpleFin edit form

* Adding the error message to a hover state.

* Use only 1 month for sync_start_date, new account restriction?

* Harden investment cash_balance calculation with error handling

- Add try/catch around SimplefinAccount::Investments::BalanceCalculator
- Fallback to zero on calculation errors or nil results
- Log warning with error details for debugging
- Prevents nil cash_balance that could cause downstream issues

* Fix SimpleFin institution fields migration and add DB constraints

- Remove destructive migration that dropped existing institution fields
- Add only new fields (institution_domain, institution_color)
- Add indexes on institution fields for query performance
- Add NOT NULL constraints on required fields (institution_id, institution_name)
- Fix schema jsonb consistency for raw_institution_payload

* Improve SimpleFin holdings error handling and BigDecimal consistency

* Add security attribute to external link in SimpleFin edit form

* Improve SimpleFin sync timing and add user-configurable date range

- Fix initial sync timing issue: change from 1 month to 7 days default lookback
- Add user-selectable sync start date in account setup UI
- Implement chunked historical sync that respects user-selected date range
- Add sync_start_date field to SimpleFin items
- Handle new accounts on existing connections with same date picker

This addresses SimpleFin API limitations and gives users control over
how much transaction history to sync during initial setup.

* Fix SimpleFin sync status to show detailed results instead of "Never synced"

- Modify sync completion logic to always complete even with unlinked accounts
- Add sync_stats column to track account sync statistics during sync process
- Update sync status display to show "X synced, Y need setup" instead of "Never synced"
- Store detailed sync statistics (total, linked, unlinked accounts) in sync record
- Add sync_status_summary method to provide meaningful status text
- Remove early return that prevented sync completion when accounts need setup

Resolves issue where successful account syncing still showed "Never synced" status.

* Fix Transaction persistence before Entry creation in SimpleFin processor

Persist Transaction with create! instead of new to ensure it has an ID before
creating Entry that references it as entryable. Prevents foreign key errors.

* Fix indifferent access for SimpleFin institution data extraction

The accounts_snapshot parameter comes from JSON with string keys, but the
code was accessing with symbol keys which could silently fail. Convert to
indifferent access to handle both string and symbol keys properly.

* Localize hardcoded deletion in progress string

Replace hardcoded "(deletion in progress...)" text with I18n translation
to maintain consistency with the rest of the view.

* Fix SimpleFin item update test to properly verify rebind/destroy behavior

The test now creates a different SimplefinItem instance and mocks
create_simplefin_item! to return it, ensuring the controller operates
on a new record instead of the same instance. This properly exercises
the rebind/destroy logic and verifies the original item is scheduled
for deletion.

* Fix potential transaction data loss in SimpleFin importer

Prevent wiping stored transactions when API omits transaction data by only
updating raw_transactions_payload when transactions are actually present
in the response, preserving existing transaction data when API doesn't
include transactions key.

* Fix SimpleFin sync chunking and enforce 3-year historical limit

- Fix SimpleFin's actual API limit from 365 days to 60 days per request
- Implement proper backward-walking chunked sync for historical data
- Enforce 3-year maximum historical data limit (60 days × 22 requests)
- Update date picker to reflect 3-year limit and better defaults
- Add comprehensive logging for debugging sync issues

* Add dedicated raw_holdings_payload storage for SimpleFin accounts

- Add raw_holdings_payload column to simplefin_accounts table
- Separates holdings data from general account data for cleaner processing
- Follows same pattern as raw_transactions_payload for consistency
- Enables proper SimpleFin holdings processing pipeline

* Enhance SimpleFin holdings storage with external ID tracking

- Add external_id and cost_basis columns to holdings table
- Update holdings processor to use external_id for precise matching
- Capture all available SimpleFin holdings data (shares, market_value, cost_basis, etc.)
- Use SimpleFin holding ID as external_id with "simplefin_" prefix
- Calculate price from market_value/shares when available
- Store raw holdings payload in simplefin_accounts for complete data retention

This enables better holdings tracking than composite key approach and ensures
we capture all SimpleFin data even if not immediately used in our models.

* Simplify SimpleFin transaction enrichment

- Add MerchantDetector that uses payee field directly for merchant creation
- Enhance SimpleFin entry name generation combining payee + description
- Remove transaction processor category matching logic
- Create dedicated SimpleFin entry processor

Uses SimpleFin's clean payee data without unnecessary filtering.

* Add source field to ProviderMerchant and fix data enrichment

- Add source field to ProviderMerchant model for provider-specific merchant tracking
- Fix DataEnrichment to handle string transaction IDs correctly with to_i conversion

Enables per-provider merchant deduplication and fixes transaction lookups.

* Fix SimpleFin controller parameter handling

- Convert string account_ids to integers for proper account lookup
- Ensure account selection works correctly with form submissions

Fixes account filtering when setting up SimpleFin sync.

* Fix linting issues - auto-corrected whitespace and formatting

* Derive institution domain from URL when missing in SimpleFin items

* Fix render_error to preserve persisted record for edit context

* Add unique index to prevent duplicate holdings

* Fix potential NameError in holdings processor rescue block

* Guard against missing SimpleFin transaction IDs

* Fix SimpleFin amount parsing error handling

Re-raise ArgumentError instead of silently returning BigDecimal("0")
to prevent misleading $0 transactions from invalid amount data.

* Fix SimpleFin chunked import data loss bug

Merge transaction arrays instead of overwriting to prevent data loss
during chunked imports. Preserve most recent holdings data only.

* Add external_id uniqueness validation to Holding model

* Fix holdings cost_basis precision and add external_id unique constraint

* Fix SimpleFin test mock expectations and remove debug statements

- Fixed SimplefinItemsControllerTest by properly mocking Provider::Simplefin
  instead of over-mocking the create_simplefin_item! method
- Removed DEBUG puts statements from SimplefinItem::Importer

* Fix linting issues - auto-corrected whitespace and formatting

---------

Signed-off-by: Himmelschmidt <46351743+Himmelschmidt@users.noreply.github.com>
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2025-10-22 19:51:24 +02:00
Juan José Mata
7c5ddd674d Make branding configurable (#173)
* Remove orphan function

* Add centralized branding helpers and update locales

* Remove _plus and add (proper) brand

* No longer Sure, configurable

* Consistency with compose file naming

* Missed `product_name` mapping

* Fix brand/product name in mailers

* Product name in email reset flow

* Fix i18n errors/tests

* Fix password mailer brand/product name (again)

* Missed hardcoded `Sure` in onboarding goals

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: Juan José Mata <jjmata@jjmata.com>

* PR nitpick on documentation

* Missing interpolation key for invited UI

* Orphan assets

* New logos

---------

Signed-off-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
2025-10-22 19:14:03 +02:00
soky srm
8cd109a5b2 Implement support for generic OpenAI api (#213)
* Implement support for generic OpenAI api

- Implements support to route requests to any openAI capable provider ( Deepsek, Qwen, VLLM, LM Studio, Ollama ).
- Keeps support for pure OpenAI and uses the new better responses api
- Uses the /chat/completions api for the generic providers
- If uri_base is not set, uses default implementation.

* Fix json handling and indentation

* Fix linter error indent

* Fix tests to set env vars

* Fix updating settings

* Change to prefix checking for OAI models

* FIX check model if custom uri is set

* Change chat to sync calls

Some local models don't support streaming. Revert to sync calls for generic OAI api

* Fix tests

* Fix tests

* Fix for gpt5 message extraction

- Finds the message output by filtering for "type" == "message" instead of assuming it's at index 0
- Safely extracts the text using safe navigation operators (&.)
- Raises a clear error if no message content is found
- Parses the JSON as before

* Add more langfuse logging

- Add Langfuse to auto categorizer and merchant detector
- Fix monitoring on streaming chat responses
- Add Langfuse traces also for model errors now

* Update app/models/provider/openai.rb

Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: soky srm <sokysrm@gmail.com>

* handle nil function results explicitly

* Exposing some config vars.

* Linter and nitpick comments

* Drop back to `gpt-4.1` as default for now

* Linter

* Fix for strict tool schema in Gemini

- This fixes tool calling in Gemini OpenAI api
- Fix for getTransactions function, page size is not used.

---------

Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
2025-10-22 16:02:50 +02:00