* Add AI merchant enhancement and dedup
* Enhancements
Add error if job is already running
add note that we also merge merchants
* Allow updating provider website
* Review fixes
* Update provider_merchant.rb
* Linter and fixes
* FIX transaction quick menu modal
* Make categories global
This solves us A LOT of cash flow and budgeting problems.
* Update schema.rb
* Update auto_categorizer.rb
* Update income_statement.rb
* FIX budget sub-categories
* FIX sub-categories and tests
* Add 2 step migration
* Initial plan
* Fix nil references in Recording failed LLM usage code paths
Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com>
* Replace error&.message with rescue-guarded safe_error_message helper
error&.message only guards against nil; it still raises when the error
object's .message implementation itself throws (e.g. OpenAI errors that
call data on nil). Replace with a safe_error_message helper that wraps
error&.message in a rescue block, returning a descriptive fallback
string on secondary failures. Apply the helper in both record_usage_error
(usage_recorder.rb) and record_llm_usage (openai.rb), including the
regex branch of extract_http_status_code in both files.
Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com>
* Fix: use cookie/crumb auth in healthy? chart endpoint check
The health check was calling /v8/finance/chart/AAPL via the plain
unauthenticated client. Yahoo Finance requires cookie + crumb
authentication on the chart endpoint, so the health check would
fail even when credentials are valid. Updated healthy? to use
fetch_cookie_and_crumb + authenticated_client, consistent with
fetch_security_prices and fetch_chart_data.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: add cookie/crumb auth to all /v8/finance/chart/ calls
fetch_security_prices and fetch_chart_data (used for exchange rates)
were calling the chart endpoint without cookie/crumb authentication,
inconsistent with healthy? and fetch_security_info. Added auth to both,
including the same retry-on-Unauthorized pattern already used in
fetch_security_info.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Update user-agent strings in yahoo_finance.rb
Updated user-agent strings to reflect current browser versions
Signed-off-by: Serge L <serge@souritech.ca>
* Fix: Add stale-crumb retry to healthy? and fetch_chart_data
Yahoo Finance returns 200 OK with {"chart":{"error":{"code":"Unauthorized"}}}
when a cached crumb expires server-side. Both healthy? and fetch_chart_data
now mirror the retry pattern already in fetch_security_prices: detect the
Unauthorized body, clear the crumb cache, fetch fresh credentials, and
retry the request once. Adds a test for the healthy? retry path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Refactor: Extract fetch_authenticated_chart helper to DRY crumb retry logic
The cookie/crumb fetch + stale-crumb retry pattern was duplicated across
healthy?, fetch_security_prices, and fetch_chart_data. Extract it into a
single private fetch_authenticated_chart(symbol, params) helper that
centralizes the retry logic; all three call sites now delegate to it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: Catch JSON::ParserError in fetch_chart_data rescue clause
After moving JSON.parse inside fetch_authenticated_chart, a malformed
Yahoo response would throw JSON::ParserError through fetch_chart_data's
rescue Faraday::Error, breaking the inverse currency pair fallback.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: Raise AuthenticationError if retry still returns Unauthorized
After refreshing the crumb and retrying, if Yahoo still returns an
Unauthorized error body the helper now raises AuthenticationError instead
of silently returning the error payload. This prevents callers from
misinterpreting a persistent auth failure as missing chart data.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* Fix: Raise AuthenticationError after failed retry in fetch_security_info
Mirrors the same post-retry Unauthorized check added to fetch_authenticated_chart.
Without this, a persistent auth failure on the quoteSummary endpoint would
surface as a generic "No security info found" error instead of an AuthenticationError.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Signed-off-by: Serge L <serge@souritech.ca>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace direct trace.update calls with client trace upserts so OpenAI provider is compatible with langfuse-ruby 0.1.6 behavior. Add richer warning logs that include full exception details for trace creation, trace upserts, and generation logging failures. Add tests for client-based trace upserts and detailed error logging.
* Add Indexa Capital provider scaffold
Generate Indexa Capital provider scaffolding and align credential fields with the API authentication requirements.
* Fix PR 926 lint and schema CI failures
* Implement Indexa Capital provider with real API integration
- Rewrite all broken view templates (were meta-ERB from code generator)
- Create missing select_accounts.html.erb template
- Implement real API calls: list_accounts via /users/me, get_holdings
via /accounts/{number}/fiscal-results, get_account_balance via
/accounts/{number}/performance
- Add API token auth support (stored token > env token > credentials)
- Add api_token column with encryption support
- Redesign settings panel: API token prominent, credentials collapsible
- Fix account balances display using performance endpoint portfolios
- Fix accounts index empty-state guard missing indexa_capital_items
- Simplify activities fetch job (no activities API endpoint exists)
- Fix i18n interpolation (%%{ -> %{) throughout locale file
* Add tests for Indexa Capital provider integration
- IndexaCapitalItemTest: validations, credentials, scopes, sync status
- IndexaCapitalAccountTest: upsert, holdings, account provider linking
- Provider::IndexaCapitalTest: auth modes, API stubs, error handling
- IndexaCapitalItemsControllerTest: CRUD, setup, linking, authorization
- Fixtures for items (token + credentials) and accounts (mutual + pension)
52 tests, 98 assertions, 0 failures
* Address code review feedback from PR #933
- Fix zero balance bug: use `nil?` instead of `present?` so 0 is stored
- Fix has_indexa_capital_credentials? to check api_token (was ignored)
- Fix build_provider to delegate to Provided concern (was ignoring token)
- Fix IndexaCapital section outside encryption_error guard in settings
- Add account_number sanitization to prevent path traversal in API URLs
- Replace all skipped processor tests with real working tests
- Add zero-balance and path-traversal test coverage
61 tests, 107 assertions, 0 failures
* Address code review round 2: credentials validation, RuboCop, test quality
- Fix RuboCop SpaceInsideArrayLiteralBrackets in credentials check
- Chain where.not calls so all three username/document/password must be present
- Require all three credentials (||) instead of any one (&&) in validate_configuration!
- Move attr_reader to private to avoid exposing credentials publicly
- Parse dates with Date.parse in extract_balance for robustness
- Remove stale TODO and Crypto from supported_account_types
- Order build_provider query deterministically by created_at
- Replace no-op holdings assertion with meaningful assert_difference
* Address code review round 3: JSON parse safety and test precision
- Rescue JSON::ParserError on 2xx responses for clearer error messages
- Fix weak balance assertion: set balance to 0 before processing, assert
expected value (27093.01 = sum of holdings amounts)
* Include Indexa Capital in automatic family sync
Add indexa_capital_items to Family::Syncer#child_syncables so balances
and holdings refresh on daily auto-sync and login sync, not only on
manual sync button clicks.
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* feat: add SSL_CA_FILE and SSL_VERIFY environment variables to support self-signed certificates in self-hosted environments
* fix: NoMethodError by defining SSL helper methods before configure block executes
* refactor: Refactor SessionsController to use shared SslConfigurable module and simplify SSL initializer redundant checks
* refactor: improve SSL configuration robustness and error detection accuracy
* fix:HTTParty SSL options, add file validation guards, prevent Tempfile GC, and redact URLs in error logs
* fix: Fix SSL concern indentation and stub Simplefin POST correctly in tests
* fix: normalize ssl_verify to always return boolean instead of nil
* fix: solve failing SimpleFin test
* refactor: trim unused error-handling code from SslConfigurable, replace Tempfile with fixed-path CA bundle, fix namespace pollution in initializers, and add unit tests for core SSL configuration and Langfuse CRL callback.
* fix: added require ileutils in the initializer and require ostruct in the test file.
* fix: solve autoload conflict that broke provider loading, validate all certs in PEM bundles, and add missing requires.
* Add import row generation from PDF extracted data
- Add generate_rows_from_extracted_data method to PdfImport
- Add import! method to create transactions from PDF rows
- Update ProcessPdfJob to generate rows after extraction
- Update configured?, cleaned?, publishable? for PDF workflow
- Add column_keys, required_column_keys, mapping_steps
- Set bank statements to pending status for user review
- Add tests for new functionality
Closes#844
* Add tests for BankStatementExtractor
- Test transaction extraction from PDF content
- Test deduplication across chunk boundaries
- Test amount normalization for various formats
- Test graceful handling of malformed JSON responses
- Test error handling for empty/nil PDF content
* Fix supports_pdf_processing? to validate effective model
The validation was always checking @default_model, but process_pdf
allows overriding the model via parameter. This could cause a
vision-capable override model to be rejected, or a non-vision-capable
override to pass validation only to fail during processing.
Changes:
- supports_pdf_processing? now accepts optional model parameter
- process_pdf passes effective model to validation
- Raise Provider::Openai::Error inside with_provider_response for
consistent error handling
Addresses review feedback from PR#808
* Fix insert_all! bug: explicitly set import_id
Rails insert_all! on associations does NOT auto-set the foreign key.
Added import_id explicitly and use Import::Row.insert_all! directly.
Also reload rows before counting to ensure accurate count.
* Fix pending status showing as processing for bank statements with rows
When bank statement PDF imports have extracted rows, show a 'Ready for Review'
screen with a link to the confirm path instead of the 'Processing' spinner.
This addresses the PR feedback that users couldn't reach the review flow even
though rows were created.
* Gate publishable? on account.present? to prevent import failure
PDF imports are created without an account, and import! raises if account
is missing. This prevents users from hitting publish and having the job fail.
* Wrap generate_rows_from_extracted_data in transaction for atomicity
- Clear rows and reset count even when no transactions extracted
- Use transaction block to prevent partial updates on failure
- Use mapped_rows.size instead of reload for count
* Localize transactions count string with i18n helper
* Add AccountMapping step for PDF imports when account is nil
PDF imports need account selection before publishing. This adds
Import::AccountMapping to mapping_steps when account is nil,
matching the behavior of TransactionImport and TradeImport.
Addresses PR#846 feedback about account selection for PDF imports.
* Only include CategoryMapping when rows have non-empty categories
PDF extraction doesn't extract categories from bank statements,
so the CategoryMapping step would show empty. Now we only include
CategoryMapping if rows actually have non-empty category values.
This prevents showing an empty mapping step for PDF imports.
* Fix PDF import UI flow and account selection
- Add direct account selection in PDF import UI instead of AccountMapping
- AccountMapping designed for CSV imports with multiple account values
- PDF imports need single account for all transactions
- Add update action and route for imports controller
- Fix controller to handle pdf_import param format from form_with
- Show Publish button when import is publishable (account set)
- Fix stepper nav: Upload/Configure/Clean non-clickable for PDF imports
- Redirect PDF imports from configuration step (auto-configured)
- Improve AI prompt to recognize M-PESA/mobile money as bank statements
- Fix migration ordering for import_rows table columns
* Add guard for invalid account_id in imports#update
Prevents silently clearing account when invalid ID is passed.
Returns error message instead of confusing 'Account saved' notice.
* Localize step names in import nav and add account guard
- Use t() helper for all step names (Upload, Configure, Clean, Map, Confirm)
- Add guard for invalid account_id in imports#update
- Prevents silently clearing account when invalid ID is passed
* Make category column migrations idempotent
Check if columns exist before adding to prevent duplicate column
errors when migrations are re-run with new timestamps.
* Add match_path for PDF import step highlighting
Fixes step detection when path is nil by using separate match_path
for current step highlighting while keeping links disabled.
* Rename category migrations and update to Rails 7.2
- Rename class to EnsureCategoryFieldsOnImportRows to avoid conflicts
- Rename class to EnsureCategoryIconOnImportRows
- Update migration version from 7.1 to 7.2 per guidelines
- Rename files to match class names
- Add match_path for PDF import step highlighting
* Use primary (black) style for Create Account and Save buttons
* Remove match_path from auto-completed PDF steps
Only step 4 (Confirm) needs match_path for active-step detection.
Steps 1-3 are purely informational and always complete.
* Add fallback for document type translation
Handles nil or unexpected document_type values gracefully.
Also removes match_path from auto-completed PDF steps.
* Use index-based step number for mobile indicator
Fixes 'Step 5 of 4' issue when Map step is dynamically removed.
* Fix hostings_controller_test: use blank? instead of nil
Setting returns empty string not nil for unset values.
* Localize step progress label and use design token
* Fix button styling: use design system Tailwind classes
btn--primary and btn--secondary CSS classes don't exist.
Use actual design system classes from DS::Buttonish.
* Fix CRLF line endings in tags_controller_test.rb
---------
Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
* feat: Add PDF import with AI-powered document analysis
This enhances the import functionality to support PDF files with AI-powered
document analysis. When a PDF is uploaded, it is processed by AI to:
- Identify the document type (bank statement, credit card statement, etc.)
- Generate a summary of the document contents
- Extract key metadata (institution, dates, balances, transaction count)
After processing, an email is sent to the user asking for next steps.
Key changes:
- Add PdfImport model for handling PDF document imports
- Add Provider::Openai::PdfProcessor for AI document analysis
- Add ProcessPdfJob for async PDF processing
- Add PdfImportMailer for user notification emails
- Update imports controller to detect and handle PDF uploads
- Add PDF import option to the new import page
- Add i18n translations for all new strings
- Add comprehensive tests for the new functionality
* Add bank statement import with AI extraction
- Create ImportBankStatement assistant function for MCP
- Add BankStatementExtractor with chunked processing for small context windows
- Register function in assistant configurable
- Make PdfImport#pdf_file_content public for extractor access
- Increase OpenAI request timeout to 600s for slow local models
- Increase DB connection pool to 20 for concurrent operations
Tested with M-Pesa bank statement via remote Ollama (qwen3:8b):
- Successfully extracted 18 transactions
- Generated CSV and created TransactionImport
- Works with 3000 char chunks for small context windows
* Add pdf-reader gem dependency
The BankStatementExtractor uses PDF::Reader to parse bank statement
PDFs, but the gem was not properly declared in the Gemfile. This would
cause NameError in production when processing bank statements.
Added pdf-reader ~> 2.12 to Gemfile dependencies.
* Fix transaction deduplication to preserve legitimate duplicates
The previous deduplication logic removed ALL duplicate transactions based
on [date, amount, name], which would drop legitimate same-day duplicates
like multiple ATM withdrawals or card authorizations.
Changed to only deduplicate transactions that appear in consecutive chunks
(chunking artifacts) while preserving all legitimate duplicates within the
same chunk or non-adjacent chunks.
* Refactor bank statement extraction to use public provider method
Address code review feedback:
- Add public extract_bank_statement method to Provider::Openai
- Remove direct access to private client via send(:client)
- Update ImportBankStatement to use new public method
- Add require 'set' to BankStatementExtractor
- Remove PII-sensitive content from error logs
- Add defensive check for nil response.error
- Handle oversized PDF pages in chunking logic
- Remove unused process_native and process_generic methods
- Update email copy to reflect feature availability
- Add guard for nil document_type in email template
- Document pdf-reader gem rationale in Gemfile
Tested with both OpenAI (gpt-4o) and Ollama (qwen3:8b):
- OpenAI: 49 transactions extracted in 30s
- Ollama: 40 transactions extracted in 368s
- All encapsulation and error handling working correctly
* Update schema.rb with ai_summary and document_type columns
* Address PR #808 review comments
- Rename :csv_file to :import_file across controllers/views/tests
- Add PDF test fixture (sample_bank_statement.pdf)
- Add supports_pdf_processing? method for graceful degradation
- Revert unrelated database.yml pool change (600->3)
- Remove month_start_day schema bleed from other PR
- Fix PdfProcessor: use .strip instead of .strip_heredoc
- Add server-side PDF magic byte validation
- Conditionally show PDF import option when AI provider available
- Fix ProcessPdfJob: sanitize errors, handle update failure
- Move pdf_file attachment from Import to PdfImport
- Document deduplication logic limitations
- Fix ImportBankStatement: catch specific exceptions only
- Remove unnecessary require 'set'
- Remove dead json_schema method from PdfProcessor
- Reduce default OpenAI timeout from 600s to 60s
- Fix nil guard in text mailer template
- Add require 'csv' to ImportBankStatement
- Remove Gemfile pdf-reader comment
* Fix RuboCop indentation in ProcessPdfJob
* Refactor PDF import check to use model predicate method
Replace is_a?(PdfImport) type check with requires_csv_workflow? predicate
that leverages STI inheritance for cleaner controller logic.
* Fix missing 'unknown' locale key and schema version mismatch
- Add 'unknown: Unknown Document' to document_types locale
- Fix schema version to match latest migration (2026_01_24_180211)
* Document OPENAI_REQUEST_TIMEOUT env variable
Added to .env.local.example and docs/hosting/ai.md
* Rename ALLOWED_MIME_TYPES to ALLOWED_CSV_MIME_TYPES for clarity
* Add comment explaining requires_csv_workflow? predicate
* Remove redundant required_column_keys from PdfImport
Base class already returns [] by default
* Add ENV toggle to disable PDF processing for non-vision endpoints
OPENAI_SUPPORTS_PDF_PROCESSING=false can be used for OpenAI-compatible
endpoints (e.g., Ollama) that don't support vision/PDF processing.
* Wire up transaction extraction for PDF bank statements
- Add extracted_data JSONB column to imports
- Add extract_transactions method to PdfImport
- Call extraction in ProcessPdfJob for bank statements
- Store transactions in extracted_data for later review
* Fix ProcessPdfJob retry logic, sanitize and localize errors
- Allow retries after partial success (classification ok, extraction failed)
- Log sanitized error message instead of raw message to avoid data leakage
- Use i18n for user-facing error messages
* Add vision-capable model validation for PDF processing
* Fix drag-and-drop test to use correct field name csv_file
* Schema bleedover from another branch
* Fix drag-drop import form field name to match controller
* Add vision capability guard to process_pdf method
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
* Add warning for TwelveData plan-restricted tickers
Fixes#800
- Add Security::PlanRestrictionTracker concern using Rails cache
- Detect plan upgrade errors during Security::Price::Importer sync
- Display amber warning on /settings/hosting with affected tickers
- Include unit tests for the new functionality
* Scope plan restriction cache by provider
Addresses review feedback:
- Cache key now includes provider name to support multiple data providers
- Methods now require provider parameter for proper scoping
- Added tests for provider-scoped restrictions
- Added documentation explaining instance-level API key architecture
* Fix RuboCop array bracket spacing
* Fix empty array bracket spacing
* Move plan upgrade detection to Provider::TwelveData
* Fix provider scoping tests to use direct cache writes
---------
Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
* fix: Show cancellation message when subscription is pending cancellation
When a subscription is cancelled via Stripe, the UI incorrectly showed
"Your contribution continues on..." instead of reflecting the cancellation
status. This fix adds tracking of `cancel_at_period_end` from Stripe webhooks
and displays "Your contribution ends on..." when a subscription has been
cancelled but is still active until the billing period ends.
https://claude.ai/code/session_01Y8ELTdK1k9o315iSq43TRN
* chore: Update schema.rb with cancel_at_period_end column
https://claude.ai/code/session_01Y8ELTdK1k9o315iSq43TRN
* Schema version
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Introduce SnapTrade integration with models, migrations, views, and activity processing logic.
* Refactor SnapTrade activities processing: improve activity fetching flow, handle pending states, and update UI elements for enhanced user feedback.
* Update Brakeman ignore file to include intentional redirect for SnapTrade OAuth portal.
* Refactor SnapTrade models, views, and processing logic: add currency extraction helper, improve pending state handling, optimize migration checks, and enhance user feedback in UI.
* Remove encryption for SnapTrade `snaptrade_user_id`, as it is an identifier, not a secret.
* Introduce `SnaptradeConnectionCleanupJob` to asynchronously handle SnapTrade connection cleanup and improve i18n for SnapTrade item status messages.
* Update SnapTrade encryption: make `snaptrade_user_secret` non-deterministic to enhance security.
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: luckyPipewrench <luckypipewrench@proton.me>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* **Add Coinbase integration with item and account management**
- Creates migrations for `coinbase_items` and `coinbase_accounts`.
- Adds models, controllers, views, and background tasks to support account linking, syncing, and transaction handling.
- Implements Coinbase API client and adapter for seamless integration.
- Supports ActiveRecord encryption for secure credential storage.
- Adds UI components for provider setup, account management, and synchronization.
* Localize Coinbase-related UI strings, refine account linking for security, and add timeouts to Coinbase API requests.
* Localize Coinbase account handling to support native currencies (USD, EUR, GBP, etc.) across balances, trades, holdings, and transactions.
* Improve Coinbase processing with timezone-safe parsing, native currency support, and immediate holdings updates.
* Improve trend percentage formatting and enhance race condition handling for Coinbase account linking.
* Fix log message wording for orphan cleanup
* Ensure `selected_accounts` parameter is sanitized by rejecting blank entries.
* Add tests for Coinbase integration: account, item, and controller coverage
- Adds unit tests for `CoinbaseAccount` and `CoinbaseItem` models.
- Adds integration tests for `CoinbaseItemsController`.
- Introduces Stimulus `select-all` controller for UI checkbox handling.
- Localizes UI strings and logging for Coinbase integration.
* Update test fixtures to use consistent placeholder API keys and secrets
* Refine `coinbase_item` tests to ensure deterministic ordering and improve scope assertions.
* Integrate `SyncStats::Collector` into Coinbase syncer to streamline statistics collection and enhance consistency.
* Localize Coinbase sync status messages and improve sync summary test coverage.
* Update `CoinbaseItem` encryption: use deterministic encryption for `api_key` and standard for `api_secret`.
* fix schema drift
* Beta labels to lower expectations
---------
Co-authored-by: luckyPipewrench <luckypipewrench@proton.me>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Rename billing to payment throughout the codebase
This change updates terminology from "billing" to "payment" to better
reflect that these are contributions/payments rather than bills.
Changes include:
- Rename BillingsController to PaymentsController
- Rename billing_email to payment_email
- Rename next_billing_date to next_payment_date
- Rename create_billing_portal_session_url to create_payment_portal_session_url
- Update routes from billing to payment
- Update all 12 locale files with new terminology
- Update views, helpers, and tests
* Update app/views/subscriptions/upgrade.html.erb
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
---------
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Generic LLMs improvements
1. app/models/provider/openai/auto_categorizer.rb:535-540
- If description is blank, now falls back to merchant name
2. app/models/provider/openai/auto_merchant_detector.rb:492-498
- Now includes merchant first, then description (joined with " - ")
* FIX linter
* FIX issue with stock price retrieval on weekend
* make weekend provisional and increase lookback
* FIX query error
* fix gap fill
The bug: When a price is provisional but the provider doesn't return a new value (weekends), we fall back to the existing DB value instead of gap-filling from Friday's correct price.
* Update importer.rb
Align provider fetch to use PROVISIONAL_LOOKBACK_DAYS for consistency. In the DB fallback, derive currency from provider_prices or db_prices and filter the query accordingly.
* Update 20260110122603_mark_suspicious_prices_provisional.rb
* Delete db/migrate/20260110122603_mark_suspicious_prices_provisional.rb
Signed-off-by: soky srm <sokysrm@gmail.com>
* Update importer.rb
* FIX tests
* FIX last tests
* Update importer_test.rb
The test doesn't properly force effective_start_date to skip old dates because there are many missing dates between the old date and recent dates. Let me fix it to properly test the subset processing scenario.
---------
Signed-off-by: soky srm <sokysrm@gmail.com>
* Feat(CoinStats): Scaffold implementation, not yet functional
* Feat(CoinStats): Implement crypto wallet balance and transactions
* Feat(CoinStats): Add tests, Minor improvements
* Feat(CoinStats): Utilize bulk fetch API endpoints
* Feat(CoinStats): Migrate strings to i8n
* Feat(CoinStats): Fix error handling in wallet link modal
* Feat(CoinStats): Implement hourly provider sync job
* Feat(CoinStats): Generate docstrings
* Fix(CoinStats): Validate API Key on provider update
* Fix(Providers): Safely handle race condition in merchance creation
* Fix(CoinStats): Don't catch system signals in account processor
* Fix(CoinStats): Preload before iterating accounts
* Fix(CoinStats): Add no opener / referrer to API dashboard link
* Fix(CoinStats): Use strict matching for symbols
* Fix(CoinStats): Remove dead code in transactions importer
* Fix(CoinStats): Avoid transaction fallback ID collisions
* Fix(CoinStats): Improve Blockchains fetch error handling
* Fix(CoinStats): Enforce NOT NULL constraint for API Key schema
* Fix(CoinStats): Migrate sync status strings to i8n
* Fix(CoinStats): Use class name rather than hardcoded string
* Fix(CoinStats): Use account currency rather than hardcoded USD
* Fix(CoinStats): Migrate from standalone to Provider class
* Fix(CoinStats): Fix test failures due to string changes
* Add tests and enhance logic for SimpleFin account synchronization and reconciliation
- Added retry logic with exponential backoff for network errors in `Provider::Simplefin`.
- Introduced tests to verify retry functionality and error handling for rate-limit, server errors, and stale data.
- Updated `SimplefinItem` to detect stale sync status and reconciliation issues.
- Enhanced UI to display stale sync warnings and data integrity notices.
- Improved SimpleFin account matching during updates with multi-tier strategy (ID, fingerprint, fuzzy match).
- Added transaction reconciliation logic to detect data gaps, transaction count drops, and duplicate transaction IDs.
* Introduce `SimplefinConnectionUpdateJob` for asynchronous SimpleFin connection updates
- Moved SimpleFin connection update logic to `SimplefinConnectionUpdateJob` to improve response times by offloading network retries, data fetching, and reconciliation tasks.
- Enhanced SimpleFin account matching with a multi-tier strategy (ID, fingerprint, fuzzy name match).
- Added retry logic and bounded latency for token claim requests in `Provider::Simplefin`.
- Updated tests to cover the new job flow and ensure correct account reconciliation during updates.
* Remove unused SimpleFin account matching logic and improve error handling in `SimplefinConnectionUpdateJob`
- Deleted the multi-tier account matching logic from `SimplefinItemsController` as it is no longer used.
- Enhanced error handling in `SimplefinConnectionUpdateJob` to gracefully handle import failures, ensuring orphaned items can be manually resolved.
- Updated job flow to conditionally set item status based on the success of import operations.
* Fix SimpleFin sync: check both legacy FK and AccountProvider for linked accounts
* Add crypto, checking, savings, and cash account detection; refine subtype selection and linking
- Enhanced `Simplefin::AccountTypeMapper` to include detection for crypto, checking, savings, and standalone cash accounts.
- Improved subtype selection UI with validation and warning indicators for missing selections.
- Updated SimpleFin account linking to handle both legacy FK and `AccountProvider` associations consistently.
- Refined job flow and importer logic for better handling of linked accounts and subtype inference.
* Improve `SimplefinConnectionUpdateJob` and holdings processing logic
- Fixed race condition in `SimplefinConnectionUpdateJob` by moving `destroy_later` calls outside of transactions.
- Updated fuzzy name match logic to use Levenshtein distance for better accuracy.
- Enhanced synthetic ticker generation in holdings processor with hash suffix for uniqueness.
* Refine SimpleFin entry processing logic and ensure `extra` data persistence
- Simplified pending flag determination to rely solely on provider-supplied values.
- Fixed potential stale values in `extra` by ensuring deep merge overwrite with `entry.transaction.save!`.
* Replace hardcoded fallback transaction description with localized string
* Refine pending flag logic in SimpleFin processor tests
- Adjust test to prevent falsely inferring pending status from missing posted dates.
- Ensure provider explicitly sets pending flag for transactions.
* Add `has_many :holdings` association to `AccountProvider` with `dependent: :nullify`
---------
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
* Update SimpleFIN relinking flow and enhance duplicate account handling
- Updated logic to allow relinking of SimpleFIN accounts while preserving legacy mappings.
- Introduced clean-up logic to hide orphaned duplicate accounts after relinking.
- Enhanced UI to display current mappings for linked accounts.
- Improved test coverage for relinking scenarios and SimpleFIN account visibility.
* Localize SimpleFIN account selection messages and remove hardcoded text
- Added translations for user-facing messages in `select_existing_account` flow (`pt-BR` and `en` locales).
- Replaced hardcoded strings in the view with localized keys.
* Localize Enable Banking and SimpleFIN account linking messages; add support for investment accounts.
- Added translations for Enable Banking and SimpleFIN account linking flows.
- Updated views and controllers to replace hardcoded strings with localized keys.
- Introduced support for investment accounts in `Provider::LunchflowAdapter`.
- Enhanced relinking logic for SimpleFIN accounts and improved test coverage for related scenarios.
---------
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
* - Add support for `SIMPLEFIN_INCLUDE_PENDING` to control pending behavior via ENV.
- Enhance debug logging for SimpleFin API requests and raw payloads.
- Refine pending flag handling in `SimplefinEntry::Processor` based on provider data and inferred conditions.
- Improve FX metadata processing for transactions with currency mismatches.
- Add new tests for pending detection, FX metadata, and edge cases involving `posted` values.
- Add pending indicator UI to transaction view.
* Document pending transaction detection, storage, and UI behavior for SimpleFIN and Plaid integrations. Add debug flags for troubleshooting.
* Add `pending?` method to `Transaction` model, refactor UI indicator, and centralize SimpleFIN configuration
- Introduced `pending?` method in `Transaction` for unified pending state detection.
- Refactored transaction pending indicator in the UI to use `pending?` method.
- Centralized SimpleFIN configuration in initializer with ENV-backed toggles.
- Updated tests for `pending?` behavior and clarified docs for pending detection logic
* Add SimpleFIN debug and runtime flags to `.env.local.example` and `.env.test.example`
- Introduced `SIMPLEFIN_INCLUDE_PENDING` and `SIMPLEFIN_DEBUG_RAW` flags for controlling pending behavior and debugging.
- Updated example environment files with descriptions for new configuration options.
* Normalize formatting for `SIMPLEFIN_INCLUDE_PENDING` and `SIMPLEFIN_DEBUG_RAW` flags in `.env.local.example` and `.env.test.example`.
---------
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
* Initial implementation
* FIX keys
* Add langfuse evals support
* FIX trace upload
* Delete .claude/settings.local.json
Signed-off-by: soky srm <sokysrm@gmail.com>
* Update client.rb
* Small LLMs improvements
* Keep batch size normal
* Update categorizer
* FIX json mode
* Add reasonable alternative to matching
* FIX thinking blocks for llms
* Implement json mode support with AUTO mode
* Make auto default for everyone
* FIX linter
* Address review
* Allow export manual categories
* FIX user export
* FIX oneshot example pollution
* Update categorization_golden_v1.yml
* Update categorization_golden_v1.yml
* Trim to 100 items
* Update auto_categorizer.rb
* FIX for auto retry in auto mode
* Separate the Eval Logic from the Auto-Categorizer
The expected_null_count parameter conflates eval-specific logic with production categorization logic.
* Force json mode on evals
* Introduce a more mixed dataset
150 items, performance from a local model:
By Difficulty:
easy: 93.22% accuracy (55/59)
medium: 93.33% accuracy (42/45)
hard: 92.86% accuracy (26/28)
edge_case: 100.0% accuracy (18/18)
* Improve datasets
Remove Data leakage from prompts
* Create eval runs as "pending"
---------
Signed-off-by: soky srm <sokysrm@gmail.com>
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Initial enable banking implementation
* Handle multiple connections
* Amount fixes
* Account type mapping
* Add option to skip accounts
* Update schema.rb
* Transaction fixes
* Provider fixes
* FIX account identifier
* FIX support unlinking
* UI style fixes
* FIX safe redirect and brakeman issue
* FIX
- pagination max fix
- wrap crud in transaction logic
* FIX api uid access
- The Enable Banking API expects the UUID (uid from the API response) to fetch balances/transactions, not the identification_hash
* FIX add new connection
* FIX erb code
* Alert/notice box overflow protection
* Give alert/notification boxes room to grow (3 lines max)
* Add "Enable Banking (beta)" to `/settings/bank_sync`
* Make Enable Banking section collapsible like all others
* Add callback hint to error message
---------
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Improvements
- Fix button visibility in reports on light theme
- Unify logic for provider syncs
- Add default option is to skip accounts linking ( no op default )
* Stability fixes and UX improvements
* FIX add unlinking when deleting lunch flow connection as well
* Wrap updates in transaction
* Some more improvements
* FIX proper provider setup check
* Make provider section collapsible
* Fix balance calculation
* Restore focus ring
* Use browser default focus
* Fix lunch flow balance for credit cards
* Track failed LLM API calls in llm_usages table
This commit adds comprehensive error tracking for failed LLM API calls:
- Updated LlmUsage model with helper methods to identify failed calls
and retrieve error details (failed?, http_status_code, error_message)
- Modified Provider::Openai to record failed API calls with error metadata
including HTTP status codes and error messages in both native and
generic chat response methods
- Enhanced UsageRecorder concern with record_usage_error method to support
error tracking for auto-categorization and auto-merchant detection
- Updated LLM usage UI to display failed calls with:
- Red background highlighting for failed rows
- Error indicator icon with "Failed" label
- Interactive tooltip on hover showing error message and HTTP status code
Failed calls are now tracked with zero tokens and null cost, storing
error details in the metadata JSONB column for visibility and debugging.
* Dark mode fixes
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Move provider config to family
* remove global settings
* Remove turbo auto submit
* Fix flash location
* Fix mssing syncer for lunchflow
* Update schema.rb
* FIX tests and encryption config
* FIX make rabbit happy
* FIX run migration in SQL
* FIX turbo frame modal
* Branding fixes
* FIX rabbit
* OCD with product names
* More OCD
* No other console.log|warn in codebase
---------
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* FIX holdings and exchange rate
- Fix holdings not showing total return instead showing day change %
- Fix exchange rate saving 0 for holidays, use LOCF - Last Observation Carried Forward
* Fix failing test
Some security prices get returned in non-standard minor units (eg. `GBp` for Great British Pence, instead of `GBP` for Great British Pounds), leading to incorrect handling of the returned price data.
This commit adds conversion logic for the two currencies I've been able to find examples of this affecting.
* Remove plaid initialiser
The initializer can be safely removed because:
- Config is lazily loaded via Provider::Registry
- reload_configuration is called after settings updates
- All calling code handles nil configs gracefully
- Initial nil state is fine - config loads on first use
* Fix for missing config
* Actually don't pollute application.rb
* Add currency loading for balances
* Fix race condition on lazy load
* Allow loans to be imported in lunch flow also
* Fix currency processor
* First pass lunch flow
* Fixes
- Fix apikey not being saved properly due to provider no reload support
- Fix proper messages if we try to link existing accounts.
* Fix better error handling
* Filter existing transactions and skip duplicates
* FIX messaging
* Branding :)
* Fix XSS and linter
* FIX provider concern
- also fix code duplication
* FIX md5 digest
* Updated determine_sync_start_date to be account-aware
* Review fixes
* Broaden error catch to not crash UI
* Fix buttons styling
* FIX process account error handling
* FIX account cap and url parsing
* Lunch Flow brand
* Found orphan i18n strings
* Remove per conversation with @sokie
---------
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Add support for dynamic config UI
* Add support for section description
* Better dynamic class settings
Added dynamic_fields hash field - Stores all undeclared settings
[] method - Checks declared fields first, then falls back to dynamic hash
[]= method - Updates declared fields normally, stores others in hash
No runtime field declaration - Fields are never dynamically created on the class
* FIX proper lookup for provider keys
- Also validate configurable values properly.
- Change Provider factory to use Rails autoloading (Zeitwerk)
* Fix factory
The derive_adapter_name method relies on string manipulation ("PlaidAccount".sub(/Account$/, "") + "Adapter" → "PlaidAdapter"), but we already have explicit registration in place.
* Make updates atomic, field-aware, and handle blanks explicitly
* Small UX detail
* Add support for PlaidEU in UI also
- This looks like partial support atm
* Implement Yahoo Finance
* Added tests
* Updated hosting controller to check for managed app_mode instead of env_override
* Suggestions from CodeRabbit and Fixes on tests
* Remove Css changes
* Fix yahoo finance impl and i18n
* Updated view to use healthy method
* remove usage
* Updated env example
* keep usage on class just to keep same format
* Ci test
* Remove some useless validations
* Remove logs
* Linter fixes
* Broke this in my conflict merge
* Wrong indentation level
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Implement providers factory
* Multiple providers sync support
- Proper Multi-Provider Syncing: When you click sync on an account with multiple providers (e.g., both Plaid and SimpleFin), all provider items are synced
- Better API: The existing account.providers method already returns all providers, and account.provider returns the first one for backward compatibility
- Correct Holdings Deletion Logic: Holdings can only be deleted if ALL providers allow it, preventing accidental deletions that would be recreated on next sync
TODO: validate this is the way we want to go? We would need to check holdings belong to which account, and then check provider allows deletion. More complex
- Database Constraints: The existing validations ensure an account can have at most one provider of each type (one PlaidAccount, one SimplefinAccount, etc.)
* Add generic provider_import_adapter
* Finish unified import strategy
* Update app/models/plaid_account.rb
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: soky srm <sokysrm@gmail.com>
* Update app/models/provider/factory.rb
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: soky srm <sokysrm@gmail.com>
* Fix account linked by plaid_id instead of external_id
* Parse numerics to BigDecimal
Parse numerics to BigDecimal before computing amount; guard nils.
Avoid String * String and float drift; also normalize date.
* Fix incorrect usage of assert_raises.
* Fix linter
* Fix processor test.
* Update current_balance_manager.rb
* Test fixes
* Fix plaid linked account test
* Add support for holding per account_provider
* Fix proper account access
Also fix account deletion for simpefin too
* FIX match tests for consistency
* Some more factory updates
* Fix account schema for multipe providers
Can do:
- Account #1 → PlaidAccount + SimplefinAccount (multiple different providers)
- Account #2 → PlaidAccount only
- Account #3 → SimplefinAccount only
Cannot do:
- Account #1 → PlaidAccount + PlaidAccount (duplicate provider type)
- PlaidAccount #123 → Account #1 + Account #2 (provider linked to multiple accounts)
* Fix account setup
- An account CAN have multiple providers (the schema shows account_providers with unique index on [account_id, provider_type])
- Each provider should maintain its own separate entries
- We should NOT update one provider's entry when another provider syncs
* Fix linter and guard migration
* FIX linter issues.
* Fixes
- Remove duplicated index
- Pass account_provider_id
- Guard holdings call to avoid NoMethodError
* Update schema and provider import fix
* Plaid doesn't allow holdings deletion
* Use ClimateControl for proper env setup
* No need for this in .git
---------
Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
