* Fix demo refresh email config lookup
Use indifferent access when reading demo config so the refresh job works whether config keys are strings or symbols. Add a regression test covering symbol-keyed config_for output.
* Remove unnecessary blank line in test file
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Remove "value" from the market_value fallback chain in the SimpleFIN
HoldingsProcessor and add it to the cost_basis fallback chain instead.
Some brokerages (Vanguard, Fidelity) use "value" to represent cost basis,
causing the system to display average cost per share as the current price
and show massive phantom losses.
https://claude.ai/code/session_01V2gC6BPT3sF7Hu4XQgUQT4
Co-authored-by: Claude <noreply@anthropic.com>
* Add scheduled demo family refresh job
Rebuild demo data daily at 5am UTC by anonymizing and enqueueing deletion of the existing demo family while immediately generating new sample data. Add super-admin email notifications with 24-hour session and signup metrics, plus tests for the new job and mailer.
* Delete demo monitoring key before family refresh
Ensure DemoFamilyRefreshJob removes ApiKey::DEMO_MONITORING_KEY from the old demo family before enqueueing async family destruction and generating replacement sample data. Adds a regression assertion that the key is gone before generator execution.
* Add post-trial inactive family cleanup with data archival
Families that expire their trial without subscribing now get cleaned up
daily. Empty families (no accounts) are destroyed immediately after a
14-day grace period. Families with meaningful data (12+ transactions,
some recent) get their data exported as NDJSON/ZIP to an ArchivedExport
record before deletion, downloadable via a token-based URL for 90 days.
- Add InactiveFamilyCleanerJob (scheduled daily at 4 AM, managed mode only)
- Add ArchivedExport model with token-based downloads
- Add inactive_trial_for_cleanup scope and requires_data_archive? to Family
- Extend DataCleanerJob to purge expired archived exports
- Add ArchivedExportsController for unauthenticated token downloads
https://claude.ai/code/session_01LR3Vo83R5s5SczYe6T33dQ
* Fix Brakeman redirect warning in ArchivedExportsController
Use rails_blob_path instead of redirecting directly to the ActiveStorage
attachment, which avoids the allow_other_host: true open redirect.
https://claude.ai/code/session_01LR3Vo83R5s5SczYe6T33dQ
* Update schema.rb with archived_exports table
Add the archived_exports table definition to schema.rb to match
the pending migration, unblocking CI tests.
https://claude.ai/code/session_01LR3Vo83R5s5SczYe6T33dQ
* Fix broken CI tests for ArchivedExports and InactiveFamilyCleaner
- ArchivedExportsController 404 test: use assert_response :not_found
instead of assert_raises since Rails rescues RecordNotFound in
integration tests and returns a 404 response.
- InactiveFamilyCleanerJob test: remove assert_no_difference on
Family.count since the inactive_trial fixture gets cleaned up by
the job. The test intent is to verify the active family survives,
which is checked by assert Family.exists?.
https://claude.ai/code/session_01LR3Vo83R5s5SczYe6T33dQ
* Wrap ArchivedExport creation in a transaction
Ensure the ArchivedExport record and its file attachment succeed
atomically. If the attach fails, the transaction rolls back so no
orphaned record is left without an export file.
https://claude.ai/code/session_01LR3Vo83R5s5SczYe6T33dQ
* Store only a digest of the download token for ArchivedExport
Replace plaintext download_token column with download_token_digest
(SHA-256 hex). The raw token is generated via SecureRandom on create,
exposed transiently via attr_reader for use in emails/logs, and only
its digest is persisted. Lookup uses find_by_download_token! which
digests the incoming token before querying.
https://claude.ai/code/session_01LR3Vo83R5s5SczYe6T33dQ
* Remove raw download token from cleanup job logs
Log a truncated digest prefix instead of the raw token, which is the
sole credential for the unauthenticated download endpoint.
https://claude.ai/code/session_01LR3Vo83R5s5SczYe6T33dQ
* Fix empty assert_no_difference block in cleaner job test
Wrap the perform_now call with both assertions so the
ArchivedExport.count check actually exercises the job.
https://claude.ai/code/session_01LR3Vo83R5s5SczYe6T33dQ
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat: Add PDF import with AI-powered document analysis
This enhances the import functionality to support PDF files with AI-powered
document analysis. When a PDF is uploaded, it is processed by AI to:
- Identify the document type (bank statement, credit card statement, etc.)
- Generate a summary of the document contents
- Extract key metadata (institution, dates, balances, transaction count)
After processing, an email is sent to the user asking for next steps.
Key changes:
- Add PdfImport model for handling PDF document imports
- Add Provider::Openai::PdfProcessor for AI document analysis
- Add ProcessPdfJob for async PDF processing
- Add PdfImportMailer for user notification emails
- Update imports controller to detect and handle PDF uploads
- Add PDF import option to the new import page
- Add i18n translations for all new strings
- Add comprehensive tests for the new functionality
* Add bank statement import with AI extraction
- Create ImportBankStatement assistant function for MCP
- Add BankStatementExtractor with chunked processing for small context windows
- Register function in assistant configurable
- Make PdfImport#pdf_file_content public for extractor access
- Increase OpenAI request timeout to 600s for slow local models
- Increase DB connection pool to 20 for concurrent operations
Tested with M-Pesa bank statement via remote Ollama (qwen3:8b):
- Successfully extracted 18 transactions
- Generated CSV and created TransactionImport
- Works with 3000 char chunks for small context windows
* Add pdf-reader gem dependency
The BankStatementExtractor uses PDF::Reader to parse bank statement
PDFs, but the gem was not properly declared in the Gemfile. This would
cause NameError in production when processing bank statements.
Added pdf-reader ~> 2.12 to Gemfile dependencies.
* Fix transaction deduplication to preserve legitimate duplicates
The previous deduplication logic removed ALL duplicate transactions based
on [date, amount, name], which would drop legitimate same-day duplicates
like multiple ATM withdrawals or card authorizations.
Changed to only deduplicate transactions that appear in consecutive chunks
(chunking artifacts) while preserving all legitimate duplicates within the
same chunk or non-adjacent chunks.
* Refactor bank statement extraction to use public provider method
Address code review feedback:
- Add public extract_bank_statement method to Provider::Openai
- Remove direct access to private client via send(:client)
- Update ImportBankStatement to use new public method
- Add require 'set' to BankStatementExtractor
- Remove PII-sensitive content from error logs
- Add defensive check for nil response.error
- Handle oversized PDF pages in chunking logic
- Remove unused process_native and process_generic methods
- Update email copy to reflect feature availability
- Add guard for nil document_type in email template
- Document pdf-reader gem rationale in Gemfile
Tested with both OpenAI (gpt-4o) and Ollama (qwen3:8b):
- OpenAI: 49 transactions extracted in 30s
- Ollama: 40 transactions extracted in 368s
- All encapsulation and error handling working correctly
* Update schema.rb with ai_summary and document_type columns
* Address PR #808 review comments
- Rename :csv_file to :import_file across controllers/views/tests
- Add PDF test fixture (sample_bank_statement.pdf)
- Add supports_pdf_processing? method for graceful degradation
- Revert unrelated database.yml pool change (600->3)
- Remove month_start_day schema bleed from other PR
- Fix PdfProcessor: use .strip instead of .strip_heredoc
- Add server-side PDF magic byte validation
- Conditionally show PDF import option when AI provider available
- Fix ProcessPdfJob: sanitize errors, handle update failure
- Move pdf_file attachment from Import to PdfImport
- Document deduplication logic limitations
- Fix ImportBankStatement: catch specific exceptions only
- Remove unnecessary require 'set'
- Remove dead json_schema method from PdfProcessor
- Reduce default OpenAI timeout from 600s to 60s
- Fix nil guard in text mailer template
- Add require 'csv' to ImportBankStatement
- Remove Gemfile pdf-reader comment
* Fix RuboCop indentation in ProcessPdfJob
* Refactor PDF import check to use model predicate method
Replace is_a?(PdfImport) type check with requires_csv_workflow? predicate
that leverages STI inheritance for cleaner controller logic.
* Fix missing 'unknown' locale key and schema version mismatch
- Add 'unknown: Unknown Document' to document_types locale
- Fix schema version to match latest migration (2026_01_24_180211)
* Document OPENAI_REQUEST_TIMEOUT env variable
Added to .env.local.example and docs/hosting/ai.md
* Rename ALLOWED_MIME_TYPES to ALLOWED_CSV_MIME_TYPES for clarity
* Add comment explaining requires_csv_workflow? predicate
* Remove redundant required_column_keys from PdfImport
Base class already returns [] by default
* Add ENV toggle to disable PDF processing for non-vision endpoints
OPENAI_SUPPORTS_PDF_PROCESSING=false can be used for OpenAI-compatible
endpoints (e.g., Ollama) that don't support vision/PDF processing.
* Wire up transaction extraction for PDF bank statements
- Add extracted_data JSONB column to imports
- Add extract_transactions method to PdfImport
- Call extraction in ProcessPdfJob for bank statements
- Store transactions in extracted_data for later review
* Fix ProcessPdfJob retry logic, sanitize and localize errors
- Allow retries after partial success (classification ok, extraction failed)
- Log sanitized error message instead of raw message to avoid data leakage
- Use i18n for user-facing error messages
* Add vision-capable model validation for PDF processing
* Fix drag-and-drop test to use correct field name csv_file
* Schema bleedover from another branch
* Fix drag-drop import form field name to match controller
* Add vision capability guard to process_pdf method
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
* Refactor: Enhance cash balance calculation and holdings management with money market classification and provider-sourced data handling
* Fix: Clear fixture holdings in test to ensure clean creation and update raw_holdings_payload format
---------
Co-authored-by: luckyPipewrench <luckypipewrench@proton.me>
* Feat(CoinStats): Scaffold implementation, not yet functional
* Feat(CoinStats): Implement crypto wallet balance and transactions
* Feat(CoinStats): Add tests, Minor improvements
* Feat(CoinStats): Utilize bulk fetch API endpoints
* Feat(CoinStats): Migrate strings to i8n
* Feat(CoinStats): Fix error handling in wallet link modal
* Feat(CoinStats): Implement hourly provider sync job
* Feat(CoinStats): Generate docstrings
* Fix(CoinStats): Validate API Key on provider update
* Fix(Providers): Safely handle race condition in merchance creation
* Fix(CoinStats): Don't catch system signals in account processor
* Fix(CoinStats): Preload before iterating accounts
* Fix(CoinStats): Add no opener / referrer to API dashboard link
* Fix(CoinStats): Use strict matching for symbols
* Fix(CoinStats): Remove dead code in transactions importer
* Fix(CoinStats): Avoid transaction fallback ID collisions
* Fix(CoinStats): Improve Blockchains fetch error handling
* Fix(CoinStats): Enforce NOT NULL constraint for API Key schema
* Fix(CoinStats): Migrate sync status strings to i8n
* Fix(CoinStats): Use class name rather than hardcoded string
* Fix(CoinStats): Use account currency rather than hardcoded USD
* Fix(CoinStats): Migrate from standalone to Provider class
* Fix(CoinStats): Fix test failures due to string changes
- Introduced tests for importer post-import logic and `SimplefinHoldingsApplyJob`.
- Refactored `ProviderImportAdapter` to improve holding resolution strategy.
- Added handling of investment and crypto holdings in importer with debounce logic for job enqueuing.
- Updated rake task to use `SimplefinHoldingsApplyJob` for holding materialization.
- Add support to link existing account with lunch-flow
The account will be promoted to a lunch flow connection now
( TBD if we want to allow un-linking? )
- Add support for proper de-dup at provider import level. This will handle de-dups for Lunch Flow, Plaid and SimpleFIN
- Fix plaid account removal on invalid credentials
* AI sidebar
* Add chat and message models with associations
* Implement AI chat functionality with sidebar and messaging system
- Add chat and messages controllers
- Create chat and message views
- Implement chat-related routes
- Add message broadcasting and user interactions
- Update application layout to support chat sidebar
- Enhance user model with initials method
* Refactor AI sidebar with enhanced chat menu and interactions
- Update sidebar layout with dynamic width and improved responsiveness
- Add new chat menu Stimulus controller for toggling between chat and chat list views
- Improve chat list display with recent chats and empty state
- Extract AI avatar to a partial for reusability
- Enhance message display and interaction styling
- Add more contextual buttons and interaction hints
* Improve chat scroll behavior and message styling
- Refactor chat scroll functionality with Stimulus controller
- Optimize message scrolling in chat views
- Update message styling for better visual hierarchy
- Enhance chat container layout with flex and auto-scroll
- Simplify message rendering across different chat views
* Extract AI avatar to a shared partial for consistent styling
- Refactor AI avatar rendering across chat views
- Replace hardcoded avatar markup with a reusable partial
- Simplify avatar display in chats and messages views
* Update sidebar controller to handle right panel width dynamically
- Add conditional width class for right sidebar panel
- Ensure consistent sidebar toggle behavior for both left and right panels
- Use specific width class for right panel (w-[375px])
* Refactor chat form and AI greeting with flexible partials
- Extract message form to a reusable partial with dynamic context support
- Create flexible AI greeting partial for consistent welcome messages
- Simplify chat and sidebar views by leveraging new partials
- Add support for different form scenarios (chat, new chat, sidebar)
- Improve code modularity and reduce duplication
* Add chat clearing functionality with dynamic menu options
- Implement clear chat action in ChatsController
- Add clear chat route to support clearing messages
- Update AI sidebar with dropdown menu for chat actions
- Preserve system message when clearing chat
- Enhance chat interaction with new menu options
* Add frontmatter to project structure documentation
- Create initial frontmatter for structure.mdc file
- Include description and configuration options
- Prepare for potential dynamic documentation rendering
* Update general project rules with additional guidelines
- Add rule for using `Current.family` instead of `current_family`
- Include new guidelines for testing, API routes, and solution approach
- Expand project-specific rules for more consistent development practices
* Add OpenAI gem and AI-friendly data representations
- Add `ruby-openai` gem for AI integration
- Implement `to_ai_readable_hash` methods in BalanceSheet and IncomeStatement
- Include Promptable module in both models
- Add savings rate calculation method in IncomeStatement
- Prepare financial models for AI-powered insights and interactions
* Enhance AI Financial Assistant with Advanced Querying and Debugging Capabilities
- Implement comprehensive AI financial query system with function-based interactions
- Add detailed debug logging for AI responses and function calls
- Extend BalanceSheet and IncomeStatement models with AI-friendly methods
- Create robust error handling and fallback mechanisms for AI queries
- Update chat and message views to support debug mode and enhanced rendering
- Add AI query routes and initial test coverage for financial assistant
* Refactor AI sidebar and chat layout with improved structure and comments
- Remove inline AI chat from application layout
- Enhance AI sidebar with more semantic HTML structure
- Add descriptive comments to clarify different sections of chat view
- Improve flex layout and scrolling behavior in chat messages container
- Optimize message rendering with more explicit class names and structure
* Add Markdown rendering support for AI chat messages
- Implement `markdown` helper method in ApplicationHelper using Redcarpet
- Update message view to render AI messages with Markdown formatting
- Add comprehensive Markdown rendering options (tables, code blocks, links)
- Enhance AI Financial Assistant prompt to encourage Markdown usage
- Remove commented Markdown CSS in Tailwind application stylesheet
* Missing comma
* Enhance AI response processing with chat history context
* Improve AI debug logging with payload size limits and internal message flag
* Enhance AI chat interaction with improved thinking indicator and scrolling behavior
* Add AI consent and enable/disable functionality for AI chat
* Upgrade Biome and refactor JavaScript template literals
- Update @biomejs/biome to latest version with caret (^) notation
- Refactor AI query and chat controllers to use template literals
- Standardize npm scripts formatting in package.json
* Add beta testing usage note to AI consent modal
* Update test fixtures and configurations for AI chat functionality
- Add family association to chat fixtures and tests
- Set consistent password digest for test users
- Enable AI for test users
- Add OpenAI access token for test environment
- Update chat and user model tests to include family context
* Simplify data model and get tests passing
* Remove structure.mdc from version control
* Integrate AI chat styles into existing prose pattern
* Match Figma design spec, implement Turbo frames and actions for chats controller
* AI rules refresh
* Consolidate Stimulus controllers, thinking state, controllers, and views
* Naming, domain alignment
* Reset migrations
* Improve data model to support tool calls and message types
* Tool calling tests and fixtures
* Tool call implementation and test
* Get assistant test working again
* Test updates
* Process tool calls within provider
* Chat UI back to working state again
* Remove stale code
* Tests passing
* Update openai class naming to avoid conflicts
* Reconfigure test env
* Rebuild gemfile
* Fix naming conflicts for ChatResponse
* Message styles
* Use OpenAI conversation state management
* Assistant function base implementation
* Add back thinking messages, clean up error handling for chat
* Fix sync error when security price has bad data from provider
* Add balance sheet function to assistant
* Add better function calling error visibility
* Add income statement function
* Simplify and clean up "thinking" interactions with Turbo frames
* Remove stale data definitions from functions
* Ensure VCR fixtures working with latest code
* basic stream implementation
* Get streaming working
* Make AI sidebar wider when left sidebar is collapsed
* Get tests working with streaming responses
* Centralize provider error handling
* Provider data boundaries
---------
Co-authored-by: Josh Pigford <josh@joshpigford.com>
* Add data enrichment
* Make data enrichment optional for self-hosters
* Add categories to data enrichment
* Only update category and merchant if nil
* Fix name overrides
* Lint fixes
* Basic plaid data model and linking
* Remove institutions, add plaid items
* Improve schema and Plaid provider
* Add webhook verification sketch
* Webhook verification
* Item accounts and balances sync setup
* Provide test encryption keys
* Fix test
* Only provide encryption keys in prod
* Try defining keys in test env
* Consolidate account sync logic
* Add back plaid account initialization
* Plaid transaction sync
* Sync UI overhaul for Plaid
* Add liability and investment syncing
* Handle investment webhooks and process current day holdings
* Remove logs
* Remove "all" period select for performance
* fix amount calc
* Remove todo comment
* Coming soon for investment historical data
* Document Plaid configuration
* Listen for holding updates
* Remove stale 1.0 import logic and model
* Fresh start
* Checkpoint before removing nav
* First working prototype
* Add trade, account, and mint import flows
* Basic working version with tests
* System tests for each import type
* Clean up mappings flow
* Clean up PR, refactor stale code, tests
* Add back row validations
* Row validations
* Fix import job test
* Fix import navigation
* Fix mint import configuration form
* Currency preset for new accounts
Introduces a basic CSV import module for bulk-importing account transactions.
Changes include:
- User can load a CSV
- User can configure the column mappings for a CSV
- Imported CSV shows invalid cells
- User can clean up their data directly in the UI
- User can see a preview of the import rows and confirm import
- Layout refactor + Import nav stepper
- System test stability improvements
* Support all currencies, handle outside DB
* Remove currencies from seed
* Fix account balance namespace
* Set default currency on authentication
* Cache currency instances
* Implement multi-currency syncs with tests
* Series fallback, passing tests
* Fix conflicts
* Make value group concrete class that works with currency values
* Fix migration conflict
* Update tests to expect multi-currency results
* Update account list to use group method
* Namespace updates
* Fetch unknown exchange rates from API
* Fix date range bug
* Ensure demo data works without external API
* Enforce cascades only at DB level
* Add trends, time series, seed data
* Remove test data
* Replace old view values with helpers
* Fix tooltip bugs in D3 chart
* Fix tests
* Fix smoke test
* Add CRUD actions for valuations
* Scaffold out inline editing with Turbo
* Refactor series logic
* Scaffold out basic sync process for accounts
* Fix tests
* Initial foundational pass at multi-currency
* Default format currency
* More work on currency and exchanging
* Re-build currencies on change
* Currency import/setup
* Background job overhaul + cheaper OXR plan support
* Lint fixes
* Test fixes
* Multi-currency setup instructions
* Allow decimals in the balance field
* Spacing fix for form
---------
Signed-off-by: Josh Pigford <josh@joshpigford.com>
