* Add SearchFamilyImportedFiles assistant function with vector store support
Implement per-Family document search using OpenAI vector stores, allowing
the AI assistant to search through uploaded financial documents (tax returns,
statements, contracts, etc.). The architecture is modular with a provider-
agnostic VectorStoreConcept interface so other RAG backends can be added.
Key components:
- Assistant::Function::SearchFamilyImportedFiles - tool callable from any LLM
- Provider::VectorStoreConcept - abstract vector store interface
- Provider::Openai vector store methods (create, upload, search, delete)
- Family::VectorSearchable concern with document management
- FamilyDocument model for tracking uploaded files
- Migration adding vector_store_id to families and family_documents table
https://claude.ai/code/session_01TSkKc7a9Yu2ugm1RvSf4dh
* Extract VectorStore adapter layer for swappable backends
Replace the Provider::VectorStoreConcept mixin with a standalone adapter
architecture under VectorStore::. This cleanly separates vector store
concerns from the LLM provider and makes it trivial to swap backends.
Components:
- VectorStore::Base — abstract interface (create/delete/upload/remove/search)
- VectorStore::Openai — uses ruby-openai gem's native vector_stores.search
- VectorStore::Pgvector — skeleton for local pgvector + embedding model
- VectorStore::Qdrant — skeleton for Qdrant vector DB
- VectorStore::Registry — resolves adapter from VECTOR_STORE_PROVIDER env
- VectorStore::Response — success/failure wrapper (like Provider::Response)
Consumers updated to go through VectorStore.adapter:
- Family::VectorSearchable
- Assistant::Function::SearchFamilyImportedFiles
- FamilyDocument
Removed: Provider::VectorStoreConcept, vector store methods from Provider::Openai
https://claude.ai/code/session_01TSkKc7a9Yu2ugm1RvSf4dh
* Add Vector Store configuration docs to ai.md
Documents how to configure the document search feature, covering all
three supported backends (OpenAI, pgvector, Qdrant), environment
variables, Docker Compose examples, supported file types, and privacy
considerations.
https://claude.ai/code/session_01TSkKc7a9Yu2ugm1RvSf4dh
* No need to specify `imported` in code
* Missed a couple more places
* Tiny reordering for the human OCD
* Update app/models/assistant/function/search_family_files.rb
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
* PR comments
* More PR comments
---------
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* First cut of a simplified "intro" UI layout
* Linter
* Add guest role and intro-only access
* Fix guest role UI defaults (#940)
Use enum predicate to avoid missing role helper.
* Remove legacy user role mapping (#941)
Drop the unused user role references in role normalization
and SSO role mapping forms to avoid implying a role that
never existed.
Refs: #0
* Remove role normalization (#942)
Remove role normalization
Roles are now stored directly without legacy mappings.
* Revert role mapping logic
* Remove `normalize_role_settings`
* Remove unnecessary migration
* Make `member` the default
* Broken `.erb`
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
* feat: Add PDF import with AI-powered document analysis
This enhances the import functionality to support PDF files with AI-powered
document analysis. When a PDF is uploaded, it is processed by AI to:
- Identify the document type (bank statement, credit card statement, etc.)
- Generate a summary of the document contents
- Extract key metadata (institution, dates, balances, transaction count)
After processing, an email is sent to the user asking for next steps.
Key changes:
- Add PdfImport model for handling PDF document imports
- Add Provider::Openai::PdfProcessor for AI document analysis
- Add ProcessPdfJob for async PDF processing
- Add PdfImportMailer for user notification emails
- Update imports controller to detect and handle PDF uploads
- Add PDF import option to the new import page
- Add i18n translations for all new strings
- Add comprehensive tests for the new functionality
* Add bank statement import with AI extraction
- Create ImportBankStatement assistant function for MCP
- Add BankStatementExtractor with chunked processing for small context windows
- Register function in assistant configurable
- Make PdfImport#pdf_file_content public for extractor access
- Increase OpenAI request timeout to 600s for slow local models
- Increase DB connection pool to 20 for concurrent operations
Tested with M-Pesa bank statement via remote Ollama (qwen3:8b):
- Successfully extracted 18 transactions
- Generated CSV and created TransactionImport
- Works with 3000 char chunks for small context windows
* Add pdf-reader gem dependency
The BankStatementExtractor uses PDF::Reader to parse bank statement
PDFs, but the gem was not properly declared in the Gemfile. This would
cause NameError in production when processing bank statements.
Added pdf-reader ~> 2.12 to Gemfile dependencies.
* Fix transaction deduplication to preserve legitimate duplicates
The previous deduplication logic removed ALL duplicate transactions based
on [date, amount, name], which would drop legitimate same-day duplicates
like multiple ATM withdrawals or card authorizations.
Changed to only deduplicate transactions that appear in consecutive chunks
(chunking artifacts) while preserving all legitimate duplicates within the
same chunk or non-adjacent chunks.
* Refactor bank statement extraction to use public provider method
Address code review feedback:
- Add public extract_bank_statement method to Provider::Openai
- Remove direct access to private client via send(:client)
- Update ImportBankStatement to use new public method
- Add require 'set' to BankStatementExtractor
- Remove PII-sensitive content from error logs
- Add defensive check for nil response.error
- Handle oversized PDF pages in chunking logic
- Remove unused process_native and process_generic methods
- Update email copy to reflect feature availability
- Add guard for nil document_type in email template
- Document pdf-reader gem rationale in Gemfile
Tested with both OpenAI (gpt-4o) and Ollama (qwen3:8b):
- OpenAI: 49 transactions extracted in 30s
- Ollama: 40 transactions extracted in 368s
- All encapsulation and error handling working correctly
* Update schema.rb with ai_summary and document_type columns
* Address PR #808 review comments
- Rename :csv_file to :import_file across controllers/views/tests
- Add PDF test fixture (sample_bank_statement.pdf)
- Add supports_pdf_processing? method for graceful degradation
- Revert unrelated database.yml pool change (600->3)
- Remove month_start_day schema bleed from other PR
- Fix PdfProcessor: use .strip instead of .strip_heredoc
- Add server-side PDF magic byte validation
- Conditionally show PDF import option when AI provider available
- Fix ProcessPdfJob: sanitize errors, handle update failure
- Move pdf_file attachment from Import to PdfImport
- Document deduplication logic limitations
- Fix ImportBankStatement: catch specific exceptions only
- Remove unnecessary require 'set'
- Remove dead json_schema method from PdfProcessor
- Reduce default OpenAI timeout from 600s to 60s
- Fix nil guard in text mailer template
- Add require 'csv' to ImportBankStatement
- Remove Gemfile pdf-reader comment
* Fix RuboCop indentation in ProcessPdfJob
* Refactor PDF import check to use model predicate method
Replace is_a?(PdfImport) type check with requires_csv_workflow? predicate
that leverages STI inheritance for cleaner controller logic.
* Fix missing 'unknown' locale key and schema version mismatch
- Add 'unknown: Unknown Document' to document_types locale
- Fix schema version to match latest migration (2026_01_24_180211)
* Document OPENAI_REQUEST_TIMEOUT env variable
Added to .env.local.example and docs/hosting/ai.md
* Rename ALLOWED_MIME_TYPES to ALLOWED_CSV_MIME_TYPES for clarity
* Add comment explaining requires_csv_workflow? predicate
* Remove redundant required_column_keys from PdfImport
Base class already returns [] by default
* Add ENV toggle to disable PDF processing for non-vision endpoints
OPENAI_SUPPORTS_PDF_PROCESSING=false can be used for OpenAI-compatible
endpoints (e.g., Ollama) that don't support vision/PDF processing.
* Wire up transaction extraction for PDF bank statements
- Add extracted_data JSONB column to imports
- Add extract_transactions method to PdfImport
- Call extraction in ProcessPdfJob for bank statements
- Store transactions in extracted_data for later review
* Fix ProcessPdfJob retry logic, sanitize and localize errors
- Allow retries after partial success (classification ok, extraction failed)
- Log sanitized error message instead of raw message to avoid data leakage
- Use i18n for user-facing error messages
* Add vision-capable model validation for PDF processing
* Fix drag-and-drop test to use correct field name csv_file
* Schema bleedover from another branch
* Fix drag-drop import form field name to match controller
* Add vision capability guard to process_pdf method
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
- Refactored `persist_holdings` to separate and conditionally upsert holdings with and without cost_basis.
- Updated `avg_cost` logic to treat 0 cost_basis as unknown and return nil when cost_basis cannot be determined.
- Modified trend and investment calculation to exclude holdings with unknown cost_basis.
- Adjusted `average_cost` formatting to handle nil values in API responses and views.
- Added comprehensive tests to ensure cost_basis preservation and fallback behavior.
- Localized `unknown` label for display when cost_basis is unavailable.
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
* Implement providers factory
* Multiple providers sync support
- Proper Multi-Provider Syncing: When you click sync on an account with multiple providers (e.g., both Plaid and SimpleFin), all provider items are synced
- Better API: The existing account.providers method already returns all providers, and account.provider returns the first one for backward compatibility
- Correct Holdings Deletion Logic: Holdings can only be deleted if ALL providers allow it, preventing accidental deletions that would be recreated on next sync
TODO: validate this is the way we want to go? We would need to check holdings belong to which account, and then check provider allows deletion. More complex
- Database Constraints: The existing validations ensure an account can have at most one provider of each type (one PlaidAccount, one SimplefinAccount, etc.)
* Add generic provider_import_adapter
* Finish unified import strategy
* Update app/models/plaid_account.rb
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: soky srm <sokysrm@gmail.com>
* Update app/models/provider/factory.rb
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: soky srm <sokysrm@gmail.com>
* Fix account linked by plaid_id instead of external_id
* Parse numerics to BigDecimal
Parse numerics to BigDecimal before computing amount; guard nils.
Avoid String * String and float drift; also normalize date.
* Fix incorrect usage of assert_raises.
* Fix linter
* Fix processor test.
* Update current_balance_manager.rb
* Test fixes
* Fix plaid linked account test
* Add support for holding per account_provider
* Fix proper account access
Also fix account deletion for simpefin too
* FIX match tests for consistency
* Some more factory updates
* Fix account schema for multipe providers
Can do:
- Account #1 → PlaidAccount + SimplefinAccount (multiple different providers)
- Account #2 → PlaidAccount only
- Account #3 → SimplefinAccount only
Cannot do:
- Account #1 → PlaidAccount + PlaidAccount (duplicate provider type)
- PlaidAccount #123 → Account #1 + Account #2 (provider linked to multiple accounts)
* Fix account setup
- An account CAN have multiple providers (the schema shows account_providers with unique index on [account_id, provider_type])
- Each provider should maintain its own separate entries
- We should NOT update one provider's entry when another provider syncs
* Fix linter and guard migration
* FIX linter issues.
* Fixes
- Remove duplicated index
- Pass account_provider_id
- Guard holdings call to avoid NoMethodError
* Update schema and provider import fix
* Plaid doesn't allow holdings deletion
* Use ClimateControl for proper env setup
* No need for this in .git
---------
Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Password reset back button also after confirmation
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
* Implement a filter for category (#215)
- Also implement an is empty/is null condition.
* Implement an LLM cost estimation page
Track costs across all the cost categories: auto categorization, auto merchant detection and chat.
Show warning with estimated cost when running a rule that contains AI.
* Update pricing
* Add google pricing
and fix inferred model everywhere.
* Update app/models/llm_usage.rb
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: soky srm <sokysrm@gmail.com>
* FIX address review
* Linter
* Address review
- Lowered log level
- extracted the duplicated record_usage method into a shared concern
* Update app/controllers/settings/llm_usages_controller.rb
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: soky srm <sokysrm@gmail.com>
* Moved attr_reader out of private
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Implement support for generic OpenAI api
- Implements support to route requests to any openAI capable provider ( Deepsek, Qwen, VLLM, LM Studio, Ollama ).
- Keeps support for pure OpenAI and uses the new better responses api
- Uses the /chat/completions api for the generic providers
- If uri_base is not set, uses default implementation.
* Fix json handling and indentation
* Fix linter error indent
* Fix tests to set env vars
* Fix updating settings
* Change to prefix checking for OAI models
* FIX check model if custom uri is set
* Change chat to sync calls
Some local models don't support streaming. Revert to sync calls for generic OAI api
* Fix tests
* Fix tests
* Fix for gpt5 message extraction
- Finds the message output by filtering for "type" == "message" instead of assuming it's at index 0
- Safely extracts the text using safe navigation operators (&.)
- Raises a clear error if no message content is found
- Parses the JSON as before
* Add more langfuse logging
- Add Langfuse to auto categorizer and merchant detector
- Fix monitoring on streaming chat responses
- Add Langfuse traces also for model errors now
* Update app/models/provider/openai.rb
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: soky srm <sokysrm@gmail.com>
* handle nil function results explicitly
* Exposing some config vars.
* Linter and nitpick comments
* Drop back to `gpt-4.1` as default for now
* Linter
* Fix for strict tool schema in Gemini
- This fixes tool calling in Gemini OpenAI api
- Fix for getTransactions function, page size is not used.
---------
Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Replace Maybe for Sure in select code areas
* Make sure passwords are consistent
* Remove (admin|member) from demo data first name
* Database and schema names finally to `sure`
* Fix broken test
* Another (benchmarking) database name to `sure_*`
* More rebranding to Sure
* Missed this Maybe mention in the same page
* Random nitpicks and more Maybes
* Demo data accounts and more Maybes
* Test data account updates
* Impersonation test accounts
* Consistency with `compose.example.yml`
* Balance sheet cache layer with cache-busting
* Update family cache timestamps during Sync
* Less blocking sync loaders
* Consolidate family data caching key logic
* Fix turbo stream broadcasts
* Remove dev delay
* Add back account group sorting
* Save work
* Subscriptions and trials domain
* Store family ID on customer
* Remove indirection of stripe calls
* Test simplifications
* Update brakeman
* Fix stripe tests in CI
* Update billing page to show subscription details
* Remove legacy columns
* Complete billing settings page
* Fix hardcoded plan name
* Handle subscriptions for self hosting mode
* Lint fixes
* AI sidebar
* Add chat and message models with associations
* Implement AI chat functionality with sidebar and messaging system
- Add chat and messages controllers
- Create chat and message views
- Implement chat-related routes
- Add message broadcasting and user interactions
- Update application layout to support chat sidebar
- Enhance user model with initials method
* Refactor AI sidebar with enhanced chat menu and interactions
- Update sidebar layout with dynamic width and improved responsiveness
- Add new chat menu Stimulus controller for toggling between chat and chat list views
- Improve chat list display with recent chats and empty state
- Extract AI avatar to a partial for reusability
- Enhance message display and interaction styling
- Add more contextual buttons and interaction hints
* Improve chat scroll behavior and message styling
- Refactor chat scroll functionality with Stimulus controller
- Optimize message scrolling in chat views
- Update message styling for better visual hierarchy
- Enhance chat container layout with flex and auto-scroll
- Simplify message rendering across different chat views
* Extract AI avatar to a shared partial for consistent styling
- Refactor AI avatar rendering across chat views
- Replace hardcoded avatar markup with a reusable partial
- Simplify avatar display in chats and messages views
* Update sidebar controller to handle right panel width dynamically
- Add conditional width class for right sidebar panel
- Ensure consistent sidebar toggle behavior for both left and right panels
- Use specific width class for right panel (w-[375px])
* Refactor chat form and AI greeting with flexible partials
- Extract message form to a reusable partial with dynamic context support
- Create flexible AI greeting partial for consistent welcome messages
- Simplify chat and sidebar views by leveraging new partials
- Add support for different form scenarios (chat, new chat, sidebar)
- Improve code modularity and reduce duplication
* Add chat clearing functionality with dynamic menu options
- Implement clear chat action in ChatsController
- Add clear chat route to support clearing messages
- Update AI sidebar with dropdown menu for chat actions
- Preserve system message when clearing chat
- Enhance chat interaction with new menu options
* Add frontmatter to project structure documentation
- Create initial frontmatter for structure.mdc file
- Include description and configuration options
- Prepare for potential dynamic documentation rendering
* Update general project rules with additional guidelines
- Add rule for using `Current.family` instead of `current_family`
- Include new guidelines for testing, API routes, and solution approach
- Expand project-specific rules for more consistent development practices
* Add OpenAI gem and AI-friendly data representations
- Add `ruby-openai` gem for AI integration
- Implement `to_ai_readable_hash` methods in BalanceSheet and IncomeStatement
- Include Promptable module in both models
- Add savings rate calculation method in IncomeStatement
- Prepare financial models for AI-powered insights and interactions
* Enhance AI Financial Assistant with Advanced Querying and Debugging Capabilities
- Implement comprehensive AI financial query system with function-based interactions
- Add detailed debug logging for AI responses and function calls
- Extend BalanceSheet and IncomeStatement models with AI-friendly methods
- Create robust error handling and fallback mechanisms for AI queries
- Update chat and message views to support debug mode and enhanced rendering
- Add AI query routes and initial test coverage for financial assistant
* Refactor AI sidebar and chat layout with improved structure and comments
- Remove inline AI chat from application layout
- Enhance AI sidebar with more semantic HTML structure
- Add descriptive comments to clarify different sections of chat view
- Improve flex layout and scrolling behavior in chat messages container
- Optimize message rendering with more explicit class names and structure
* Add Markdown rendering support for AI chat messages
- Implement `markdown` helper method in ApplicationHelper using Redcarpet
- Update message view to render AI messages with Markdown formatting
- Add comprehensive Markdown rendering options (tables, code blocks, links)
- Enhance AI Financial Assistant prompt to encourage Markdown usage
- Remove commented Markdown CSS in Tailwind application stylesheet
* Missing comma
* Enhance AI response processing with chat history context
* Improve AI debug logging with payload size limits and internal message flag
* Enhance AI chat interaction with improved thinking indicator and scrolling behavior
* Add AI consent and enable/disable functionality for AI chat
* Upgrade Biome and refactor JavaScript template literals
- Update @biomejs/biome to latest version with caret (^) notation
- Refactor AI query and chat controllers to use template literals
- Standardize npm scripts formatting in package.json
* Add beta testing usage note to AI consent modal
* Update test fixtures and configurations for AI chat functionality
- Add family association to chat fixtures and tests
- Set consistent password digest for test users
- Enable AI for test users
- Add OpenAI access token for test environment
- Update chat and user model tests to include family context
* Simplify data model and get tests passing
* Remove structure.mdc from version control
* Integrate AI chat styles into existing prose pattern
* Match Figma design spec, implement Turbo frames and actions for chats controller
* AI rules refresh
* Consolidate Stimulus controllers, thinking state, controllers, and views
* Naming, domain alignment
* Reset migrations
* Improve data model to support tool calls and message types
* Tool calling tests and fixtures
* Tool call implementation and test
* Get assistant test working again
* Test updates
* Process tool calls within provider
* Chat UI back to working state again
* Remove stale code
* Tests passing
* Update openai class naming to avoid conflicts
* Reconfigure test env
* Rebuild gemfile
* Fix naming conflicts for ChatResponse
* Message styles
* Use OpenAI conversation state management
* Assistant function base implementation
* Add back thinking messages, clean up error handling for chat
* Fix sync error when security price has bad data from provider
* Add balance sheet function to assistant
* Add better function calling error visibility
* Add income statement function
* Simplify and clean up "thinking" interactions with Turbo frames
* Remove stale data definitions from functions
* Ensure VCR fixtures working with latest code
* basic stream implementation
* Get streaming working
* Make AI sidebar wider when left sidebar is collapsed
* Get tests working with streaming responses
* Centralize provider error handling
* Provider data boundaries
---------
Co-authored-by: Josh Pigford <josh@joshpigford.com>
