* feat(ai): add Anthropic provider with chat parity (1/5)
Introduces Provider::Anthropic alongside Provider::Openai, implementing
the LlmConcept chat_response contract over the official anthropic Ruby
SDK. Batch ops, PDF, and RAG land in follow-up PRs.
- Provider::Anthropic uses Messages API for sync and streaming responses
- ChatConfig builds requests with ephemeral prompt-cache markers on the
system prompt and the last tool definition
- MessageFormatter reconstructs multi-turn history (text + tool_use +
tool_result blocks) from raw Message records, including the paired
user-role tool_result turn Anthropic requires after every tool_use
- ChatParser maps Anthropic Message into the shared ChatResponse Data
- Registry, Setting, User, Chat default model wired for ANTHROPIC_*
envs and Setting.anthropic_*; LLM_PROVIDER selects between providers
- Responder forwards raw conversation_history (Array<Message>) so
providers without hosted conversation state can rebuild context
- OpenAI provider accepts and ignores the new kwarg (no behavior change)
Tests cover provider init, model gating, MessageFormatter for all turn
shapes, ChatConfig request building (max_tokens, system cache, tool
conversion), ChatParser for text / tool_use / mixed blocks, Registry
discovery, and mocked chat_response success / error / function_request
paths. Live VCR cassettes recorded in a follow-up with a real key.
Stacked PRs: 2/5 batch ops + cost ledger, 3/5 PDF, 4/5 pgvector RAG,
5/5 settings UI + disclosure.
* fix(ai): address PR review on Anthropic provider foundation
Surface fixes raised by Codex + CodeRabbit on PR 1/5:
- Provider::Anthropic#chat_response now accepts (and ignores) a
`messages:` kwarg. Assistant::Responder passes both `messages:`
(OpenAI-shape) and `conversation_history:` (raw Message records) for
cross-provider parity, so the previous signature raised
ArgumentError on the first chat turn through the Anthropic provider.
- Provider::Anthropic#supports_model? bypasses the `claude` prefix
gate when a custom base_url is configured, mirroring the OpenAI
provider. Bedrock-shaped IDs like
`anthropic.claude-sonnet-4-5-20250929-v1:0` and
`claude-opus-4@20250514` are otherwise rejected by
Assistant::Provided#get_model_provider and the chat dies.
- Setting.anthropic_access_token is now in
EncryptedSettingFields::ENCRYPTED_FIELDS so the Anthropic API key
is encrypted at rest like every other provider secret. Previously
plaintext while siblings (openai_access_token, twelve_data_api_key,
external_assistant_token) were ciphertext.
- Chat.default_model falls back to whichever provider is actually
configured. Previously, with LLM_PROVIDER=anthropic but no
Anthropic credentials, the default model resolved to a Claude ID
that no registered provider supported, so chats failed even when
OpenAI was fully configured. Adds Provider::{Anthropic,Openai}#configured?
class methods for the readable callsite.
- Provider::Anthropic.effective_model uses
`ENV["ANTHROPIC_MODEL"].presence || Setting.anthropic_model` so the
Setting lookup is only performed when the env var is absent — the
previous `ENV.fetch(KEY, default)` evaluated the default arg
eagerly on every call.
- Provider::Anthropic::ChatConfig#anthropic_input_schema strips both
`:strict` and `"strict"` keys so JSON-decoded schemas with string
keys cannot leak the OpenAI-only flag through to Anthropic.
Test coverage added: supports_model? bypass on custom endpoints,
chat_response messages: kwarg compatibility, default_model fallback
in the three credential combinations, configured? against ENV +
Setting, strict-flag stripping for both key types, and a
`Setting.expects(:anthropic_model).never` assertion proving the
ENV-precedence test now exercises the lazy path.
All 4365 tests pass (1 pre-existing libvips env error unrelated).
* test(chat): make default_model tests resilient to ENV model overrides
CodeRabbit flagged on PR review: the new default_model tests asserted
against Provider::*::DEFAULT_MODEL, but Chat.default_model actually
returns Provider::*.effective_model.presence (which reads
OPENAI_MODEL / ANTHROPIC_MODEL from the environment). With either env
var set, the tests would fail intermittently even though routing was
correct.
- New default_model tests now assert against the provider's
effective_model directly, so they verify the routing decision
(which provider's value wins) without coupling to the constant.
- Pre-existing "creates with default model" assertions had the same
brittleness; switch them to compare against Chat.default_model so
the chosen model is whatever the env / Setting cascade resolves to.
Verified by running `ANTHROPIC_MODEL=claude-haiku-4-5 OPENAI_MODEL=gpt-4o
bin/rails test test/models/chat_test.rb` — 16 runs, 0 failures
(previously 2 pre-existing failures + 0 from the new tests).
* fix(ai): address local review on Anthropic foundation
- Provider::Anthropic#supports_pdf_processing? bypasses prefix gate for
custom endpoints, mirroring supports_model?
- Provider::Anthropic#initialize raises Error when custom_endpoint? AND
model.blank?, parity with Provider::Openai
- stream_chat_response captures partial usage on mid-stream errors and
records it via the new on_partial callback so chat_response can skip
the duplicate error row in the outer rescue
- safe_accumulated_message swallows the secondary failure when the SDK
cannot reconstruct a snapshot
- langfuse_client memoizes properly (||= instead of =) so repeated calls
don't churn Langfuse instances
- MessageFormatter sorts tool_calls by created_at then id so the
message array is deterministic across replays; skips tool_calls
missing both provider_call_id and provider_id rather than sending
`id: nil` and getting rejected by Anthropic
- Setting.anthropic_access_token default falls back through
ENV["ANTHROPIC_API_KEY"].presence (was missing .presence, so an
empty-string env value bled through)
- User#openai_configured? / #anthropic_configured? delegate to the
Provider::* class methods — single source of truth
- Assistant::Responder renames the OpenAI-shape history builder
conversation_history → openai_messages_payload so the kwarg name
matches the local method name (messages: openai_messages_payload,
conversation_history: chat_message_records)
- Assistant::Builtin stale-history comment updated to reference both
builders
Adds a streaming chat_response test using ad-hoc subclasses of the
SDK event types so the case/when dispatch matches via is_a? without
stubbing class-level === behavior.
* test(ai): add Anthropic tool_use round-trip + multi-tool turn coverage
Addresses @jjmata's "worth confirming" note on PR #1983: tool-use turns
from prior assistant messages must round-trip correctly when retrieved
from the database.
- New `ChatParser → ToolCall::Function → MessageFormatter` test walks
the full path: Anthropic response with a tool_use block →
ChatFunctionRequest → ToolCall::Function.from_function_request →
persisted on the AssistantMessage → MessageFormatter rebuild on the
next turn. Asserts the original `tool_use.id` is preserved end-to-end
as both `tool_use.id` and the paired `tool_result.tool_use_id`, and
that the original `input` hash and serialized result content survive.
- New multi-tool assistant turn test confirms two tool_use blocks on a
single assistant message render as two tool_use blocks followed by
two paired tool_result blocks in a single user-role follow-up,
matching Anthropic's required alternation.
Both tests exercise the existing PR1 code without behavior changes.
* test(ai): require "ostruct" explicitly in Anthropic provider tests
OpenStruct is moving out of Ruby's default load path (warning in 3.4+,
removed in 3.5+). Tests work today because ActiveSupport transitively
loads it, but that's incidental. Match the existing convention in
test/controllers/settings/hostings_controller_test.rb which explicitly
requires ostruct for the same reason.
* fix(ai): sanitize Langfuse warn logs, normalize tool_use.input, dedup history fetch
Addresses three open CodeRabbit findings on PR #1983.
- Provider::Anthropic Langfuse rescue branches no longer include
`e.full_message` in `Rails.logger.warn`. `full_message` bundles the
backtrace + cause chain and on some SDK error types includes the
serialized request/response payload (prompt, model output). Logs
now report `#{e.class}: #{e.message}` only. Three sites:
create_langfuse_trace, log_langfuse_generation, upsert_langfuse_trace.
Note: Provider::Openai has the same pattern (copy-pasted source) —
harmonization deferred to a follow-up cleanup PR; this commit fixes
only the Anthropic provider to keep PR scope tight.
- MessageFormatter#parse_arguments now coerces any non-Hash parsed
result to `{}`. Anthropic's Messages API requires `tool_use.input`
to be a JSON object (map); a stored ToolCall::Function record whose
arguments parse to a scalar, bool, or array (corrupt row, legacy
data, cross-provider bleed) would otherwise produce a payload the
API rejects. Normal flow stores Hash arguments end-to-end so the
fix is defensive — adds 2 tests covering scalar/array JSON strings
and non-String non-Hash inputs.
- Assistant::Responder dedups the chat-history fetch. The previous
layout fired two near-identical `chat.messages.where(...).includes(
:tool_calls).ordered` queries per LLM turn (one for the OpenAI-shape
payload, one for the raw-records kwarg). A new memoized
`complete_chat_messages` fetches once; `chat_message_records` filters
out the current message via `Array#reject`, `openai_messages_payload`
iterates the cached array unchanged. One SQL query per turn instead
of two. Memoization scope = single Responder instance (per LLM call),
so cache invalidation is not a concern.
All 4370 tests pass (1 pre-existing libvips env error unrelated).
Rubocop + brakeman clean.
* fix(ci): replace sk-ant- prefixed test placeholders
Pipelock secret scanner pattern-matches `sk-ant-*` as a real Anthropic
API key and fails the PR security-scan check. Test stubs and
ClimateControl env values used `sk-ant-test`, `sk-ant-from-setting`,
`sk-ant-x`, `sk-ant-y` as obvious placeholders, but the scanner does
not care about value entropy.
Switched to `fake-anthropic-key-*` / `fake-token-*` strings so the
scanner stops flagging them. No production code touched, no behavior
change — Provider::Anthropic still accepts any non-blank token.
The streaming code assumed every stream produced a `response.completed`
event and dereferenced its data unconditionally, causing
`undefined method 'data' for nil` whenever OpenAI emitted
`response.failed`, `response.incomplete`, or a top-level `error` event
(e.g. expired `previous_response_id`, context-window overflow,
transient upstream failures). Surface a descriptive `Provider::Error`
instead.
- Extend `ChatStreamParser` to recognise `response.failed`,
`response.incomplete`, and `error` events and emit an `error` chunk
with a `StreamErrorData` payload (event, message, code, details).
- In `Provider::Openai#native_chat_response`, detect the missing
`response` chunk, build a user-facing error message from the
collected error chunk, and raise `Provider::Error`.
- Add unit tests for the parser (8 cases) and integration tests for
the error path in the chat response flow.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Add AI merchant enhancement and dedup
* Enhancements
Add error if job is already running
add note that we also merge merchants
* Allow updating provider website
* Review fixes
* Update provider_merchant.rb
* Linter and fixes
* FIX transaction quick menu modal
* Initial plan
* Fix nil references in Recording failed LLM usage code paths
Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com>
* Replace error&.message with rescue-guarded safe_error_message helper
error&.message only guards against nil; it still raises when the error
object's .message implementation itself throws (e.g. OpenAI errors that
call data on nil). Replace with a safe_error_message helper that wraps
error&.message in a rescue block, returning a descriptive fallback
string on secondary failures. Apply the helper in both record_usage_error
(usage_recorder.rb) and record_llm_usage (openai.rb), including the
regex branch of extract_http_status_code in both files.
Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com>
Replace direct trace.update calls with client trace upserts so OpenAI provider is compatible with langfuse-ruby 0.1.6 behavior. Add richer warning logs that include full exception details for trace creation, trace upserts, and generation logging failures. Add tests for client-based trace upserts and detailed error logging.
* Add import row generation from PDF extracted data
- Add generate_rows_from_extracted_data method to PdfImport
- Add import! method to create transactions from PDF rows
- Update ProcessPdfJob to generate rows after extraction
- Update configured?, cleaned?, publishable? for PDF workflow
- Add column_keys, required_column_keys, mapping_steps
- Set bank statements to pending status for user review
- Add tests for new functionality
Closes#844
* Add tests for BankStatementExtractor
- Test transaction extraction from PDF content
- Test deduplication across chunk boundaries
- Test amount normalization for various formats
- Test graceful handling of malformed JSON responses
- Test error handling for empty/nil PDF content
* Fix supports_pdf_processing? to validate effective model
The validation was always checking @default_model, but process_pdf
allows overriding the model via parameter. This could cause a
vision-capable override model to be rejected, or a non-vision-capable
override to pass validation only to fail during processing.
Changes:
- supports_pdf_processing? now accepts optional model parameter
- process_pdf passes effective model to validation
- Raise Provider::Openai::Error inside with_provider_response for
consistent error handling
Addresses review feedback from PR#808
* Fix insert_all! bug: explicitly set import_id
Rails insert_all! on associations does NOT auto-set the foreign key.
Added import_id explicitly and use Import::Row.insert_all! directly.
Also reload rows before counting to ensure accurate count.
* Fix pending status showing as processing for bank statements with rows
When bank statement PDF imports have extracted rows, show a 'Ready for Review'
screen with a link to the confirm path instead of the 'Processing' spinner.
This addresses the PR feedback that users couldn't reach the review flow even
though rows were created.
* Gate publishable? on account.present? to prevent import failure
PDF imports are created without an account, and import! raises if account
is missing. This prevents users from hitting publish and having the job fail.
* Wrap generate_rows_from_extracted_data in transaction for atomicity
- Clear rows and reset count even when no transactions extracted
- Use transaction block to prevent partial updates on failure
- Use mapped_rows.size instead of reload for count
* Localize transactions count string with i18n helper
* Add AccountMapping step for PDF imports when account is nil
PDF imports need account selection before publishing. This adds
Import::AccountMapping to mapping_steps when account is nil,
matching the behavior of TransactionImport and TradeImport.
Addresses PR#846 feedback about account selection for PDF imports.
* Only include CategoryMapping when rows have non-empty categories
PDF extraction doesn't extract categories from bank statements,
so the CategoryMapping step would show empty. Now we only include
CategoryMapping if rows actually have non-empty category values.
This prevents showing an empty mapping step for PDF imports.
* Fix PDF import UI flow and account selection
- Add direct account selection in PDF import UI instead of AccountMapping
- AccountMapping designed for CSV imports with multiple account values
- PDF imports need single account for all transactions
- Add update action and route for imports controller
- Fix controller to handle pdf_import param format from form_with
- Show Publish button when import is publishable (account set)
- Fix stepper nav: Upload/Configure/Clean non-clickable for PDF imports
- Redirect PDF imports from configuration step (auto-configured)
- Improve AI prompt to recognize M-PESA/mobile money as bank statements
- Fix migration ordering for import_rows table columns
* Add guard for invalid account_id in imports#update
Prevents silently clearing account when invalid ID is passed.
Returns error message instead of confusing 'Account saved' notice.
* Localize step names in import nav and add account guard
- Use t() helper for all step names (Upload, Configure, Clean, Map, Confirm)
- Add guard for invalid account_id in imports#update
- Prevents silently clearing account when invalid ID is passed
* Make category column migrations idempotent
Check if columns exist before adding to prevent duplicate column
errors when migrations are re-run with new timestamps.
* Add match_path for PDF import step highlighting
Fixes step detection when path is nil by using separate match_path
for current step highlighting while keeping links disabled.
* Rename category migrations and update to Rails 7.2
- Rename class to EnsureCategoryFieldsOnImportRows to avoid conflicts
- Rename class to EnsureCategoryIconOnImportRows
- Update migration version from 7.1 to 7.2 per guidelines
- Rename files to match class names
- Add match_path for PDF import step highlighting
* Use primary (black) style for Create Account and Save buttons
* Remove match_path from auto-completed PDF steps
Only step 4 (Confirm) needs match_path for active-step detection.
Steps 1-3 are purely informational and always complete.
* Add fallback for document type translation
Handles nil or unexpected document_type values gracefully.
Also removes match_path from auto-completed PDF steps.
* Use index-based step number for mobile indicator
Fixes 'Step 5 of 4' issue when Map step is dynamically removed.
* Fix hostings_controller_test: use blank? instead of nil
Setting returns empty string not nil for unset values.
* Localize step progress label and use design token
* Fix button styling: use design system Tailwind classes
btn--primary and btn--secondary CSS classes don't exist.
Use actual design system classes from DS::Buttonish.
* Fix CRLF line endings in tags_controller_test.rb
---------
Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
* feat: Add PDF import with AI-powered document analysis
This enhances the import functionality to support PDF files with AI-powered
document analysis. When a PDF is uploaded, it is processed by AI to:
- Identify the document type (bank statement, credit card statement, etc.)
- Generate a summary of the document contents
- Extract key metadata (institution, dates, balances, transaction count)
After processing, an email is sent to the user asking for next steps.
Key changes:
- Add PdfImport model for handling PDF document imports
- Add Provider::Openai::PdfProcessor for AI document analysis
- Add ProcessPdfJob for async PDF processing
- Add PdfImportMailer for user notification emails
- Update imports controller to detect and handle PDF uploads
- Add PDF import option to the new import page
- Add i18n translations for all new strings
- Add comprehensive tests for the new functionality
* Add bank statement import with AI extraction
- Create ImportBankStatement assistant function for MCP
- Add BankStatementExtractor with chunked processing for small context windows
- Register function in assistant configurable
- Make PdfImport#pdf_file_content public for extractor access
- Increase OpenAI request timeout to 600s for slow local models
- Increase DB connection pool to 20 for concurrent operations
Tested with M-Pesa bank statement via remote Ollama (qwen3:8b):
- Successfully extracted 18 transactions
- Generated CSV and created TransactionImport
- Works with 3000 char chunks for small context windows
* Add pdf-reader gem dependency
The BankStatementExtractor uses PDF::Reader to parse bank statement
PDFs, but the gem was not properly declared in the Gemfile. This would
cause NameError in production when processing bank statements.
Added pdf-reader ~> 2.12 to Gemfile dependencies.
* Fix transaction deduplication to preserve legitimate duplicates
The previous deduplication logic removed ALL duplicate transactions based
on [date, amount, name], which would drop legitimate same-day duplicates
like multiple ATM withdrawals or card authorizations.
Changed to only deduplicate transactions that appear in consecutive chunks
(chunking artifacts) while preserving all legitimate duplicates within the
same chunk or non-adjacent chunks.
* Refactor bank statement extraction to use public provider method
Address code review feedback:
- Add public extract_bank_statement method to Provider::Openai
- Remove direct access to private client via send(:client)
- Update ImportBankStatement to use new public method
- Add require 'set' to BankStatementExtractor
- Remove PII-sensitive content from error logs
- Add defensive check for nil response.error
- Handle oversized PDF pages in chunking logic
- Remove unused process_native and process_generic methods
- Update email copy to reflect feature availability
- Add guard for nil document_type in email template
- Document pdf-reader gem rationale in Gemfile
Tested with both OpenAI (gpt-4o) and Ollama (qwen3:8b):
- OpenAI: 49 transactions extracted in 30s
- Ollama: 40 transactions extracted in 368s
- All encapsulation and error handling working correctly
* Update schema.rb with ai_summary and document_type columns
* Address PR #808 review comments
- Rename :csv_file to :import_file across controllers/views/tests
- Add PDF test fixture (sample_bank_statement.pdf)
- Add supports_pdf_processing? method for graceful degradation
- Revert unrelated database.yml pool change (600->3)
- Remove month_start_day schema bleed from other PR
- Fix PdfProcessor: use .strip instead of .strip_heredoc
- Add server-side PDF magic byte validation
- Conditionally show PDF import option when AI provider available
- Fix ProcessPdfJob: sanitize errors, handle update failure
- Move pdf_file attachment from Import to PdfImport
- Document deduplication logic limitations
- Fix ImportBankStatement: catch specific exceptions only
- Remove unnecessary require 'set'
- Remove dead json_schema method from PdfProcessor
- Reduce default OpenAI timeout from 600s to 60s
- Fix nil guard in text mailer template
- Add require 'csv' to ImportBankStatement
- Remove Gemfile pdf-reader comment
* Fix RuboCop indentation in ProcessPdfJob
* Refactor PDF import check to use model predicate method
Replace is_a?(PdfImport) type check with requires_csv_workflow? predicate
that leverages STI inheritance for cleaner controller logic.
* Fix missing 'unknown' locale key and schema version mismatch
- Add 'unknown: Unknown Document' to document_types locale
- Fix schema version to match latest migration (2026_01_24_180211)
* Document OPENAI_REQUEST_TIMEOUT env variable
Added to .env.local.example and docs/hosting/ai.md
* Rename ALLOWED_MIME_TYPES to ALLOWED_CSV_MIME_TYPES for clarity
* Add comment explaining requires_csv_workflow? predicate
* Remove redundant required_column_keys from PdfImport
Base class already returns [] by default
* Add ENV toggle to disable PDF processing for non-vision endpoints
OPENAI_SUPPORTS_PDF_PROCESSING=false can be used for OpenAI-compatible
endpoints (e.g., Ollama) that don't support vision/PDF processing.
* Wire up transaction extraction for PDF bank statements
- Add extracted_data JSONB column to imports
- Add extract_transactions method to PdfImport
- Call extraction in ProcessPdfJob for bank statements
- Store transactions in extracted_data for later review
* Fix ProcessPdfJob retry logic, sanitize and localize errors
- Allow retries after partial success (classification ok, extraction failed)
- Log sanitized error message instead of raw message to avoid data leakage
- Use i18n for user-facing error messages
* Add vision-capable model validation for PDF processing
* Fix drag-and-drop test to use correct field name csv_file
* Schema bleedover from another branch
* Fix drag-drop import form field name to match controller
* Add vision capability guard to process_pdf method
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
* Initial implementation
* FIX keys
* Add langfuse evals support
* FIX trace upload
* Delete .claude/settings.local.json
Signed-off-by: soky srm <sokysrm@gmail.com>
* Update client.rb
* Small LLMs improvements
* Keep batch size normal
* Update categorizer
* FIX json mode
* Add reasonable alternative to matching
* FIX thinking blocks for llms
* Implement json mode support with AUTO mode
* Make auto default for everyone
* FIX linter
* Address review
* Allow export manual categories
* FIX user export
* FIX oneshot example pollution
* Update categorization_golden_v1.yml
* Update categorization_golden_v1.yml
* Trim to 100 items
* Update auto_categorizer.rb
* FIX for auto retry in auto mode
* Separate the Eval Logic from the Auto-Categorizer
The expected_null_count parameter conflates eval-specific logic with production categorization logic.
* Force json mode on evals
* Introduce a more mixed dataset
150 items, performance from a local model:
By Difficulty:
easy: 93.22% accuracy (55/59)
medium: 93.33% accuracy (42/45)
hard: 92.86% accuracy (26/28)
edge_case: 100.0% accuracy (18/18)
* Improve datasets
Remove Data leakage from prompts
* Create eval runs as "pending"
---------
Signed-off-by: soky srm <sokysrm@gmail.com>
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Track failed LLM API calls in llm_usages table
This commit adds comprehensive error tracking for failed LLM API calls:
- Updated LlmUsage model with helper methods to identify failed calls
and retrieve error details (failed?, http_status_code, error_message)
- Modified Provider::Openai to record failed API calls with error metadata
including HTTP status codes and error messages in both native and
generic chat response methods
- Enhanced UsageRecorder concern with record_usage_error method to support
error tracking for auto-categorization and auto-merchant detection
- Updated LLM usage UI to display failed calls with:
- Red background highlighting for failed rows
- Error indicator icon with "Failed" label
- Interactive tooltip on hover showing error message and HTTP status code
Failed calls are now tracked with zero tokens and null cost, storing
error details in the metadata JSONB column for visibility and debugging.
* Dark mode fixes
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Add comprehensive AI/LLM configuration documentation
* Fix Chat.start! to use default model when model is nil or empty
* Ensure all controllers use Chat.default_model for consistency
* Move AI doc inside `hosting/`
* Probably too much error handling
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Password reset back button also after confirmation
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
* Implement a filter for category (#215)
- Also implement an is empty/is null condition.
* Implement an LLM cost estimation page
Track costs across all the cost categories: auto categorization, auto merchant detection and chat.
Show warning with estimated cost when running a rule that contains AI.
* Update pricing
* Add google pricing
and fix inferred model everywhere.
* Update app/models/llm_usage.rb
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: soky srm <sokysrm@gmail.com>
* FIX address review
* Linter
* Address review
- Lowered log level
- extracted the duplicated record_usage method into a shared concern
* Update app/controllers/settings/llm_usages_controller.rb
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: soky srm <sokysrm@gmail.com>
* Moved attr_reader out of private
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Implement support for generic OpenAI api
- Implements support to route requests to any openAI capable provider ( Deepsek, Qwen, VLLM, LM Studio, Ollama ).
- Keeps support for pure OpenAI and uses the new better responses api
- Uses the /chat/completions api for the generic providers
- If uri_base is not set, uses default implementation.
* Fix json handling and indentation
* Fix linter error indent
* Fix tests to set env vars
* Fix updating settings
* Change to prefix checking for OAI models
* FIX check model if custom uri is set
* Change chat to sync calls
Some local models don't support streaming. Revert to sync calls for generic OAI api
* Fix tests
* Fix tests
* Fix for gpt5 message extraction
- Finds the message output by filtering for "type" == "message" instead of assuming it's at index 0
- Safely extracts the text using safe navigation operators (&.)
- Raises a clear error if no message content is found
- Parses the JSON as before
* Add more langfuse logging
- Add Langfuse to auto categorizer and merchant detector
- Fix monitoring on streaming chat responses
- Add Langfuse traces also for model errors now
* Update app/models/provider/openai.rb
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: soky srm <sokysrm@gmail.com>
* handle nil function results explicitly
* Exposing some config vars.
* Linter and nitpick comments
* Drop back to `gpt-4.1` as default for now
* Linter
* Fix for strict tool schema in Gemini
- This fixes tool calling in Gemini OpenAI api
- Fix for getTransactions function, page size is not used.
---------
Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Domain model sketch
* Scaffold out rules domain
* Migrations
* Remove existing data enrichment for clean slate
* Sketch out business logic and basic tests
* Simplify rule scope building and action executions
* Get generator working again
* Basic implementation + tests
* Remove manual merchant management (rules will replace)
* Revert "Remove manual merchant management (rules will replace)"
This reverts commit 83dcbd9ff0aa7bbee211796b71aa48b71df5e57e.
* Family and Provider merchants model
* Fix brakeman warnings
* Fix notification loader
* Update notification position
* Add Rule action and condition registries
* Rule form with compound conditions and tests
* Split out notification types, add CTA type
* Rules form builder and Stimulus controller
* Clean up rule registry domain
* Clean up rules stimulus controller
* CTA message for rule when user changes transaction category
* Fix tests
* Lint updates
* Centralize notifications in Notifiable concern
* Implement category rule prompts with auto backoff and option to disable
* Fix layout bug caused by merge conflict
* Initialize rule with correct action for category CTA
* Add rule deletions, get rules working
* Complete dynamic rule form, split Stimulus controllers by resource
* Fix failing tests
* Change test password to avoid chromium conflicts
* Update integration tests
* Centralize all test password references
* Add re-apply rule action
* Rule confirm modal
* Run migrations
* Trigger rule notification after inline category updates
* Clean up rule styles
* Basic attribute locking for rules
* Apply attribute locks on user edits
* Log data enrichments, only apply rules to unlocked attributes
* Fix merge errors
* Additional merge conflict fixes
* Form UI improvements, ignore attribute locks on manual rule application
* Batch AI auto-categorization of transactions
* Auto merchant detection, ai enrichment in batches
* Fix Plaid merchant assignments
* Plaid category matching
* Cleanup 1
* Test cleanup
* Remove stale route
* Fix desktop chat UI issues
* Fix mobile nav styling issues
* AI sidebar
* Add chat and message models with associations
* Implement AI chat functionality with sidebar and messaging system
- Add chat and messages controllers
- Create chat and message views
- Implement chat-related routes
- Add message broadcasting and user interactions
- Update application layout to support chat sidebar
- Enhance user model with initials method
* Refactor AI sidebar with enhanced chat menu and interactions
- Update sidebar layout with dynamic width and improved responsiveness
- Add new chat menu Stimulus controller for toggling between chat and chat list views
- Improve chat list display with recent chats and empty state
- Extract AI avatar to a partial for reusability
- Enhance message display and interaction styling
- Add more contextual buttons and interaction hints
* Improve chat scroll behavior and message styling
- Refactor chat scroll functionality with Stimulus controller
- Optimize message scrolling in chat views
- Update message styling for better visual hierarchy
- Enhance chat container layout with flex and auto-scroll
- Simplify message rendering across different chat views
* Extract AI avatar to a shared partial for consistent styling
- Refactor AI avatar rendering across chat views
- Replace hardcoded avatar markup with a reusable partial
- Simplify avatar display in chats and messages views
* Update sidebar controller to handle right panel width dynamically
- Add conditional width class for right sidebar panel
- Ensure consistent sidebar toggle behavior for both left and right panels
- Use specific width class for right panel (w-[375px])
* Refactor chat form and AI greeting with flexible partials
- Extract message form to a reusable partial with dynamic context support
- Create flexible AI greeting partial for consistent welcome messages
- Simplify chat and sidebar views by leveraging new partials
- Add support for different form scenarios (chat, new chat, sidebar)
- Improve code modularity and reduce duplication
* Add chat clearing functionality with dynamic menu options
- Implement clear chat action in ChatsController
- Add clear chat route to support clearing messages
- Update AI sidebar with dropdown menu for chat actions
- Preserve system message when clearing chat
- Enhance chat interaction with new menu options
* Add frontmatter to project structure documentation
- Create initial frontmatter for structure.mdc file
- Include description and configuration options
- Prepare for potential dynamic documentation rendering
* Update general project rules with additional guidelines
- Add rule for using `Current.family` instead of `current_family`
- Include new guidelines for testing, API routes, and solution approach
- Expand project-specific rules for more consistent development practices
* Add OpenAI gem and AI-friendly data representations
- Add `ruby-openai` gem for AI integration
- Implement `to_ai_readable_hash` methods in BalanceSheet and IncomeStatement
- Include Promptable module in both models
- Add savings rate calculation method in IncomeStatement
- Prepare financial models for AI-powered insights and interactions
* Enhance AI Financial Assistant with Advanced Querying and Debugging Capabilities
- Implement comprehensive AI financial query system with function-based interactions
- Add detailed debug logging for AI responses and function calls
- Extend BalanceSheet and IncomeStatement models with AI-friendly methods
- Create robust error handling and fallback mechanisms for AI queries
- Update chat and message views to support debug mode and enhanced rendering
- Add AI query routes and initial test coverage for financial assistant
* Refactor AI sidebar and chat layout with improved structure and comments
- Remove inline AI chat from application layout
- Enhance AI sidebar with more semantic HTML structure
- Add descriptive comments to clarify different sections of chat view
- Improve flex layout and scrolling behavior in chat messages container
- Optimize message rendering with more explicit class names and structure
* Add Markdown rendering support for AI chat messages
- Implement `markdown` helper method in ApplicationHelper using Redcarpet
- Update message view to render AI messages with Markdown formatting
- Add comprehensive Markdown rendering options (tables, code blocks, links)
- Enhance AI Financial Assistant prompt to encourage Markdown usage
- Remove commented Markdown CSS in Tailwind application stylesheet
* Missing comma
* Enhance AI response processing with chat history context
* Improve AI debug logging with payload size limits and internal message flag
* Enhance AI chat interaction with improved thinking indicator and scrolling behavior
* Add AI consent and enable/disable functionality for AI chat
* Upgrade Biome and refactor JavaScript template literals
- Update @biomejs/biome to latest version with caret (^) notation
- Refactor AI query and chat controllers to use template literals
- Standardize npm scripts formatting in package.json
* Add beta testing usage note to AI consent modal
* Update test fixtures and configurations for AI chat functionality
- Add family association to chat fixtures and tests
- Set consistent password digest for test users
- Enable AI for test users
- Add OpenAI access token for test environment
- Update chat and user model tests to include family context
* Simplify data model and get tests passing
* Remove structure.mdc from version control
* Integrate AI chat styles into existing prose pattern
* Match Figma design spec, implement Turbo frames and actions for chats controller
* AI rules refresh
* Consolidate Stimulus controllers, thinking state, controllers, and views
* Naming, domain alignment
* Reset migrations
* Improve data model to support tool calls and message types
* Tool calling tests and fixtures
* Tool call implementation and test
* Get assistant test working again
* Test updates
* Process tool calls within provider
* Chat UI back to working state again
* Remove stale code
* Tests passing
* Update openai class naming to avoid conflicts
* Reconfigure test env
* Rebuild gemfile
* Fix naming conflicts for ChatResponse
* Message styles
* Use OpenAI conversation state management
* Assistant function base implementation
* Add back thinking messages, clean up error handling for chat
* Fix sync error when security price has bad data from provider
* Add balance sheet function to assistant
* Add better function calling error visibility
* Add income statement function
* Simplify and clean up "thinking" interactions with Turbo frames
* Remove stale data definitions from functions
* Ensure VCR fixtures working with latest code
* basic stream implementation
* Get streaming working
* Make AI sidebar wider when left sidebar is collapsed
* Get tests working with streaming responses
* Centralize provider error handling
* Provider data boundaries
---------
Co-authored-by: Josh Pigford <josh@joshpigford.com>
