mirror of
https://github.com/we-promise/sure.git
synced 2026-06-06 11:19:02 +00:00
76d4c2a2feebd6ab4658517dcbd3bc4bc1722f20
51 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Guillem Arias Fauste
|
8ccc434b3d |
feat(ai): Anthropic native PDF processing (3/5) (#1985)
* feat(ai): add Anthropic provider with chat parity (1/5)
Introduces Provider::Anthropic alongside Provider::Openai, implementing
the LlmConcept chat_response contract over the official anthropic Ruby
SDK. Batch ops, PDF, and RAG land in follow-up PRs.
- Provider::Anthropic uses Messages API for sync and streaming responses
- ChatConfig builds requests with ephemeral prompt-cache markers on the
system prompt and the last tool definition
- MessageFormatter reconstructs multi-turn history (text + tool_use +
tool_result blocks) from raw Message records, including the paired
user-role tool_result turn Anthropic requires after every tool_use
- ChatParser maps Anthropic Message into the shared ChatResponse Data
- Registry, Setting, User, Chat default model wired for ANTHROPIC_*
envs and Setting.anthropic_*; LLM_PROVIDER selects between providers
- Responder forwards raw conversation_history (Array<Message>) so
providers without hosted conversation state can rebuild context
- OpenAI provider accepts and ignores the new kwarg (no behavior change)
Tests cover provider init, model gating, MessageFormatter for all turn
shapes, ChatConfig request building (max_tokens, system cache, tool
conversion), ChatParser for text / tool_use / mixed blocks, Registry
discovery, and mocked chat_response success / error / function_request
paths. Live VCR cassettes recorded in a follow-up with a real key.
Stacked PRs: 2/5 batch ops + cost ledger, 3/5 PDF, 4/5 pgvector RAG,
5/5 settings UI + disclosure.
* fix(ai): address PR review on Anthropic provider foundation
Surface fixes raised by Codex + CodeRabbit on PR 1/5:
- Provider::Anthropic#chat_response now accepts (and ignores) a
`messages:` kwarg. Assistant::Responder passes both `messages:`
(OpenAI-shape) and `conversation_history:` (raw Message records) for
cross-provider parity, so the previous signature raised
ArgumentError on the first chat turn through the Anthropic provider.
- Provider::Anthropic#supports_model? bypasses the `claude` prefix
gate when a custom base_url is configured, mirroring the OpenAI
provider. Bedrock-shaped IDs like
`anthropic.claude-sonnet-4-5-20250929-v1:0` and
`claude-opus-4@20250514` are otherwise rejected by
Assistant::Provided#get_model_provider and the chat dies.
- Setting.anthropic_access_token is now in
EncryptedSettingFields::ENCRYPTED_FIELDS so the Anthropic API key
is encrypted at rest like every other provider secret. Previously
plaintext while siblings (openai_access_token, twelve_data_api_key,
external_assistant_token) were ciphertext.
- Chat.default_model falls back to whichever provider is actually
configured. Previously, with LLM_PROVIDER=anthropic but no
Anthropic credentials, the default model resolved to a Claude ID
that no registered provider supported, so chats failed even when
OpenAI was fully configured. Adds Provider::{Anthropic,Openai}#configured?
class methods for the readable callsite.
- Provider::Anthropic.effective_model uses
`ENV["ANTHROPIC_MODEL"].presence || Setting.anthropic_model` so the
Setting lookup is only performed when the env var is absent — the
previous `ENV.fetch(KEY, default)` evaluated the default arg
eagerly on every call.
- Provider::Anthropic::ChatConfig#anthropic_input_schema strips both
`:strict` and `"strict"` keys so JSON-decoded schemas with string
keys cannot leak the OpenAI-only flag through to Anthropic.
Test coverage added: supports_model? bypass on custom endpoints,
chat_response messages: kwarg compatibility, default_model fallback
in the three credential combinations, configured? against ENV +
Setting, strict-flag stripping for both key types, and a
`Setting.expects(:anthropic_model).never` assertion proving the
ENV-precedence test now exercises the lazy path.
All 4365 tests pass (1 pre-existing libvips env error unrelated).
* test(chat): make default_model tests resilient to ENV model overrides
CodeRabbit flagged on PR review: the new default_model tests asserted
against Provider::*::DEFAULT_MODEL, but Chat.default_model actually
returns Provider::*.effective_model.presence (which reads
OPENAI_MODEL / ANTHROPIC_MODEL from the environment). With either env
var set, the tests would fail intermittently even though routing was
correct.
- New default_model tests now assert against the provider's
effective_model directly, so they verify the routing decision
(which provider's value wins) without coupling to the constant.
- Pre-existing "creates with default model" assertions had the same
brittleness; switch them to compare against Chat.default_model so
the chosen model is whatever the env / Setting cascade resolves to.
Verified by running `ANTHROPIC_MODEL=claude-haiku-4-5 OPENAI_MODEL=gpt-4o
bin/rails test test/models/chat_test.rb` — 16 runs, 0 failures
(previously 2 pre-existing failures + 0 from the new tests).
* fix(ai): address local review on Anthropic foundation
- Provider::Anthropic#supports_pdf_processing? bypasses prefix gate for
custom endpoints, mirroring supports_model?
- Provider::Anthropic#initialize raises Error when custom_endpoint? AND
model.blank?, parity with Provider::Openai
- stream_chat_response captures partial usage on mid-stream errors and
records it via the new on_partial callback so chat_response can skip
the duplicate error row in the outer rescue
- safe_accumulated_message swallows the secondary failure when the SDK
cannot reconstruct a snapshot
- langfuse_client memoizes properly (||= instead of =) so repeated calls
don't churn Langfuse instances
- MessageFormatter sorts tool_calls by created_at then id so the
message array is deterministic across replays; skips tool_calls
missing both provider_call_id and provider_id rather than sending
`id: nil` and getting rejected by Anthropic
- Setting.anthropic_access_token default falls back through
ENV["ANTHROPIC_API_KEY"].presence (was missing .presence, so an
empty-string env value bled through)
- User#openai_configured? / #anthropic_configured? delegate to the
Provider::* class methods — single source of truth
- Assistant::Responder renames the OpenAI-shape history builder
conversation_history → openai_messages_payload so the kwarg name
matches the local method name (messages: openai_messages_payload,
conversation_history: chat_message_records)
- Assistant::Builtin stale-history comment updated to reference both
builders
Adds a streaming chat_response test using ad-hoc subclasses of the
SDK event types so the case/when dispatch matches via is_a? without
stubbing class-level === behavior.
* test(ai): add Anthropic tool_use round-trip + multi-tool turn coverage
Addresses @jjmata's "worth confirming" note on PR #1983: tool-use turns
from prior assistant messages must round-trip correctly when retrieved
from the database.
- New `ChatParser → ToolCall::Function → MessageFormatter` test walks
the full path: Anthropic response with a tool_use block →
ChatFunctionRequest → ToolCall::Function.from_function_request →
persisted on the AssistantMessage → MessageFormatter rebuild on the
next turn. Asserts the original `tool_use.id` is preserved end-to-end
as both `tool_use.id` and the paired `tool_result.tool_use_id`, and
that the original `input` hash and serialized result content survive.
- New multi-tool assistant turn test confirms two tool_use blocks on a
single assistant message render as two tool_use blocks followed by
two paired tool_result blocks in a single user-role follow-up,
matching Anthropic's required alternation.
Both tests exercise the existing PR1 code without behavior changes.
* test(ai): require "ostruct" explicitly in Anthropic provider tests
OpenStruct is moving out of Ruby's default load path (warning in 3.4+,
removed in 3.5+). Tests work today because ActiveSupport transitively
loads it, but that's incidental. Match the existing convention in
test/controllers/settings/hostings_controller_test.rb which explicitly
requires ostruct for the same reason.
* fix(ai): sanitize Langfuse warn logs, normalize tool_use.input, dedup history fetch
Addresses three open CodeRabbit findings on PR #1983.
- Provider::Anthropic Langfuse rescue branches no longer include
`e.full_message` in `Rails.logger.warn`. `full_message` bundles the
backtrace + cause chain and on some SDK error types includes the
serialized request/response payload (prompt, model output). Logs
now report `#{e.class}: #{e.message}` only. Three sites:
create_langfuse_trace, log_langfuse_generation, upsert_langfuse_trace.
Note: Provider::Openai has the same pattern (copy-pasted source) —
harmonization deferred to a follow-up cleanup PR; this commit fixes
only the Anthropic provider to keep PR scope tight.
- MessageFormatter#parse_arguments now coerces any non-Hash parsed
result to `{}`. Anthropic's Messages API requires `tool_use.input`
to be a JSON object (map); a stored ToolCall::Function record whose
arguments parse to a scalar, bool, or array (corrupt row, legacy
data, cross-provider bleed) would otherwise produce a payload the
API rejects. Normal flow stores Hash arguments end-to-end so the
fix is defensive — adds 2 tests covering scalar/array JSON strings
and non-String non-Hash inputs.
- Assistant::Responder dedups the chat-history fetch. The previous
layout fired two near-identical `chat.messages.where(...).includes(
:tool_calls).ordered` queries per LLM turn (one for the OpenAI-shape
payload, one for the raw-records kwarg). A new memoized
`complete_chat_messages` fetches once; `chat_message_records` filters
out the current message via `Array#reject`, `openai_messages_payload`
iterates the cached array unchanged. One SQL query per turn instead
of two. Memoization scope = single Responder instance (per LLM call),
so cache invalidation is not a concern.
All 4370 tests pass (1 pre-existing libvips env error unrelated).
Rubocop + brakeman clean.
* fix(ci): replace sk-ant- prefixed test placeholders
Pipelock secret scanner pattern-matches `sk-ant-*` as a real Anthropic
API key and fails the PR security-scan check. Test stubs and
ClimateControl env values used `sk-ant-test`, `sk-ant-from-setting`,
`sk-ant-x`, `sk-ant-y` as obvious placeholders, but the scanner does
not care about value entropy.
Switched to `fake-anthropic-key-*` / `fake-token-*` strings so the
scanner stops flagging them. No production code touched, no behavior
change — Provider::Anthropic still accepts any non-blank token.
* feat(ai): add Anthropic batch ops + LLM cost ledger (2/5)
Implements auto_categorize, auto_detect_merchants, and
enhance_provider_merchants on Provider::Anthropic via forced tool calls,
plus the cost-ledger plumbing they need.
- Provider::Anthropic::AutoCategorizer, AutoMerchantDetector,
ProviderMerchantEnhancer each define a single output tool whose
input_schema mirrors the desired output, then force the model to call
it via tool_choice: { type: "tool", name: ..., disable_parallel_tool_use: true }.
Anthropic guarantees the tool_use.input matches the schema, so there
is no JSON parsing fragility, no <think> tag stripping, and no
json_object/json_schema fallback ladders.
- Concerns::UsageRecorder mirrors the OpenAI sibling but persists
cache_creation_input_tokens / cache_read_input_tokens to dedicated
columns instead of metadata.
- Migration adds cache_creation_tokens, cache_read_tokens (nullable
integers) to llm_usages. OpenAI rows leave them null.
- LlmUsage::PRICING gains Claude 4.x rows (opus-4-7 $15/$75, sonnet-4-6
$3/$15, haiku-4-5 $1/$5 per MTok). infer_provider returns "anthropic"
for claude-* via the existing exact/prefix lookup.
- Provider::Anthropic#chat_response now persists cache columns directly
rather than stashing them in metadata.
- 25-transaction batch cap mirrors the OpenAI provider so the cost
ledger sees the same shape regardless of which provider ran a batch.
Tests cover the forced-tool-call path, null/None normalization,
case-insensitive merchant matching, the missing-tool_use error path,
and Anthropic-specific pricing + provider inference on LlmUsage.
Stacked on #1983 (PR 1/5). 3/5 PDF + vision next.
* fix(ai): attribute Bedrock model IDs to anthropic + clean nil enum
- LlmUsage.infer_provider now returns "anthropic" for Bedrock /
Vertex shaped IDs (anthropic.* and anthropic/*), so cost-ledger
filtering by provider stays correct even when no per-MTok rate is
stored. Previously these IDs fell through to the "openai" default.
- AutoCategorizer drops the redundant nil sentinel from the
category_name enum — the union type [string, null] already permits
null, and some JSON Schema validators reject nil literals inside
enum arrays.
* test(ai): require "ostruct" in Anthropic batch op tests
Same rationale as the PR1 ostruct fix — explicit require so the tests
don't depend on ActiveSupport's transitive load when Ruby 3.5+ removes
OpenStruct from the default load path.
* feat(ai): Anthropic native PDF processing (3/5)
Implements process_pdf and extract_bank_statement on Provider::Anthropic
using the native `document` content block — no rasterization, no text
pre-extraction.
- Provider::Anthropic::PdfProcessor classifies the document, summarizes
it, and extracts statement metadata via a forced report_document_analysis
tool whose input_schema mirrors the existing Provider::Openai output
(document_type from Import::DOCUMENT_TYPES, summary, extracted_data).
- Provider::Anthropic::BankStatementExtractor returns the same
{ transactions, period, account_holder, account_number, bank_name,
opening_balance, closing_balance } shape via report_bank_statement so
downstream pdf_import code is provider-agnostic.
- Both attach the PDF as
{ type: "document", source: { type: "base64", media_type: "application/pdf", data: <b64> } }
— Claude 3.5+ / 4.x accept this natively (up to 32MB / 100 pages).
No pdf-reader, no pdftoppm, no chunking for typical statements.
- supports_pdf_processing? (introduced in PR 1) already returns true for
claude-* models, gating process_pdf with a clear error otherwise.
- Cost ledger rows are persisted via the shared UsageRecorder concern,
including cache_creation/cache_read tokens.
Tests verify the document block shape, tool_choice forcing, normalized
document_type for unknown classifications, transaction normalization
(date / amount / reference → notes), and the missing-tool_use error
path. Blank pdf_content raises before any client call.
Stacked on #1984 (PR 2/5). 4/5 pgvector RAG next.
* fix(ai): guard PDF size + surface bank-statement truncation
- PdfProcessor and BankStatementExtractor raise upfront when
pdf_content.bytesize exceeds MAX_PDF_BYTES (32 MB, matching
Anthropic's hard limit). Previously a 100 MB PDF would be
base64-encoded (~133 MB) and packed into the JSON body before
the API rejected it — peak heap ~270 MB per Sidekiq worker.
- BankStatementExtractor inspects response.stop_reason; when the
model hit max_tokens it logs a warning and flags result[:truncated]
so downstream callers know the transaction list may be incomplete.
- ISO date pattern added to statement_period_start/end schema in
PdfProcessor so the model can't return "March 2026" — Anthropic
enforces the regex via the tool's input_schema.
Tests cover the size guard (raises before any client.messages call),
truncated-result flagging, and the warning log path.
* test(ai): require "ostruct" in Anthropic PDF tests
Match the explicit ostruct require added in PR1/PR2 — same Ruby 3.5+
load-path reason.
* fix(llm-usage): include Anthropic cache tokens in estimated_cost
calculate_cost only priced prompt + completion tokens, so estimated_cost
under-reported every cached call — the cache_creation/cache_read columns this PR
added were tracked but never billed. Verified against the Anthropic dashboard: a
cached chat turn billed $0.05 but the ledger recorded $0.038; the gap was exactly
the unpriced cache tokens.
Price them relative to the input rate (Anthropic: cache write 1.25x, read 0.1x)
and thread the cache counts from both recorders (chat + batch). OpenAI rows leave
the columns null (treated as 0), so they're unaffected. Ledger now reproduces the
dashboard ($0.054 for the test turn).
* chore(ai): guard chat usage double-record; flag deferred Anthropic batch wiring
- Hardening: guard the success-path record_llm_usage with
`unless partial_usage_recorded` so a future change that emits partial usage on
a normal stream can't silently double-bill (the symptom investigated in the
#1984 review). No behavior change today — on_partial only fires from the
mid-stream-error rescue, which re-raises past this line.
- Notice: the family auto-categorize / merchant-detect / merchant-enhance flows
still hardcode get_provider(:openai). Provider::Anthropic now implements those
batch ops but they aren't wired into the family flows yet — documented with
TODOs at each site for the follow-up.
* chore(ai): point family-flow TODOs at tracking issue #2113
* chore(ai): flag deferred Anthropic PDF wiring (TODO #2113)
The PDF import + bank-statement-extract flows hardcode get_provider(:openai).
Provider::Anthropic implements process_pdf / extract_bank_statement (this PR)
but they aren't wired into these paths yet — documented with TODOs at each
site. Tracked in #2113 (broadened to cover batch ops + PDF).
* chore(anthropic-pdf): drop redundant strip_heredoc; document no-dedup
- The squiggly heredoc (<<~) already strips indentation, so the trailing
.strip_heredoc was a no-op in both PDF extractors.
- Document why BankStatementExtractor intentionally does NOT deduplicate (unlike
the OpenAI extractor): we send the whole PDF as one native document block, so
there are no overlapping-chunk artifacts to dedupe, and deduping would wrongly
merge legitimate same-day, same-amount transactions.
* fix(anthropic): cap PDF bytes below the base64-encoded request limit
Anthropic's 32 MB limit is on the Messages request body, and the PDF is sent
base64-encoded (~4/3 larger) alongside the JSON envelope, so a 32 MB raw PDF
encodes to ~42 MB and is rejected. Cap the raw bytes at 3/4 of the request
budget minus a 1 MB envelope reserve (~23 MiB). Addresses Codex review on #1985.
|
||
|
Brad
|
1b8b21760b |
feat(provider): Akahu integration (#1921)
* First pass of Akahu * fix up sync all * conflicts * fix db migration issue? - fix auto selection of akahu account type * Address Akahu PR feedback * Complete provider metadata * Fix PR 1921 CI tests * PR feedback * PR feedback * post merge --------- Co-authored-by: failing <failing@users.noreply.github.com> Co-authored-by: Juan José Mata <jjmata@jjmata.com> Co-authored-by: sure-admin <sure-admin@splashblot.com> |
||
|
Guillem Arias Fauste
|
991cb959c1 |
feat(ai): Anthropic batch ops + LLM cost ledger (2/5) (#1984)
* feat(ai): add Anthropic provider with chat parity (1/5)
Introduces Provider::Anthropic alongside Provider::Openai, implementing
the LlmConcept chat_response contract over the official anthropic Ruby
SDK. Batch ops, PDF, and RAG land in follow-up PRs.
- Provider::Anthropic uses Messages API for sync and streaming responses
- ChatConfig builds requests with ephemeral prompt-cache markers on the
system prompt and the last tool definition
- MessageFormatter reconstructs multi-turn history (text + tool_use +
tool_result blocks) from raw Message records, including the paired
user-role tool_result turn Anthropic requires after every tool_use
- ChatParser maps Anthropic Message into the shared ChatResponse Data
- Registry, Setting, User, Chat default model wired for ANTHROPIC_*
envs and Setting.anthropic_*; LLM_PROVIDER selects between providers
- Responder forwards raw conversation_history (Array<Message>) so
providers without hosted conversation state can rebuild context
- OpenAI provider accepts and ignores the new kwarg (no behavior change)
Tests cover provider init, model gating, MessageFormatter for all turn
shapes, ChatConfig request building (max_tokens, system cache, tool
conversion), ChatParser for text / tool_use / mixed blocks, Registry
discovery, and mocked chat_response success / error / function_request
paths. Live VCR cassettes recorded in a follow-up with a real key.
Stacked PRs: 2/5 batch ops + cost ledger, 3/5 PDF, 4/5 pgvector RAG,
5/5 settings UI + disclosure.
* fix(ai): address PR review on Anthropic provider foundation
Surface fixes raised by Codex + CodeRabbit on PR 1/5:
- Provider::Anthropic#chat_response now accepts (and ignores) a
`messages:` kwarg. Assistant::Responder passes both `messages:`
(OpenAI-shape) and `conversation_history:` (raw Message records) for
cross-provider parity, so the previous signature raised
ArgumentError on the first chat turn through the Anthropic provider.
- Provider::Anthropic#supports_model? bypasses the `claude` prefix
gate when a custom base_url is configured, mirroring the OpenAI
provider. Bedrock-shaped IDs like
`anthropic.claude-sonnet-4-5-20250929-v1:0` and
`claude-opus-4@20250514` are otherwise rejected by
Assistant::Provided#get_model_provider and the chat dies.
- Setting.anthropic_access_token is now in
EncryptedSettingFields::ENCRYPTED_FIELDS so the Anthropic API key
is encrypted at rest like every other provider secret. Previously
plaintext while siblings (openai_access_token, twelve_data_api_key,
external_assistant_token) were ciphertext.
- Chat.default_model falls back to whichever provider is actually
configured. Previously, with LLM_PROVIDER=anthropic but no
Anthropic credentials, the default model resolved to a Claude ID
that no registered provider supported, so chats failed even when
OpenAI was fully configured. Adds Provider::{Anthropic,Openai}#configured?
class methods for the readable callsite.
- Provider::Anthropic.effective_model uses
`ENV["ANTHROPIC_MODEL"].presence || Setting.anthropic_model` so the
Setting lookup is only performed when the env var is absent — the
previous `ENV.fetch(KEY, default)` evaluated the default arg
eagerly on every call.
- Provider::Anthropic::ChatConfig#anthropic_input_schema strips both
`:strict` and `"strict"` keys so JSON-decoded schemas with string
keys cannot leak the OpenAI-only flag through to Anthropic.
Test coverage added: supports_model? bypass on custom endpoints,
chat_response messages: kwarg compatibility, default_model fallback
in the three credential combinations, configured? against ENV +
Setting, strict-flag stripping for both key types, and a
`Setting.expects(:anthropic_model).never` assertion proving the
ENV-precedence test now exercises the lazy path.
All 4365 tests pass (1 pre-existing libvips env error unrelated).
* test(chat): make default_model tests resilient to ENV model overrides
CodeRabbit flagged on PR review: the new default_model tests asserted
against Provider::*::DEFAULT_MODEL, but Chat.default_model actually
returns Provider::*.effective_model.presence (which reads
OPENAI_MODEL / ANTHROPIC_MODEL from the environment). With either env
var set, the tests would fail intermittently even though routing was
correct.
- New default_model tests now assert against the provider's
effective_model directly, so they verify the routing decision
(which provider's value wins) without coupling to the constant.
- Pre-existing "creates with default model" assertions had the same
brittleness; switch them to compare against Chat.default_model so
the chosen model is whatever the env / Setting cascade resolves to.
Verified by running `ANTHROPIC_MODEL=claude-haiku-4-5 OPENAI_MODEL=gpt-4o
bin/rails test test/models/chat_test.rb` — 16 runs, 0 failures
(previously 2 pre-existing failures + 0 from the new tests).
* fix(ai): address local review on Anthropic foundation
- Provider::Anthropic#supports_pdf_processing? bypasses prefix gate for
custom endpoints, mirroring supports_model?
- Provider::Anthropic#initialize raises Error when custom_endpoint? AND
model.blank?, parity with Provider::Openai
- stream_chat_response captures partial usage on mid-stream errors and
records it via the new on_partial callback so chat_response can skip
the duplicate error row in the outer rescue
- safe_accumulated_message swallows the secondary failure when the SDK
cannot reconstruct a snapshot
- langfuse_client memoizes properly (||= instead of =) so repeated calls
don't churn Langfuse instances
- MessageFormatter sorts tool_calls by created_at then id so the
message array is deterministic across replays; skips tool_calls
missing both provider_call_id and provider_id rather than sending
`id: nil` and getting rejected by Anthropic
- Setting.anthropic_access_token default falls back through
ENV["ANTHROPIC_API_KEY"].presence (was missing .presence, so an
empty-string env value bled through)
- User#openai_configured? / #anthropic_configured? delegate to the
Provider::* class methods — single source of truth
- Assistant::Responder renames the OpenAI-shape history builder
conversation_history → openai_messages_payload so the kwarg name
matches the local method name (messages: openai_messages_payload,
conversation_history: chat_message_records)
- Assistant::Builtin stale-history comment updated to reference both
builders
Adds a streaming chat_response test using ad-hoc subclasses of the
SDK event types so the case/when dispatch matches via is_a? without
stubbing class-level === behavior.
* test(ai): add Anthropic tool_use round-trip + multi-tool turn coverage
Addresses @jjmata's "worth confirming" note on PR #1983: tool-use turns
from prior assistant messages must round-trip correctly when retrieved
from the database.
- New `ChatParser → ToolCall::Function → MessageFormatter` test walks
the full path: Anthropic response with a tool_use block →
ChatFunctionRequest → ToolCall::Function.from_function_request →
persisted on the AssistantMessage → MessageFormatter rebuild on the
next turn. Asserts the original `tool_use.id` is preserved end-to-end
as both `tool_use.id` and the paired `tool_result.tool_use_id`, and
that the original `input` hash and serialized result content survive.
- New multi-tool assistant turn test confirms two tool_use blocks on a
single assistant message render as two tool_use blocks followed by
two paired tool_result blocks in a single user-role follow-up,
matching Anthropic's required alternation.
Both tests exercise the existing PR1 code without behavior changes.
* test(ai): require "ostruct" explicitly in Anthropic provider tests
OpenStruct is moving out of Ruby's default load path (warning in 3.4+,
removed in 3.5+). Tests work today because ActiveSupport transitively
loads it, but that's incidental. Match the existing convention in
test/controllers/settings/hostings_controller_test.rb which explicitly
requires ostruct for the same reason.
* fix(ai): sanitize Langfuse warn logs, normalize tool_use.input, dedup history fetch
Addresses three open CodeRabbit findings on PR #1983.
- Provider::Anthropic Langfuse rescue branches no longer include
`e.full_message` in `Rails.logger.warn`. `full_message` bundles the
backtrace + cause chain and on some SDK error types includes the
serialized request/response payload (prompt, model output). Logs
now report `#{e.class}: #{e.message}` only. Three sites:
create_langfuse_trace, log_langfuse_generation, upsert_langfuse_trace.
Note: Provider::Openai has the same pattern (copy-pasted source) —
harmonization deferred to a follow-up cleanup PR; this commit fixes
only the Anthropic provider to keep PR scope tight.
- MessageFormatter#parse_arguments now coerces any non-Hash parsed
result to `{}`. Anthropic's Messages API requires `tool_use.input`
to be a JSON object (map); a stored ToolCall::Function record whose
arguments parse to a scalar, bool, or array (corrupt row, legacy
data, cross-provider bleed) would otherwise produce a payload the
API rejects. Normal flow stores Hash arguments end-to-end so the
fix is defensive — adds 2 tests covering scalar/array JSON strings
and non-String non-Hash inputs.
- Assistant::Responder dedups the chat-history fetch. The previous
layout fired two near-identical `chat.messages.where(...).includes(
:tool_calls).ordered` queries per LLM turn (one for the OpenAI-shape
payload, one for the raw-records kwarg). A new memoized
`complete_chat_messages` fetches once; `chat_message_records` filters
out the current message via `Array#reject`, `openai_messages_payload`
iterates the cached array unchanged. One SQL query per turn instead
of two. Memoization scope = single Responder instance (per LLM call),
so cache invalidation is not a concern.
All 4370 tests pass (1 pre-existing libvips env error unrelated).
Rubocop + brakeman clean.
* fix(ci): replace sk-ant- prefixed test placeholders
Pipelock secret scanner pattern-matches `sk-ant-*` as a real Anthropic
API key and fails the PR security-scan check. Test stubs and
ClimateControl env values used `sk-ant-test`, `sk-ant-from-setting`,
`sk-ant-x`, `sk-ant-y` as obvious placeholders, but the scanner does
not care about value entropy.
Switched to `fake-anthropic-key-*` / `fake-token-*` strings so the
scanner stops flagging them. No production code touched, no behavior
change — Provider::Anthropic still accepts any non-blank token.
* feat(ai): add Anthropic batch ops + LLM cost ledger (2/5)
Implements auto_categorize, auto_detect_merchants, and
enhance_provider_merchants on Provider::Anthropic via forced tool calls,
plus the cost-ledger plumbing they need.
- Provider::Anthropic::AutoCategorizer, AutoMerchantDetector,
ProviderMerchantEnhancer each define a single output tool whose
input_schema mirrors the desired output, then force the model to call
it via tool_choice: { type: "tool", name: ..., disable_parallel_tool_use: true }.
Anthropic guarantees the tool_use.input matches the schema, so there
is no JSON parsing fragility, no <think> tag stripping, and no
json_object/json_schema fallback ladders.
- Concerns::UsageRecorder mirrors the OpenAI sibling but persists
cache_creation_input_tokens / cache_read_input_tokens to dedicated
columns instead of metadata.
- Migration adds cache_creation_tokens, cache_read_tokens (nullable
integers) to llm_usages. OpenAI rows leave them null.
- LlmUsage::PRICING gains Claude 4.x rows (opus-4-7 $15/$75, sonnet-4-6
$3/$15, haiku-4-5 $1/$5 per MTok). infer_provider returns "anthropic"
for claude-* via the existing exact/prefix lookup.
- Provider::Anthropic#chat_response now persists cache columns directly
rather than stashing them in metadata.
- 25-transaction batch cap mirrors the OpenAI provider so the cost
ledger sees the same shape regardless of which provider ran a batch.
Tests cover the forced-tool-call path, null/None normalization,
case-insensitive merchant matching, the missing-tool_use error path,
and Anthropic-specific pricing + provider inference on LlmUsage.
Stacked on #1983 (PR 1/5). 3/5 PDF + vision next.
* fix(ai): attribute Bedrock model IDs to anthropic + clean nil enum
- LlmUsage.infer_provider now returns "anthropic" for Bedrock /
Vertex shaped IDs (anthropic.* and anthropic/*), so cost-ledger
filtering by provider stays correct even when no per-MTok rate is
stored. Previously these IDs fell through to the "openai" default.
- AutoCategorizer drops the redundant nil sentinel from the
category_name enum — the union type [string, null] already permits
null, and some JSON Schema validators reject nil literals inside
enum arrays.
* test(ai): require "ostruct" in Anthropic batch op tests
Same rationale as the PR1 ostruct fix — explicit require so the tests
don't depend on ActiveSupport's transitive load when Ruby 3.5+ removes
OpenStruct from the default load path.
* fix(llm-usage): include Anthropic cache tokens in estimated_cost
calculate_cost only priced prompt + completion tokens, so estimated_cost
under-reported every cached call — the cache_creation/cache_read columns this PR
added were tracked but never billed. Verified against the Anthropic dashboard: a
cached chat turn billed $0.05 but the ledger recorded $0.038; the gap was exactly
the unpriced cache tokens.
Price them relative to the input rate (Anthropic: cache write 1.25x, read 0.1x)
and thread the cache counts from both recorders (chat + batch). OpenAI rows leave
the columns null (treated as 0), so they're unaffected. Ledger now reproduces the
dashboard ($0.054 for the test turn).
* chore(ai): guard chat usage double-record; flag deferred Anthropic batch wiring
- Hardening: guard the success-path record_llm_usage with
`unless partial_usage_recorded` so a future change that emits partial usage on
a normal stream can't silently double-bill (the symptom investigated in the
#1984 review). No behavior change today — on_partial only fires from the
mid-stream-error rescue, which re-raises past this line.
- Notice: the family auto-categorize / merchant-detect / merchant-enhance flows
still hardcode get_provider(:openai). Provider::Anthropic now implements those
batch ops but they aren't wired into the family flows yet — documented with
TODOs at each site for the follow-up.
* chore(ai): point family-flow TODOs at tracking issue #2113
* fix(ai): address review findings on cost ledger + categorizer schema
Three AI-review findings on #1984:
- category_name enum omitted null (codex + coderabbit): the prompt + type allow
Claude to abstain on uncertain transactions, but JSON Schema `enum` restricted
the value to category names, so null was invalid — forcing miscategorization.
Append nil to the enum (the consumer already normalizes null -> uncategorized).
- Cache pricing applied to all providers (coderabbit): the 1.25x/0.1x cache
multipliers are Anthropic-specific. Gate them on provider == "anthropic" so a
non-Anthropic caller passing cache counts isn't billed with the wrong rates.
- Negative cache-token counts (coderabbit): add DB check constraints
(cache_*_tokens IS NULL OR >= 0), per the repo's DB-level-validation convention.
Tests: enum includes nil; non-Anthropic cache tokens aren't priced.
|
||
|
Guillem Arias Fauste
|
8251b7e4d6 |
feat(ai): add Anthropic provider with chat parity (1/5) (#1983)
* feat(ai): add Anthropic provider with chat parity (1/5)
Introduces Provider::Anthropic alongside Provider::Openai, implementing
the LlmConcept chat_response contract over the official anthropic Ruby
SDK. Batch ops, PDF, and RAG land in follow-up PRs.
- Provider::Anthropic uses Messages API for sync and streaming responses
- ChatConfig builds requests with ephemeral prompt-cache markers on the
system prompt and the last tool definition
- MessageFormatter reconstructs multi-turn history (text + tool_use +
tool_result blocks) from raw Message records, including the paired
user-role tool_result turn Anthropic requires after every tool_use
- ChatParser maps Anthropic Message into the shared ChatResponse Data
- Registry, Setting, User, Chat default model wired for ANTHROPIC_*
envs and Setting.anthropic_*; LLM_PROVIDER selects between providers
- Responder forwards raw conversation_history (Array<Message>) so
providers without hosted conversation state can rebuild context
- OpenAI provider accepts and ignores the new kwarg (no behavior change)
Tests cover provider init, model gating, MessageFormatter for all turn
shapes, ChatConfig request building (max_tokens, system cache, tool
conversion), ChatParser for text / tool_use / mixed blocks, Registry
discovery, and mocked chat_response success / error / function_request
paths. Live VCR cassettes recorded in a follow-up with a real key.
Stacked PRs: 2/5 batch ops + cost ledger, 3/5 PDF, 4/5 pgvector RAG,
5/5 settings UI + disclosure.
* fix(ai): address PR review on Anthropic provider foundation
Surface fixes raised by Codex + CodeRabbit on PR 1/5:
- Provider::Anthropic#chat_response now accepts (and ignores) a
`messages:` kwarg. Assistant::Responder passes both `messages:`
(OpenAI-shape) and `conversation_history:` (raw Message records) for
cross-provider parity, so the previous signature raised
ArgumentError on the first chat turn through the Anthropic provider.
- Provider::Anthropic#supports_model? bypasses the `claude` prefix
gate when a custom base_url is configured, mirroring the OpenAI
provider. Bedrock-shaped IDs like
`anthropic.claude-sonnet-4-5-20250929-v1:0` and
`claude-opus-4@20250514` are otherwise rejected by
Assistant::Provided#get_model_provider and the chat dies.
- Setting.anthropic_access_token is now in
EncryptedSettingFields::ENCRYPTED_FIELDS so the Anthropic API key
is encrypted at rest like every other provider secret. Previously
plaintext while siblings (openai_access_token, twelve_data_api_key,
external_assistant_token) were ciphertext.
- Chat.default_model falls back to whichever provider is actually
configured. Previously, with LLM_PROVIDER=anthropic but no
Anthropic credentials, the default model resolved to a Claude ID
that no registered provider supported, so chats failed even when
OpenAI was fully configured. Adds Provider::{Anthropic,Openai}#configured?
class methods for the readable callsite.
- Provider::Anthropic.effective_model uses
`ENV["ANTHROPIC_MODEL"].presence || Setting.anthropic_model` so the
Setting lookup is only performed when the env var is absent — the
previous `ENV.fetch(KEY, default)` evaluated the default arg
eagerly on every call.
- Provider::Anthropic::ChatConfig#anthropic_input_schema strips both
`:strict` and `"strict"` keys so JSON-decoded schemas with string
keys cannot leak the OpenAI-only flag through to Anthropic.
Test coverage added: supports_model? bypass on custom endpoints,
chat_response messages: kwarg compatibility, default_model fallback
in the three credential combinations, configured? against ENV +
Setting, strict-flag stripping for both key types, and a
`Setting.expects(:anthropic_model).never` assertion proving the
ENV-precedence test now exercises the lazy path.
All 4365 tests pass (1 pre-existing libvips env error unrelated).
* test(chat): make default_model tests resilient to ENV model overrides
CodeRabbit flagged on PR review: the new default_model tests asserted
against Provider::*::DEFAULT_MODEL, but Chat.default_model actually
returns Provider::*.effective_model.presence (which reads
OPENAI_MODEL / ANTHROPIC_MODEL from the environment). With either env
var set, the tests would fail intermittently even though routing was
correct.
- New default_model tests now assert against the provider's
effective_model directly, so they verify the routing decision
(which provider's value wins) without coupling to the constant.
- Pre-existing "creates with default model" assertions had the same
brittleness; switch them to compare against Chat.default_model so
the chosen model is whatever the env / Setting cascade resolves to.
Verified by running `ANTHROPIC_MODEL=claude-haiku-4-5 OPENAI_MODEL=gpt-4o
bin/rails test test/models/chat_test.rb` — 16 runs, 0 failures
(previously 2 pre-existing failures + 0 from the new tests).
* fix(ai): address local review on Anthropic foundation
- Provider::Anthropic#supports_pdf_processing? bypasses prefix gate for
custom endpoints, mirroring supports_model?
- Provider::Anthropic#initialize raises Error when custom_endpoint? AND
model.blank?, parity with Provider::Openai
- stream_chat_response captures partial usage on mid-stream errors and
records it via the new on_partial callback so chat_response can skip
the duplicate error row in the outer rescue
- safe_accumulated_message swallows the secondary failure when the SDK
cannot reconstruct a snapshot
- langfuse_client memoizes properly (||= instead of =) so repeated calls
don't churn Langfuse instances
- MessageFormatter sorts tool_calls by created_at then id so the
message array is deterministic across replays; skips tool_calls
missing both provider_call_id and provider_id rather than sending
`id: nil` and getting rejected by Anthropic
- Setting.anthropic_access_token default falls back through
ENV["ANTHROPIC_API_KEY"].presence (was missing .presence, so an
empty-string env value bled through)
- User#openai_configured? / #anthropic_configured? delegate to the
Provider::* class methods — single source of truth
- Assistant::Responder renames the OpenAI-shape history builder
conversation_history → openai_messages_payload so the kwarg name
matches the local method name (messages: openai_messages_payload,
conversation_history: chat_message_records)
- Assistant::Builtin stale-history comment updated to reference both
builders
Adds a streaming chat_response test using ad-hoc subclasses of the
SDK event types so the case/when dispatch matches via is_a? without
stubbing class-level === behavior.
* test(ai): add Anthropic tool_use round-trip + multi-tool turn coverage
Addresses @jjmata's "worth confirming" note on PR #1983: tool-use turns
from prior assistant messages must round-trip correctly when retrieved
from the database.
- New `ChatParser → ToolCall::Function → MessageFormatter` test walks
the full path: Anthropic response with a tool_use block →
ChatFunctionRequest → ToolCall::Function.from_function_request →
persisted on the AssistantMessage → MessageFormatter rebuild on the
next turn. Asserts the original `tool_use.id` is preserved end-to-end
as both `tool_use.id` and the paired `tool_result.tool_use_id`, and
that the original `input` hash and serialized result content survive.
- New multi-tool assistant turn test confirms two tool_use blocks on a
single assistant message render as two tool_use blocks followed by
two paired tool_result blocks in a single user-role follow-up,
matching Anthropic's required alternation.
Both tests exercise the existing PR1 code without behavior changes.
* test(ai): require "ostruct" explicitly in Anthropic provider tests
OpenStruct is moving out of Ruby's default load path (warning in 3.4+,
removed in 3.5+). Tests work today because ActiveSupport transitively
loads it, but that's incidental. Match the existing convention in
test/controllers/settings/hostings_controller_test.rb which explicitly
requires ostruct for the same reason.
* fix(ai): sanitize Langfuse warn logs, normalize tool_use.input, dedup history fetch
Addresses three open CodeRabbit findings on PR #1983.
- Provider::Anthropic Langfuse rescue branches no longer include
`e.full_message` in `Rails.logger.warn`. `full_message` bundles the
backtrace + cause chain and on some SDK error types includes the
serialized request/response payload (prompt, model output). Logs
now report `#{e.class}: #{e.message}` only. Three sites:
create_langfuse_trace, log_langfuse_generation, upsert_langfuse_trace.
Note: Provider::Openai has the same pattern (copy-pasted source) —
harmonization deferred to a follow-up cleanup PR; this commit fixes
only the Anthropic provider to keep PR scope tight.
- MessageFormatter#parse_arguments now coerces any non-Hash parsed
result to `{}`. Anthropic's Messages API requires `tool_use.input`
to be a JSON object (map); a stored ToolCall::Function record whose
arguments parse to a scalar, bool, or array (corrupt row, legacy
data, cross-provider bleed) would otherwise produce a payload the
API rejects. Normal flow stores Hash arguments end-to-end so the
fix is defensive — adds 2 tests covering scalar/array JSON strings
and non-String non-Hash inputs.
- Assistant::Responder dedups the chat-history fetch. The previous
layout fired two near-identical `chat.messages.where(...).includes(
:tool_calls).ordered` queries per LLM turn (one for the OpenAI-shape
payload, one for the raw-records kwarg). A new memoized
`complete_chat_messages` fetches once; `chat_message_records` filters
out the current message via `Array#reject`, `openai_messages_payload`
iterates the cached array unchanged. One SQL query per turn instead
of two. Memoization scope = single Responder instance (per LLM call),
so cache invalidation is not a concern.
All 4370 tests pass (1 pre-existing libvips env error unrelated).
Rubocop + brakeman clean.
* fix(ci): replace sk-ant- prefixed test placeholders
Pipelock secret scanner pattern-matches `sk-ant-*` as a real Anthropic
API key and fails the PR security-scan check. Test stubs and
ClimateControl env values used `sk-ant-test`, `sk-ant-from-setting`,
`sk-ant-x`, `sk-ant-y` as obvious placeholders, but the scanner does
not care about value entropy.
Switched to `fake-anthropic-key-*` / `fake-token-*` strings so the
scanner stops flagging them. No production code touched, no behavior
change — Provider::Anthropic still accepts any non-blank token.
|
||
|
Sure Admin (bot)
|
2df10ca4ef |
Retry Enable Banking sync with provider-corrected date range (#1801)
* Clamp Enable Banking sync window * Pipelock noise --------- Co-authored-by: KiloClaw <kiloclaw@openclaw.ai> Co-authored-by: Juan José Mata <jjmata@jjmata.com> |
||
|
ghost
|
95f6451b39 |
feat(sync): add Brex provider connections (#1752)
* feat(sync): add Brex provider schema Adds Brex item and account tables with per-family credentials, scoped upstream account uniqueness, encrypted token storage, and sanitized provider payload columns. * feat(sync): add Brex provider core Adds Brex item/account models, provider client and adapter support, family connection helpers, and provider enum registration for read-only Brex cash and card data. * feat(sync): add Brex import pipeline Adds Brex account discovery, linked-account sync, cash/card balance processors, transaction import, sanitized metadata handling, and idempotent provider entry processing. * feat(sync): add Brex connection flows Adds Mercury-style Brex connection management, explicit item-scoped account selection and linking, settings provider UI, account index visibility, localized copy, and per-item cache handling. * test(sync): cover Brex provider workflows Adds targeted coverage for Brex provider requests, adapter config, item/account guards, importer behavior, entry processing, and Mercury-style controller flows. * fix(sync): align Brex API edge cases Tightens Brex account fetching against the official card-account response shape, sends transaction start filters as RFC3339 date-times, and keeps provider error bodies out of user-facing messages while expanding provider client guard coverage. * fix(sync): harden Brex provider integration Restrict Brex API base URLs to official hosts, tighten account-selection UI behavior, and add tests for invalid credentials, cache scoping, and provider setup edge cases. * test(sync): avoid Brex secret-shaped fixtures * refactor(sync): extract Brex account flows * fix(sync): address Brex provider review feedback * fix(sync): address Brex review follow-ups Move remaining Brex review cleanup into focused model behavior, tighten link/setup edge cases, localize summaries, and add regression coverage from CodeRabbit feedback. Also records the security-review pass as no-findings after diff-scoped inspection and Brakeman validation. * refactor(sync): split Brex account flow controllers Route Brex account selection and setup actions through small namespaced controllers while keeping existing URLs and helpers stable. Business flow remains in BrexItem::AccountFlow; the main Brex item controller now only handles connection CRUD, provider-panel rendering, destroy, and sync. * fix(sync): address Brex CodeRabbit review * fix(sync): address Brex follow-up review * fix(sync): address Brex review follow-ups * fix(sync): address Brex sync review findings * fix(sync): polish Brex review copy and errors * fix(sync): register Brex provider health * fix(sync): polish Brex bank sync presentation * fix(sync): address Brex review follow-ups * fix(sync): tighten Brex setup params * test(api): stabilize usage rate-limit window * fix(sync): polish Brex setup flow nits * fix(sync): harden Brex setup params * fix(sync): finalize Brex review cleanup --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
plind
|
12d799e0b8 |
fix(binance): support CRYPTO: prefix and USD stablecoins (#1771)
* fix(binance): support CRYPTO: prefix and USD stablecoins Holdings processors (CoinStats, Coinbase, Kraken, SimpleFIN, Lunchflow, Binance) store crypto securities with a "CRYPTO:" prefix, but Provider::BinancePublic#parse_ticker only accepted Binance-search-style tickers like "BTCUSD". As a result, every fetched price for tickers like CRYPTO:USDT, CRYPTO:USDC, CRYPTO:SOL, CRYPTO:TRUMP, CRYPTO:KAITO failed with "Unsupported Binance ticker". - Strip the CRYPTO: prefix in parse_ticker. - Short-circuit USD-pegged stablecoins (USDT, USDC, BUSD, DAI, FDUSD, TUSD, USDP, PYUSD) to a synthetic flat 1.0 USD price. Binance has no self-pair (USDTUSDT is invalid), and the few stablecoin/USDT pairs that do exist hover at ~1.0 with sub-cent noise. - Default prefixed bare base assets (CRYPTO:SOL etc.) to the …USDT pair (USD). Only when prefixed, so unprefixed garbage like BTCBNB / BTCGBP still returns nil and the existing rejection tests still pass. - fetch_security_info returns links: nil for stablecoins rather than a broken /trade/ URL. Closes #1441. * fix(binance): strip CRYPTO: prefix in search_securities Security::Resolver calls search_provider with the raw holdings-processor symbol (CRYPTO:SOL, CRYPTO:USDT) before any price fetch. Without prefix handling here, first-time crypto imports never resolve to an online Binance security and the new stablecoin/prefix paths in parse_ticker were unreachable for that flow. - Strip CRYPTO: from the search query. - Short-circuit USD stablecoins to a synthetic search result (no exchangeInfo call, no Binance self-pair to find). - Teach parse_ticker the "{stablecoin}USD" form produced by the synthetic result so price fetches route to stablecoin_prices. --------- Co-authored-by: plind-junior <plind-junior@users.noreply.github.com> |
||
|
ghost
|
be598aecf0 |
feat(providers): add Kraken exchange sync (#1759)
* feat(providers): add Kraken exchange sync Adds family-scoped Kraken API-key connections, read-only balance and trade import, account setup/linking flows, provider status wiring, and focused test coverage. Closes #1758 * test(providers): avoid Kraken sample secret false positive * fix(providers): address Kraken review findings * fix(providers): address Kraken review cleanup * test(imports): stabilize transaction import ordering |
||
|
Juan José Mata
|
81cdccb768 |
[codex] Complete Sophtron account mapping (#1698)
* Complete Sophtron account mapping * Clarify Sophtron login challenge flow * Add Sophtron connection UI timeout * Treat Sophtron timeout jobs as failed * Reset failed Sophtron connection state * Handle stale Sophtron connection jobs * Advance Sophtron polling timeout * Shorten Sophtron connection timeout * Fix Sophtron modal polling updates * Stabilize Sophtron MFA polling * Give Sophtron OTP challenges more time * Clarify Sophtron institution login failures * Extend Sophtron polling during login progress * Probe Sophtron accounts after completed MFA step * Align Sophtron dialogs with design system * Start Sophtron initial load after linking accounts * Fix Sophtron initial transaction load * Fail Sophtron sync without institution connection * Fix tests * Wrap Sophtron account linking in transaction * Wrap Sophtron provider responses * Fix Sophtron MFA security tests * Guard Sophtron MFA challenge arrays * Respect Sophtron initial load window * Use unique Sophtron MFA answer field ids * Address Sophtron review follow-ups * Fix Sophtron transaction sync refresh * Avoid blocking Sophtron refresh polling * Move Sophtron account helpers to model * Keep Sophtron grouping provider-level * Start new Sophtron institution links * Isolate Sophtron institution connections --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
GermanDZ
|
9cc52b9d35 |
fix: handle OpenAI Responses API stream errors instead of crashing (#1669)
The streaming code assumed every stream produced a `response.completed` event and dereferenced its data unconditionally, causing `undefined method 'data' for nil` whenever OpenAI emitted `response.failed`, `response.incomplete`, or a top-level `error` event (e.g. expired `previous_response_id`, context-window overflow, transient upstream failures). Surface a descriptive `Provider::Error` instead. - Extend `ChatStreamParser` to recognise `response.failed`, `response.incomplete`, and `error` events and emit an `error` chunk with a `StreamErrorData` payload (event, message, code, details). - In `Provider::Openai#native_chat_response`, detect the missing `response` chunk, build a user-facing error message from the collected error chunk, and raise `Provider::Error`. - Add unit tests for the parser (8 cases) and integration tests for the error path in the chat response flow. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Abhinav Dhiman
|
139c89d0f4 |
feat(investments): add India investment subtypes and exchange support (#1659)
* feat(investments): add India investment subtypes and exchange support * fix(yahoo-finance): scope Indian exchange de-duplication per company instead of globally Resolves feedback from Codex and CodeRabbit on #1413. prefer_indian_exchange previously collapsed all Indian securities into a single entry, silently dropping unrelated tickers. Now groups Indian listings by name and only de-duplicates within each group, so distinct companies (e.g. Reliance and Infosys) are preserved while NSE/BSE dual-listings still prefer NSE. - Derive India subtype keys dynamically from Investment::SUBTYPES in tests - Fix missing keyword arguments in Security.new test calls * refactor(yahoo-finance): generalize exchange config and dual-listing de-duplication Replaces hardcoded Indian exchange logic with a declarative EXCHANGE_CONFIG hash that maps ISO MIC codes to Yahoo-specific settings (symbol suffix, default currency, dual-listing group, and preference rank). This makes adding new markets a one-line hash entry instead of scattered conditionals. * fix(yahoo-finance): normalize security names for dual-listing de-duplication * fix(yahoo-finance): skip dual-listing de-duplication when filtering by exchange * fix: address PR review feedback for India market support - fix cache key mismatch in fetch_security_price by normalizing symbol before building cache key - remove dead YAHOO_EXCHANGE_CURRENCY constant - tighten normalize_symbol guard to use end_with?(suffix) instead of include?('.') - remove misleading '# India' comment from Property::SUBTYPES - remove 'rented' property subtype in favor of 'investment_property' - rename 'demat' to 'indian_stocks' for clarity - add INR to CURRENCY_REGION_MAP so India appears first for INR users - add dotted-symbol regression test for normalize_symbol * fix(investments): rename 'demat' subtype to 'indian_stocks' and remove trailing comma |
||
|
ghost
|
6c84fc760e |
fix(mercury): support named multiple API connections (#1627)
* fix(mercury): support named multiple connections * fix(mercury): address multi-connection review feedback * fix(mercury): localize connection labels * fix(mercury): strip API tokens before provider calls * test(mercury): localize provider config assertions * fix(mercury): address multi-connection review * refactor(mercury): simplify connection selection failure |
||
|
Juan José Mata
|
7b2b1dd367 |
Rebase PR #784 and fix OpenAI model/chat regressions (#1384)
* Wire conversation history through OpenAI responses API * Fix RuboCop hash brace spacing in assistant tests * Pipelock ignores * Batch fixes --------- Co-authored-by: sokiee <sokysrm@gmail.com> |
||
|
soky srm
|
dcebda05de |
Move back to brandfetch (#1427)
* Move back to brandfetch * Update security.rb * Update security.rb |
||
|
soky srm
|
0aca297e9c |
Add binance security provider for crypto (#1424)
* Binance as securities provider * Disable twelve data crypto results * Add logo support and new currency pairs * FIX importer fallback * Add price clamping and optiimize retrieval * Review * Update adding-a-securities-provider.md * day gap miss fix * New fixes * Brandfetch doesn't support crypto. add new CDN * Update _investment_performance.html.erb |
||
|
soky srm
|
be42988adf |
Add throttling and cross-rate for twelve data (#1396)
* Add throttling and cross-rate for twelve data * FIX yahoo precision also * FIXES * Update importer.rb * Fixes * Revert job * Fixes |
||
|
Louis
|
455c74dcfa |
Add Binance support, heavily inspired by the Coinbase one (#1317)
* feat: add Binance support (Items, Accounts, Importers, Processor, and Sync) * refactor: deduplicate 'stablecoins' constant and push stale_rate filter to SQL --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
Serge L
|
15cfcf585d |
Fix: Yahoo Finance provider Cookie/Crumb Auth (#1082)
* Fix: use cookie/crumb auth in healthy? chart endpoint check The health check was calling /v8/finance/chart/AAPL via the plain unauthenticated client. Yahoo Finance requires cookie + crumb authentication on the chart endpoint, so the health check would fail even when credentials are valid. Updated healthy? to use fetch_cookie_and_crumb + authenticated_client, consistent with fetch_security_prices and fetch_chart_data. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix: add cookie/crumb auth to all /v8/finance/chart/ calls fetch_security_prices and fetch_chart_data (used for exchange rates) were calling the chart endpoint without cookie/crumb authentication, inconsistent with healthy? and fetch_security_info. Added auth to both, including the same retry-on-Unauthorized pattern already used in fetch_security_info. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Update user-agent strings in yahoo_finance.rb Updated user-agent strings to reflect current browser versions Signed-off-by: Serge L <serge@souritech.ca> * Fix: Add stale-crumb retry to healthy? and fetch_chart_data Yahoo Finance returns 200 OK with {"chart":{"error":{"code":"Unauthorized"}}} when a cached crumb expires server-side. Both healthy? and fetch_chart_data now mirror the retry pattern already in fetch_security_prices: detect the Unauthorized body, clear the crumb cache, fetch fresh credentials, and retry the request once. Adds a test for the healthy? retry path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Refactor: Extract fetch_authenticated_chart helper to DRY crumb retry logic The cookie/crumb fetch + stale-crumb retry pattern was duplicated across healthy?, fetch_security_prices, and fetch_chart_data. Extract it into a single private fetch_authenticated_chart(symbol, params) helper that centralizes the retry logic; all three call sites now delegate to it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix: Catch JSON::ParserError in fetch_chart_data rescue clause After moving JSON.parse inside fetch_authenticated_chart, a malformed Yahoo response would throw JSON::ParserError through fetch_chart_data's rescue Faraday::Error, breaking the inverse currency pair fallback. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix: Raise AuthenticationError if retry still returns Unauthorized After refreshing the crumb and retrying, if Yahoo still returns an Unauthorized error body the helper now raises AuthenticationError instead of silently returning the error payload. This prevents callers from misinterpreting a persistent auth failure as missing chart data. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix: Raise AuthenticationError after failed retry in fetch_security_info Mirrors the same post-retry Unauthorized check added to fetch_authenticated_chart. Without this, a persistent auth failure on the quoteSummary endpoint would surface as a generic "No security info found" error instead of an AuthenticationError. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Signed-off-by: Serge L <serge@souritech.ca> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
Juan José Mata
|
90e94f0ad1 |
Use Langfuse client trace upsert API (#1041)
Replace direct trace.update calls with client trace upserts so OpenAI provider is compatible with langfuse-ruby 0.1.6 behavior. Add richer warning logs that include full exception details for trace creation, trace upserts, and generation logging failures. Add tests for client-based trace upserts and detailed error logging. |
||
|
David Gil
|
ba442d5f26 |
Implement Indexa Capital provider with real API integration (#933)
* Add Indexa Capital provider scaffold
Generate Indexa Capital provider scaffolding and align credential fields with the API authentication requirements.
* Fix PR 926 lint and schema CI failures
* Implement Indexa Capital provider with real API integration
- Rewrite all broken view templates (were meta-ERB from code generator)
- Create missing select_accounts.html.erb template
- Implement real API calls: list_accounts via /users/me, get_holdings
via /accounts/{number}/fiscal-results, get_account_balance via
/accounts/{number}/performance
- Add API token auth support (stored token > env token > credentials)
- Add api_token column with encryption support
- Redesign settings panel: API token prominent, credentials collapsible
- Fix account balances display using performance endpoint portfolios
- Fix accounts index empty-state guard missing indexa_capital_items
- Simplify activities fetch job (no activities API endpoint exists)
- Fix i18n interpolation (%%{ -> %{) throughout locale file
* Add tests for Indexa Capital provider integration
- IndexaCapitalItemTest: validations, credentials, scopes, sync status
- IndexaCapitalAccountTest: upsert, holdings, account provider linking
- Provider::IndexaCapitalTest: auth modes, API stubs, error handling
- IndexaCapitalItemsControllerTest: CRUD, setup, linking, authorization
- Fixtures for items (token + credentials) and accounts (mutual + pension)
52 tests, 98 assertions, 0 failures
* Address code review feedback from PR #933
- Fix zero balance bug: use `nil?` instead of `present?` so 0 is stored
- Fix has_indexa_capital_credentials? to check api_token (was ignored)
- Fix build_provider to delegate to Provided concern (was ignoring token)
- Fix IndexaCapital section outside encryption_error guard in settings
- Add account_number sanitization to prevent path traversal in API URLs
- Replace all skipped processor tests with real working tests
- Add zero-balance and path-traversal test coverage
61 tests, 107 assertions, 0 failures
* Address code review round 2: credentials validation, RuboCop, test quality
- Fix RuboCop SpaceInsideArrayLiteralBrackets in credentials check
- Chain where.not calls so all three username/document/password must be present
- Require all three credentials (||) instead of any one (&&) in validate_configuration!
- Move attr_reader to private to avoid exposing credentials publicly
- Parse dates with Date.parse in extract_balance for robustness
- Remove stale TODO and Crypto from supported_account_types
- Order build_provider query deterministically by created_at
- Replace no-op holdings assertion with meaningful assert_difference
* Address code review round 3: JSON parse safety and test precision
- Rescue JSON::ParserError on 2xx responses for clearer error messages
- Fix weak balance assertion: set balance to 0 before processing, assert
expected value (27093.01 = sum of holdings amounts)
* Include Indexa Capital in automatic family sync
Add indexa_capital_items to Family::Syncer#child_syncables so balances
and holdings refresh on daily auto-sync and login sync, not only on
manual sync button clicks.
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
|
||
|
BitToby
|
ba6e286b41 |
feat: add SSL_CA_FILE and SSL_VERIFY environment variables to support… (#894)
* feat: add SSL_CA_FILE and SSL_VERIFY environment variables to support self-signed certificates in self-hosted environments * fix: NoMethodError by defining SSL helper methods before configure block executes * refactor: Refactor SessionsController to use shared SslConfigurable module and simplify SSL initializer redundant checks * refactor: improve SSL configuration robustness and error detection accuracy * fix:HTTParty SSL options, add file validation guards, prevent Tempfile GC, and redact URLs in error logs * fix: Fix SSL concern indentation and stub Simplefin POST correctly in tests * fix: normalize ssl_verify to always return boolean instead of nil * fix: solve failing SimpleFin test * refactor: trim unused error-handling code from SslConfigurable, replace Tempfile with fixed-path CA bundle, fix namespace pollution in initializers, and add unit tests for core SSL configuration and Langfuse CRL callback. * fix: added require ileutils in the initializer and require ostruct in the test file. * fix: solve autoload conflict that broke provider loading, validate all certs in PEM bundles, and add missing requires. |
||
|
MkDev11
|
0afdb1d0fd |
Feature/pdf import transaction rows (#846)
* Add import row generation from PDF extracted data - Add generate_rows_from_extracted_data method to PdfImport - Add import! method to create transactions from PDF rows - Update ProcessPdfJob to generate rows after extraction - Update configured?, cleaned?, publishable? for PDF workflow - Add column_keys, required_column_keys, mapping_steps - Set bank statements to pending status for user review - Add tests for new functionality Closes #844 * Add tests for BankStatementExtractor - Test transaction extraction from PDF content - Test deduplication across chunk boundaries - Test amount normalization for various formats - Test graceful handling of malformed JSON responses - Test error handling for empty/nil PDF content * Fix supports_pdf_processing? to validate effective model The validation was always checking @default_model, but process_pdf allows overriding the model via parameter. This could cause a vision-capable override model to be rejected, or a non-vision-capable override to pass validation only to fail during processing. Changes: - supports_pdf_processing? now accepts optional model parameter - process_pdf passes effective model to validation - Raise Provider::Openai::Error inside with_provider_response for consistent error handling Addresses review feedback from PR#808 * Fix insert_all! bug: explicitly set import_id Rails insert_all! on associations does NOT auto-set the foreign key. Added import_id explicitly and use Import::Row.insert_all! directly. Also reload rows before counting to ensure accurate count. * Fix pending status showing as processing for bank statements with rows When bank statement PDF imports have extracted rows, show a 'Ready for Review' screen with a link to the confirm path instead of the 'Processing' spinner. This addresses the PR feedback that users couldn't reach the review flow even though rows were created. * Gate publishable? on account.present? to prevent import failure PDF imports are created without an account, and import! raises if account is missing. This prevents users from hitting publish and having the job fail. * Wrap generate_rows_from_extracted_data in transaction for atomicity - Clear rows and reset count even when no transactions extracted - Use transaction block to prevent partial updates on failure - Use mapped_rows.size instead of reload for count * Localize transactions count string with i18n helper * Add AccountMapping step for PDF imports when account is nil PDF imports need account selection before publishing. This adds Import::AccountMapping to mapping_steps when account is nil, matching the behavior of TransactionImport and TradeImport. Addresses PR#846 feedback about account selection for PDF imports. * Only include CategoryMapping when rows have non-empty categories PDF extraction doesn't extract categories from bank statements, so the CategoryMapping step would show empty. Now we only include CategoryMapping if rows actually have non-empty category values. This prevents showing an empty mapping step for PDF imports. * Fix PDF import UI flow and account selection - Add direct account selection in PDF import UI instead of AccountMapping - AccountMapping designed for CSV imports with multiple account values - PDF imports need single account for all transactions - Add update action and route for imports controller - Fix controller to handle pdf_import param format from form_with - Show Publish button when import is publishable (account set) - Fix stepper nav: Upload/Configure/Clean non-clickable for PDF imports - Redirect PDF imports from configuration step (auto-configured) - Improve AI prompt to recognize M-PESA/mobile money as bank statements - Fix migration ordering for import_rows table columns * Add guard for invalid account_id in imports#update Prevents silently clearing account when invalid ID is passed. Returns error message instead of confusing 'Account saved' notice. * Localize step names in import nav and add account guard - Use t() helper for all step names (Upload, Configure, Clean, Map, Confirm) - Add guard for invalid account_id in imports#update - Prevents silently clearing account when invalid ID is passed * Make category column migrations idempotent Check if columns exist before adding to prevent duplicate column errors when migrations are re-run with new timestamps. * Add match_path for PDF import step highlighting Fixes step detection when path is nil by using separate match_path for current step highlighting while keeping links disabled. * Rename category migrations and update to Rails 7.2 - Rename class to EnsureCategoryFieldsOnImportRows to avoid conflicts - Rename class to EnsureCategoryIconOnImportRows - Update migration version from 7.1 to 7.2 per guidelines - Rename files to match class names - Add match_path for PDF import step highlighting * Use primary (black) style for Create Account and Save buttons * Remove match_path from auto-completed PDF steps Only step 4 (Confirm) needs match_path for active-step detection. Steps 1-3 are purely informational and always complete. * Add fallback for document type translation Handles nil or unexpected document_type values gracefully. Also removes match_path from auto-completed PDF steps. * Use index-based step number for mobile indicator Fixes 'Step 5 of 4' issue when Map step is dynamically removed. * Fix hostings_controller_test: use blank? instead of nil Setting returns empty string not nil for unset values. * Localize step progress label and use design token * Fix button styling: use design system Tailwind classes btn--primary and btn--secondary CSS classes don't exist. Use actual design system classes from DS::Buttonish. * Fix CRLF line endings in tags_controller_test.rb --------- Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com> |
||
|
Juan José Mata
|
6b5a5b1877 |
fix: Show cancellation message when subscription is pending cancellation (#752)
* fix: Show cancellation message when subscription is pending cancellation When a subscription is cancelled via Stripe, the UI incorrectly showed "Your contribution continues on..." instead of reflecting the cancellation status. This fix adds tracking of `cancel_at_period_end` from Stripe webhooks and displays "Your contribution ends on..." when a subscription has been cancelled but is still active until the billing period ends. https://claude.ai/code/session_01Y8ELTdK1k9o315iSq43TRN * chore: Update schema.rb with cancel_at_period_end column https://claude.ai/code/session_01Y8ELTdK1k9o315iSq43TRN * Schema version --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
soky srm
|
179552657c |
Mercury integration (#723)
* Initial mercury impl * FIX both mercury and generator class * Finish mercury integration and provider generator * Fix schema * Fix linter and tags * Update routes.rb * Avoid schema drift --------- Signed-off-by: soky srm <sokysrm@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
soky srm
|
0e22c60286 |
FIX Yahoo issues (#636)
* FIX Yahoo issues * Update yahoo_finance_test.rb * FIX proper cookie access |
||
|
Ethan
|
3b4ab735b0 |
Add (beta) CoinStats Crypto Wallet Integration with Balance and Transaction Syncing (#512)
* Feat(CoinStats): Scaffold implementation, not yet functional * Feat(CoinStats): Implement crypto wallet balance and transactions * Feat(CoinStats): Add tests, Minor improvements * Feat(CoinStats): Utilize bulk fetch API endpoints * Feat(CoinStats): Migrate strings to i8n * Feat(CoinStats): Fix error handling in wallet link modal * Feat(CoinStats): Implement hourly provider sync job * Feat(CoinStats): Generate docstrings * Fix(CoinStats): Validate API Key on provider update * Fix(Providers): Safely handle race condition in merchance creation * Fix(CoinStats): Don't catch system signals in account processor * Fix(CoinStats): Preload before iterating accounts * Fix(CoinStats): Add no opener / referrer to API dashboard link * Fix(CoinStats): Use strict matching for symbols * Fix(CoinStats): Remove dead code in transactions importer * Fix(CoinStats): Avoid transaction fallback ID collisions * Fix(CoinStats): Improve Blockchains fetch error handling * Fix(CoinStats): Enforce NOT NULL constraint for API Key schema * Fix(CoinStats): Migrate sync status strings to i8n * Fix(CoinStats): Use class name rather than hardcoded string * Fix(CoinStats): Use account currency rather than hardcoded USD * Fix(CoinStats): Migrate from standalone to Provider class * Fix(CoinStats): Fix test failures due to string changes |
||
|
LPW
|
c12c585a0e |
Harden SimpleFin sync: retries, safer imports, manual relinking, and data-quality reconciliation (#544)
* Add tests and enhance logic for SimpleFin account synchronization and reconciliation - Added retry logic with exponential backoff for network errors in `Provider::Simplefin`. - Introduced tests to verify retry functionality and error handling for rate-limit, server errors, and stale data. - Updated `SimplefinItem` to detect stale sync status and reconciliation issues. - Enhanced UI to display stale sync warnings and data integrity notices. - Improved SimpleFin account matching during updates with multi-tier strategy (ID, fingerprint, fuzzy match). - Added transaction reconciliation logic to detect data gaps, transaction count drops, and duplicate transaction IDs. * Introduce `SimplefinConnectionUpdateJob` for asynchronous SimpleFin connection updates - Moved SimpleFin connection update logic to `SimplefinConnectionUpdateJob` to improve response times by offloading network retries, data fetching, and reconciliation tasks. - Enhanced SimpleFin account matching with a multi-tier strategy (ID, fingerprint, fuzzy name match). - Added retry logic and bounded latency for token claim requests in `Provider::Simplefin`. - Updated tests to cover the new job flow and ensure correct account reconciliation during updates. * Remove unused SimpleFin account matching logic and improve error handling in `SimplefinConnectionUpdateJob` - Deleted the multi-tier account matching logic from `SimplefinItemsController` as it is no longer used. - Enhanced error handling in `SimplefinConnectionUpdateJob` to gracefully handle import failures, ensuring orphaned items can be manually resolved. - Updated job flow to conditionally set item status based on the success of import operations. * Fix SimpleFin sync: check both legacy FK and AccountProvider for linked accounts * Add crypto, checking, savings, and cash account detection; refine subtype selection and linking - Enhanced `Simplefin::AccountTypeMapper` to include detection for crypto, checking, savings, and standalone cash accounts. - Improved subtype selection UI with validation and warning indicators for missing selections. - Updated SimpleFin account linking to handle both legacy FK and `AccountProvider` associations consistently. - Refined job flow and importer logic for better handling of linked accounts and subtype inference. * Improve `SimplefinConnectionUpdateJob` and holdings processing logic - Fixed race condition in `SimplefinConnectionUpdateJob` by moving `destroy_later` calls outside of transactions. - Updated fuzzy name match logic to use Levenshtein distance for better accuracy. - Enhanced synthetic ticker generation in holdings processor with hash suffix for uniqueness. * Refine SimpleFin entry processing logic and ensure `extra` data persistence - Simplified pending flag determination to rely solely on provider-supplied values. - Fixed potential stale values in `extra` by ensuring deep merge overwrite with `entry.transaction.save!`. * Replace hardcoded fallback transaction description with localized string * Refine pending flag logic in SimpleFin processor tests - Adjust test to prevent falsely inferring pending status from missing posted dates. - Ensure provider explicitly sets pending flag for transactions. * Add `has_many :holdings` association to `AccountProvider` with `dependent: :nullify` --------- Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com> |
||
|
LPW
|
e9dbf5f4e7 |
Fix Broken Account Re-linking Feature (#469)
* Update SimpleFIN relinking flow and enhance duplicate account handling - Updated logic to allow relinking of SimpleFIN accounts while preserving legacy mappings. - Introduced clean-up logic to hide orphaned duplicate accounts after relinking. - Enhanced UI to display current mappings for linked accounts. - Improved test coverage for relinking scenarios and SimpleFIN account visibility. * Localize SimpleFIN account selection messages and remove hardcoded text - Added translations for user-facing messages in `select_existing_account` flow (`pt-BR` and `en` locales). - Replaced hardcoded strings in the view with localized keys. * Localize Enable Banking and SimpleFIN account linking messages; add support for investment accounts. - Added translations for Enable Banking and SimpleFIN account linking flows. - Updated views and controllers to replace hardcoded strings with localized keys. - Introduced support for investment accounts in `Provider::LunchflowAdapter`. - Enhanced relinking logic for SimpleFIN accounts and improved test coverage for related scenarios. --------- Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com> |
||
|
Matthew Kilpatrick
|
ff43e71dec |
Fix Yahoo minor unit handling (#335)
Some security prices get returned in non-standard minor units (eg. `GBp` for Great British Pence, instead of `GBP` for Great British Pounds), leading to incorrect handling of the returned price data. This commit adds conversion logic for the two currencies I've been able to find examples of this affecting. |
||
|
Juan José Mata
|
e065c98396 |
Revert "Make bin/rails test pass in Dev Container" to avoid side-effects (#302)
* Revert "Add dummy PLAID_CLIENT_ID and PLAID_SECRET to env (#165)"
This reverts commit
|
||
|
soky srm
|
96713ee8b4 |
Add support for dynamic config UI (#256)
* Add support for dynamic config UI
* Add support for section description
* Better dynamic class settings
Added dynamic_fields hash field - Stores all undeclared settings
[] method - Checks declared fields first, then falls back to dynamic hash
[]= method - Updates declared fields normally, stores others in hash
No runtime field declaration - Fields are never dynamically created on the class
* FIX proper lookup for provider keys
- Also validate configurable values properly.
- Change Provider factory to use Rails autoloading (Zeitwerk)
* Fix factory
The derive_adapter_name method relies on string manipulation ("PlaidAccount".sub(/Account$/, "") + "Adapter" → "PlaidAdapter"), but we already have explicit registration in place.
* Make updates atomic, field-aware, and handle blanks explicitly
* Small UX detail
* Add support for PlaidEU in UI also
- This looks like partial support atm
|
||
|
João Felipe
|
9fefe57de5 |
Feature/yahoo finance (#123)
* Implement Yahoo Finance * Added tests * Updated hosting controller to check for managed app_mode instead of env_override * Suggestions from CodeRabbit and Fixes on tests * Remove Css changes * Fix yahoo finance impl and i18n * Updated view to use healthy method * remove usage * Updated env example * keep usage on class just to keep same format * Ci test * Remove some useless validations * Remove logs * Linter fixes * Broke this in my conflict merge * Wrong indentation level --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
soky srm
|
4fb0a3856e |
Providers factory (#250)
* Implement providers factory * Multiple providers sync support - Proper Multi-Provider Syncing: When you click sync on an account with multiple providers (e.g., both Plaid and SimpleFin), all provider items are synced - Better API: The existing account.providers method already returns all providers, and account.provider returns the first one for backward compatibility - Correct Holdings Deletion Logic: Holdings can only be deleted if ALL providers allow it, preventing accidental deletions that would be recreated on next sync TODO: validate this is the way we want to go? We would need to check holdings belong to which account, and then check provider allows deletion. More complex - Database Constraints: The existing validations ensure an account can have at most one provider of each type (one PlaidAccount, one SimplefinAccount, etc.) * Add generic provider_import_adapter * Finish unified import strategy * Update app/models/plaid_account.rb Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: soky srm <sokysrm@gmail.com> * Update app/models/provider/factory.rb Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Signed-off-by: soky srm <sokysrm@gmail.com> * Fix account linked by plaid_id instead of external_id * Parse numerics to BigDecimal Parse numerics to BigDecimal before computing amount; guard nils. Avoid String * String and float drift; also normalize date. * Fix incorrect usage of assert_raises. * Fix linter * Fix processor test. * Update current_balance_manager.rb * Test fixes * Fix plaid linked account test * Add support for holding per account_provider * Fix proper account access Also fix account deletion for simpefin too * FIX match tests for consistency * Some more factory updates * Fix account schema for multipe providers Can do: - Account #1 → PlaidAccount + SimplefinAccount (multiple different providers) - Account #2 → PlaidAccount only - Account #3 → SimplefinAccount only Cannot do: - Account #1 → PlaidAccount + PlaidAccount (duplicate provider type) - PlaidAccount #123 → Account #1 + Account #2 (provider linked to multiple accounts) * Fix account setup - An account CAN have multiple providers (the schema shows account_providers with unique index on [account_id, provider_type]) - Each provider should maintain its own separate entries - We should NOT update one provider's entry when another provider syncs * Fix linter and guard migration * FIX linter issues. * Fixes - Remove duplicated index - Pass account_provider_id - Guard holdings call to avoid NoMethodError * Update schema and provider import fix * Plaid doesn't allow holdings deletion * Use ClimateControl for proper env setup * No need for this in .git --------- Signed-off-by: soky srm <sokysrm@gmail.com> Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
Copilot
|
a8f318c3f9 |
Fix "Messages is invalid" error for Ollama/custom LLM providers and add comprehensive AI documentation (#225)
* Add comprehensive AI/LLM configuration documentation * Fix Chat.start! to use default model when model is nil or empty * Ensure all controllers use Chat.default_model for consistency * Move AI doc inside `hosting/` * Probably too much error handling --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
Juan José Mata
|
54bc37a651 |
Remove Synth Finance references (#47)
* Remove Synth Finance integration * Linter noise * Fix failing (old) test, use it for Twelve Data --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
Josh Pigford
|
4f5068e7e5 |
feat(assistant): improve chat functionality and update tests - refactor configurable model, update OpenAI provider, enhance chat form UI, and improve test coverage (#2316)
Updated model to GPT 4.1 |
||
|
Zach Gollwitzer
|
03a146222d |
Plaid sync domain improvements (#2267)
Breaks our Plaid sync process out into more manageable classes. Notably, this moves the sync process to a distinct, 2-step flow: 1. Import stage - we first make API calls and import Plaid data to "mirror" tables 2. Processing stage - read the raw data, apply business rules, build internal domain models and sync balances This provides several benefits: - Plaid syncs can now be "replayed" without fetching API data again - Mirror tables provide better audit and debugging capabilities - Eliminates the "all or nothing" sync behavior that is currently in place, which is brittle |
||
|
Zach Gollwitzer
|
5da4bb6dc3 |
Subscription tests and domain (#2209)
* Save work * Subscriptions and trials domain * Store family ID on customer * Remove indirection of stripe calls * Test simplifications * Update brakeman * Fix stripe tests in CI * Update billing page to show subscription details * Remove legacy columns * Complete billing settings page * Fix hardcoded plan name * Handle subscriptions for self hosting mode * Lint fixes |
||
|
Zach Gollwitzer
|
a51c4d2cba |
New onboarding, trials, Stripe integration (#2185)
* New onboarding, trials, Stripe integration * Fix tests * Lint fixes * Fix subscription endpoints |
||
|
Zach Gollwitzer
|
297a695d0f |
Transaction rules engine V1 (#1900)
* Domain model sketch * Scaffold out rules domain * Migrations * Remove existing data enrichment for clean slate * Sketch out business logic and basic tests * Simplify rule scope building and action executions * Get generator working again * Basic implementation + tests * Remove manual merchant management (rules will replace) * Revert "Remove manual merchant management (rules will replace)" This reverts commit 83dcbd9ff0aa7bbee211796b71aa48b71df5e57e. * Family and Provider merchants model * Fix brakeman warnings * Fix notification loader * Update notification position * Add Rule action and condition registries * Rule form with compound conditions and tests * Split out notification types, add CTA type * Rules form builder and Stimulus controller * Clean up rule registry domain * Clean up rules stimulus controller * CTA message for rule when user changes transaction category * Fix tests * Lint updates * Centralize notifications in Notifiable concern * Implement category rule prompts with auto backoff and option to disable * Fix layout bug caused by merge conflict * Initialize rule with correct action for category CTA * Add rule deletions, get rules working * Complete dynamic rule form, split Stimulus controllers by resource * Fix failing tests * Change test password to avoid chromium conflicts * Update integration tests * Centralize all test password references * Add re-apply rule action * Rule confirm modal * Run migrations * Trigger rule notification after inline category updates * Clean up rule styles * Basic attribute locking for rules * Apply attribute locks on user edits * Log data enrichments, only apply rules to unlocked attributes * Fix merge errors * Additional merge conflict fixes * Form UI improvements, ignore attribute locks on manual rule application * Batch AI auto-categorization of transactions * Auto merchant detection, ai enrichment in batches * Fix Plaid merchant assignments * Plaid category matching * Cleanup 1 * Test cleanup * Remove stale route * Fix desktop chat UI issues * Fix mobile nav styling issues |
||
|
Zach Gollwitzer
|
5cf758bd03 |
improvements(ai): Improve AI streaming UI/UX interactions + better separation of AI provider responsibilities (#2039)
* Start refactor * Interface updates * Rework Assistant, Provider, and tests for better domain boundaries * Consolidate and simplify OpenAI provider and provider concepts * Clean up assistant streaming * Improve assistant message orchestration logic * Clean up "thinking" UI interactions * Remove stale class * Regenerate VCR test responses |
||
|
Zach Gollwitzer
|
2f6b11c18f |
Personal finance AI (v1) (#2022)
* AI sidebar * Add chat and message models with associations * Implement AI chat functionality with sidebar and messaging system - Add chat and messages controllers - Create chat and message views - Implement chat-related routes - Add message broadcasting and user interactions - Update application layout to support chat sidebar - Enhance user model with initials method * Refactor AI sidebar with enhanced chat menu and interactions - Update sidebar layout with dynamic width and improved responsiveness - Add new chat menu Stimulus controller for toggling between chat and chat list views - Improve chat list display with recent chats and empty state - Extract AI avatar to a partial for reusability - Enhance message display and interaction styling - Add more contextual buttons and interaction hints * Improve chat scroll behavior and message styling - Refactor chat scroll functionality with Stimulus controller - Optimize message scrolling in chat views - Update message styling for better visual hierarchy - Enhance chat container layout with flex and auto-scroll - Simplify message rendering across different chat views * Extract AI avatar to a shared partial for consistent styling - Refactor AI avatar rendering across chat views - Replace hardcoded avatar markup with a reusable partial - Simplify avatar display in chats and messages views * Update sidebar controller to handle right panel width dynamically - Add conditional width class for right sidebar panel - Ensure consistent sidebar toggle behavior for both left and right panels - Use specific width class for right panel (w-[375px]) * Refactor chat form and AI greeting with flexible partials - Extract message form to a reusable partial with dynamic context support - Create flexible AI greeting partial for consistent welcome messages - Simplify chat and sidebar views by leveraging new partials - Add support for different form scenarios (chat, new chat, sidebar) - Improve code modularity and reduce duplication * Add chat clearing functionality with dynamic menu options - Implement clear chat action in ChatsController - Add clear chat route to support clearing messages - Update AI sidebar with dropdown menu for chat actions - Preserve system message when clearing chat - Enhance chat interaction with new menu options * Add frontmatter to project structure documentation - Create initial frontmatter for structure.mdc file - Include description and configuration options - Prepare for potential dynamic documentation rendering * Update general project rules with additional guidelines - Add rule for using `Current.family` instead of `current_family` - Include new guidelines for testing, API routes, and solution approach - Expand project-specific rules for more consistent development practices * Add OpenAI gem and AI-friendly data representations - Add `ruby-openai` gem for AI integration - Implement `to_ai_readable_hash` methods in BalanceSheet and IncomeStatement - Include Promptable module in both models - Add savings rate calculation method in IncomeStatement - Prepare financial models for AI-powered insights and interactions * Enhance AI Financial Assistant with Advanced Querying and Debugging Capabilities - Implement comprehensive AI financial query system with function-based interactions - Add detailed debug logging for AI responses and function calls - Extend BalanceSheet and IncomeStatement models with AI-friendly methods - Create robust error handling and fallback mechanisms for AI queries - Update chat and message views to support debug mode and enhanced rendering - Add AI query routes and initial test coverage for financial assistant * Refactor AI sidebar and chat layout with improved structure and comments - Remove inline AI chat from application layout - Enhance AI sidebar with more semantic HTML structure - Add descriptive comments to clarify different sections of chat view - Improve flex layout and scrolling behavior in chat messages container - Optimize message rendering with more explicit class names and structure * Add Markdown rendering support for AI chat messages - Implement `markdown` helper method in ApplicationHelper using Redcarpet - Update message view to render AI messages with Markdown formatting - Add comprehensive Markdown rendering options (tables, code blocks, links) - Enhance AI Financial Assistant prompt to encourage Markdown usage - Remove commented Markdown CSS in Tailwind application stylesheet * Missing comma * Enhance AI response processing with chat history context * Improve AI debug logging with payload size limits and internal message flag * Enhance AI chat interaction with improved thinking indicator and scrolling behavior * Add AI consent and enable/disable functionality for AI chat * Upgrade Biome and refactor JavaScript template literals - Update @biomejs/biome to latest version with caret (^) notation - Refactor AI query and chat controllers to use template literals - Standardize npm scripts formatting in package.json * Add beta testing usage note to AI consent modal * Update test fixtures and configurations for AI chat functionality - Add family association to chat fixtures and tests - Set consistent password digest for test users - Enable AI for test users - Add OpenAI access token for test environment - Update chat and user model tests to include family context * Simplify data model and get tests passing * Remove structure.mdc from version control * Integrate AI chat styles into existing prose pattern * Match Figma design spec, implement Turbo frames and actions for chats controller * AI rules refresh * Consolidate Stimulus controllers, thinking state, controllers, and views * Naming, domain alignment * Reset migrations * Improve data model to support tool calls and message types * Tool calling tests and fixtures * Tool call implementation and test * Get assistant test working again * Test updates * Process tool calls within provider * Chat UI back to working state again * Remove stale code * Tests passing * Update openai class naming to avoid conflicts * Reconfigure test env * Rebuild gemfile * Fix naming conflicts for ChatResponse * Message styles * Use OpenAI conversation state management * Assistant function base implementation * Add back thinking messages, clean up error handling for chat * Fix sync error when security price has bad data from provider * Add balance sheet function to assistant * Add better function calling error visibility * Add income statement function * Simplify and clean up "thinking" interactions with Turbo frames * Remove stale data definitions from functions * Ensure VCR fixtures working with latest code * basic stream implementation * Get streaming working * Make AI sidebar wider when left sidebar is collapsed * Get tests working with streaming responses * Centralize provider error handling * Provider data boundaries --------- Co-authored-by: Josh Pigford <josh@joshpigford.com> |
||
|
Zach Gollwitzer
|
19cc63c8f4 |
Use Redis for ActiveJob and ActionCable (#2004)
* Use Redis for ActiveJob and ActionCable * Fix alwaysApply setting * Update queue names and weights * Tweak weights * Update job queues * Update docker setup guide * Remove deprecated upgrade columns from users table * Refactor Redis configuration for Sidekiq and caching in production environment * Add Sidekiq Sentry monitoring * queue naming fix * Clean up schema |
||
|
Zach Gollwitzer
|
f65b93a352 |
Data provider simplification, tests, and documentation (#1997)
* Ignore env.test from source control * Simplification of providers interface * Synth tests * Update money to use new find rates method * Remove unused issues code * Additional issue feature removals * Update price data fetching and tests * Update documentation for providers * Security test fixes * Fix self host test * Update synth usage data access * Remove AI pr schema changes |
||
|
Josh Pigford
|
490f44589e |
First pass at security price reference (#1388)
* First pass at security price reference
* Data cleanup
* Synth security fetching does better with a mic_code
* Update test suite
😭
* Update schema.rb
* Update generator.rb
|
||
|
Tony Vincent
|
6fa40e0fa2 |
Fetch exchange rates in bulk from synth (#1069)
* Fetch exchnage rates in bulk * Handle paginated response * Rename method and improve tests * Change argument names * Use standard date format |
||
|
Zach Gollwitzer
|
6e74414cb2 | Add source headers to Synth calls (#1062) | ||
|
Zach Gollwitzer
|
453a54e5e6 |
Add security prices provider (Synth integration) (#1039)
* User tickers as primary lookup symbol instead of isin * Add security price provider * Fetch security prices in bulk to improve sync performance * Fetch prices in bulk, better mocking for tests |
||
|
Zach Gollwitzer
|
6767aaed1d |
Handle missing exchange rate provider, allow fallback for missing rates (#955)
* Clean up exchange rate logic * Remove stale method |
||
|
Zach Gollwitzer
|
5aca2ff9b6 |
Add zero-config self hosting on Render (#612)
* v1 of backend implementation for self hosting * Add docs * Add upgrades controller * Add global helpers for self hosting mode * Add self host settings controller * Conditionally show self hosting settings * Environment and config updates * Complete upgrade prompting flow * Update config for forked repo * Move configuration of github provider within class * Add upgrades cron * Update deploy button * Update guides * Fix render deployer * Typo * Enable auto upgrades * Fix cron * Make upgrade modes more clear and consistent * Trigger new available version * Fix logic for displaying upgrade prompts * Finish implementation * Fix regression * Trigger new version * Add i18n translations * trigger new version * reduce caching time for testing * Decrease cache for testing * trigger upgrade * trigger upgrade * Only trigger deploy once * trigger upgrade * If target is commit, always upgrade if any upgrade is available * trigger upgrade * trigger upgrade * Test release * Change back to maybe repo for defaults * Fix lint errors * Clearer naming * Fix relative link * Add abs path * Relative link * Update docs |