* Add RentCast and Realie AVM providers for property valuation
Adds Automated Valuation Model (AVM) provider support so self-hosted
users can create property accounts from a US address lookup instead of
entering details manually:
- Provider::Rentcast and Provider::Realie clients, each fetching the
property record (type, year built, square footage) and value estimate
in a single API request, registered under a new :property_valuations
registry concept
- API key fields (encrypted, ENV-overridable) with monthly usage display
in Settings > Self-Hosting under "Property Valuation Providers"
- New property flow: when a provider key is configured, the method
selector offers "Add via RentCast/Realie" alongside manual entry;
the lookup form reuses the manual flow's localized address fields
and creates the account active with fetched attributes, valuation as
balance, and the address saved
- SyncPropertyValuationsJob refreshes linked property valuations once a
day (never hourly) via config/schedule.yml; each provider enforces its
monthly request cap (RentCast 50, Realie 25) with a calendar-month
counter shared between creation lookups and refreshes
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Allow ENV override of AVM provider monthly request caps
The RentCast (50/month) and Realie (25/month) budgets default to the
free tier limits but can now be raised for paid plans via
RENTCAST_MAX_REQUESTS_PER_MONTH / REALIE_MAX_REQUESTS_PER_MONTH,
mirroring the AlphaVantage and Tiingo request limit overrides. The
settings descriptions and usage display reflect the effective cap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address review feedback: durable request counters, safer refresh job
- Move monthly AVM request counters from Rails.cache to a new
provider_request_counts table with atomic upsert increments, so the
hard budget caps survive cache eviction, restarts, and Redis flushes
- SyncPropertyValuationsJob: only mark a property synced when the
balance update succeeds (wrapped in a transaction), process stalest
valuations first so a tight budget goes where it matters, and report
failures via DebugLogEntry.capture instead of Rails.logger
- Realie: reject lookups whose returned city/ZIP contradict the entered
address (street+state queries can match the wrong city); document that
numeric use codes are unpublished and leave the subtype unset
- Add Faraday open/request timeouts to both provider clients
- Localize all user-facing provider error messages
(config/locales/models/provider/en.yml)
- Add a check constraint restricting properties.avm_provider to known
providers
- Use the min-h-80 scale token instead of min-h-[320px]
- Tests: exact-argument expectations on the lookup stub, provider stubs
in the no-provider test, RentCast/Realie API key update tests,
ProviderRequestCount unit tests, failed-balance regression test
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Stub AVM provider registry lookups in settings system test
Settings::HostingsController#show now resolves the RentCast and Realie
providers for usage display; the system test's partial get_provider
stubbing treats the new lookups as unexpected invocations without these.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Address second-round review: candidate scan, provider reuse, shared row partial
- Realie: when the address lookup returns multiple candidates, pick the
first one consistent with the entered city/ZIP instead of judging only
the first array element; the location-mismatch error now only fires
when no candidate matches
- SyncPropertyValuationsJob: resolve one provider instance per key for
the whole run, so the per-instance request throttle actually spaces
requests across properties instead of resetting on each iteration
- Extract the AVM method selector row into a shared partial so the
manual and provider options render one shape
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add retry middleware and sync index from review feedback
- Both AVM clients now retry transient connection failures (same
Faraday retry config as Tiingo) so a network blip doesn't burn one of
the tight monthly budget's requests
- Partial index on properties(avm_provider, avm_last_synced_on) backing
the daily sync job's filter and stalest-first ordering
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Key the AVM sync index on avm_last_synced_on
The daily job orders by avm_last_synced_on ASC NULLS FIRST across all
AVM-linked properties, so leading the partial index with avm_provider
prevented it from serving the sort. Rekeyed on avm_last_synced_on with
matching null ordering; the partial predicate still covers the filter.
Amended in place since the migration is unmerged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Map Realie numeric use codes; skip incomplete addresses in sync job
- Realie documents its numeric use codes at
docs.realie.ai/api-reference/feature-key (earlier skip reason was
wrong — the table exists). Map the residential/land/agricultural codes
to Property subtypes, leaving codes without a subtype equivalent
(e.g. 1006 mobile/manufactured) unset like the RentCast mapping.
- The daily job now skips properties whose address is missing a street
or state before resolving a provider, logging a warn-level
DebugLogEntry, so a malformed address can't burn a monthly-budget
request every day.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Guard currency mismatches in refresh; reset AVM keys in test teardown
- The daily job now skips (with a warn-level DebugLogEntry) properties
whose account currency no longer matches the provider's valuation
currency, checked via the concept's valuation_currency before spending
a request — writing a USD valuation into a re-currencied account would
corrupt the balance
- Hostings controller test teardown now clears rentcast_api_key and
realie_api_key alongside the other cached global settings
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Validate AVM lookup inputs before spending a provider request
The form marks name and address fields required, but a forged or
JS-less submission bypasses that and would burn one of the tight
monthly-budget requests on a lookup that can't produce a property.
Property::AvmImport now validates name and the full address locally
(localized error) before calling the provider.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Add preview-and-confirm step to the AVM property lookup
RentCast's AVM endpoint is location-based: a plausible but nonexistent
address still geocodes and returns an area-derived estimate with no
property record behind it. Instead of silently creating an account from
that, the lookup now shows what the provider returned — type, year
built, area (each "Unknown" when missing), and the estimated value —
with an explicit notice when no property record was found, and only the
user's confirmation creates the account.
The confirm step reuses the fetched data via the form (sanitized
server-side), so the flow still costs exactly one provider request.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Require a complete US address before daily valuation refresh
The job's guard only required street and state, but a later-blanked
city or ZIP silently disables the Realie wrong-city check (blank
entered fields count as "no mismatch"), so a refresh could accept
another property's valuation. The guard now mirrors the import-time
completeness check and also skips addresses edited to a non-US country.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Sign the AVM preview payload; blur valuation in privacy mode
- The confirm step now rebuilds everything from a signed, 1-hour
message-verifier token generated at lookup time, proving the lookup
(and its counted provider request) actually ran — a direct confirm
POST with fabricated data can no longer create provider-linked
properties that the daily refresh job would then spend quota on.
Forged/expired tokens re-render the lookup form with a clear error.
- The preview's market value now carries the privacy-sensitive class so
privacy mode blurs it like other account amounts.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Treat zero Realie valuations as absent; bind preview token to family
Review follow-ups from PR #2727:
- Realie returns modelValue: 0 when it can't produce an AVM estimate;
zero now falls back to the assessed market value instead of syncing
a $0 balance, and an all-zero record errors out.
- Monetary values now parse via BigDecimal(value.to_s) rather than
Float#to_d, avoiding binary float rounding artifacts in balances.
- The signed AVM preview token's purpose is scoped to the family that
ran the lookup, so a leaked token can't be replayed cross-family.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Refresh AVM property valuations monthly instead of daily
Per @jjmata's review on #2727: valuations change slowly and providers
enforce tight monthly request caps, so a daily cron adds no value.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Prevent Yahoo crumb cache poisoning
* Make Yahoo Finance health status rate-limit aware
* Normalize Yahoo Colombian listings
Map Yahoo's BVC code to XBOG so search results derive Colombia from canonical exchange metadata and price requests use the .CL suffix with a COP fallback.
Spec: .scratch/normalize-yahoo-colombia-listings/spec.md
* Move Yahoo health checks to background
* feat(exchange-rates): add Frankfurter as an exchange-rate provider
Frankfurter (frankfurter.dev) is a free, keyless FX rates API backed by
ECB daily reference rates, with no published rate limit and no auth
flow to maintain (unlike Yahoo Finance's reverse-engineered cookie/
crumb auth or TwelveData's fast-exhausting free tier).
Follows the Provider::MoexPublic template: Faraday client with retry
middleware, SslConfigurable for self-hosted CA support, a light
RateLimitable throttle, and a FRANKFURTER_URL env escape hatch for
self-hosters. Registered as exchange-rates-only (no security/stock
data) and added to the hosting settings dropdown.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* refactor(provider): switch Frankfurter to the v2 API
v1 is explicitly marked "frozen" on Frankfurter's own root endpoint; v2
is "current" and covers 201 currencies across 84 central banks vs v1's
~30 ECB-only. Confirmed via the v2 OpenAPI spec and live requests:
- Single-date lookups now use GET /rate/{base}/{quote}?date=..., which
carries weekends/holidays forward server-side (a Saturday returns a
real rate directly), so the provider no longer needs its own
lookback-window logic.
- Range lookups now use GET /rates?base=..."es=...&from=...&to=...,
a flat array of { date, base, quote, rate } records (v2's shape)
instead of v1's { "rates": { date => currencies } } hash.
- Every calendar day in a range is present (v2 gapfills itself), rather
than v1's omit-non-trading-days behavior.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(provider): sanitize currency codes before URL path interpolation
from/to were only upcased before being interpolated directly into the
URL path in fetch_exchange_rate (GET /rate/{from}/{to}). Low risk since
currency codes come from validated internal sources, but adds cheap
defense-in-depth: strip anything that isn't A-Z, matching the ISO 4217
format real currency codes always take.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
The securities-provider checkboxes used raw Tailwind utilities
(rounded border-primary text-primary focus:ring-primary) instead of the
design-system .checkbox component. In dark mode text-primary resolves to
white, so a checked box rendered a white check on a white fill and the
checkmark was invisible.
Switch to the theme-aware .checkbox checkbox--light classes used by every
other checkbox in the app (settings/preferences, transaction filters,
etc.), which render a dark check on a light fill in dark mode.
Adds Provider::TinkoffInvest, a token-based securities provider built on the
public T-Invest REST gateway (invest-public-api.tinkoff.ru/rest). It serves
prices for Russian instruments (shares, ETF/БПИФ, bonds) and, crucially, brand
logos via the T-Invest CDN — the authoritative logo source for MOEX
instruments, which ISS (MoexPublic) does not provide.
- Registry: register `tinkoff_invest` under the :securities concept; token via
ENV TINKOFF_INVEST_API_KEY or encrypted Setting.tinkoff_invest_api_key.
- Logos independent of the price provider: Security#import_brand_logo consults
T-Invest for a logo whenever a token is configured (after the price-provider
metadata fetch, so it never short-circuits website_url backfill). Gated on
token presence, not the securities checklist.
- display_logo_url: with no website domain, a stored provider logo (T-Invest)
now beats the ticker-only Brandfetch lettermark; when a domain exists,
Brandfetch still wins (unchanged).
- MoexPublic no longer reports moex.com as the issuer website — it's the
exchange, not the issuer, and would make Brandfetch render the exchange logo
for every instrument and shadow the real brand logo.
- Prices: GetCandles (daily, paged) + GetLastPrices; Quotation units+nano/1e9;
bonds priced as percent-of-par x nominal (missing nominal raises, not 0).
- Settings: encrypted token field (always shown) + provider checkbox + en locale.
- Tests for search/info/logo-url/prices/bond/incomplete-candle and display logic.
Co-authored-by: Claude <noreply@anthropic.com>
* feat(prices): add Moscow Exchange (MOEX ISS) securities + FX provider
Add Provider::MoexPublic, a keyless provider built on the free MOEX ISS API
(https://iss.moex.com/iss), modeled on Provider::BinancePublic.
Securities: shares, funds/ETF/БПИФ (e.g. LQDT), and bonds (OFZ + corporate).
Bonds are priced clean — LAST% × FACEVALUE / 100 in the instrument currency,
with per-row FACEVALUE for amortizing issues; NKD/accrued coupon excluded.
Exchange rates: also implements ExchangeRateConcept for RUB↔{USD,EUR,CNY} via
selt TOM instruments (USD000UTSTOM/EUR_RUB__TOM/CNYRUB_TOM); the selt quote is
X/RUB, inverted for RUB→X, nil for non-RUB-crossed pairs.
Details:
- Board/engine resolution via the ISS primary-board flag with a hardcoded
priority fallback (TQBR, TQTF, TQOB, TQCB, …).
- Instrument currency from CURRENCYID/FACEUNIT (handles USD/CNY eurobonds &
FX funds), normalizing legacy SUR/RUR → RUB; default RUB.
- Full history via from/till + start= pagination; current price fallback chain
LAST → MARKETPRICE → LCURRENTPRICE → LCLOSEPRICE → PREVPRICE → latest history
close.
- Bare SECID identity, exchange_operating_mic=MISX, country_code=nil (wildcard
like Binance); search accepts .ME/.MOEX/.MISX/.MCX aliases and ISIN.
- RateLimitable throttling, SslConfigurable, Faraday retry/timeouts; all public
methods wrapped in with_provider_response.
Wired into Provider::Registry for both :securities and :exchange_rates, the
hosting provider-selection UI, locales, and config/exchanges.yml (MISX).
Docker-tested (devcontainer, Ruby 3.4.9): 29 new tests green, full provider
suite + i18n green, rubocop clean; smoke-tested against live ISS (SBER price,
OFZ clean price, USD/RUB FX).
* fix(moex): address review — FX weekend lookback, dead branch, translated hints
- fetch_exchange_rate now fetches a 10-day lookback window (not just the exact
day) so a weekend/holiday request resolves to the prior trading day's close,
matching Yahoo's behavior (Codex P2).
- Remove dead identical if/else branches in history_row_price (CodeRabbit).
- Translate moex_public_hint into ca/fr/hu/vi/zh-CN instead of English copy
(CodeRabbit).
- Add a test covering the FX prior-trading-day lookback.
* fix(moex): guard ISS date parsing; doc TQTE in board priority
Address maintainer review (jjmata):
- parse_iss_date wraps Date.parse so a malformed ISS TRADEDATE skips just that
row (with a contextual log warning) instead of failing the whole history/FX
fetch. Used in history_row_price and fx_history.
- Add TQTE to the BOARD_PRIORITY doc comment (it was in the constant but missing
from the comment).
- Add a test covering the unparseable-date skip.
settings/hostings/_sync_settings.html.erb rendered the 'configured via env'
notice with bg-warning-50 / border-warning-200 / text-warning-600 / text-warning-800
-- none of which exist as Tailwind utilities, so the box rendered fully unstyled
(no tint, no border, default text color). Replace the hand-built box with the
canonical DS::Alert(:warning), matching the warning-surface recipe and the
ds-notice-neutral-text convention.
* feat(ai): add Anthropic provider with chat parity (1/5)
Introduces Provider::Anthropic alongside Provider::Openai, implementing
the LlmConcept chat_response contract over the official anthropic Ruby
SDK. Batch ops, PDF, and RAG land in follow-up PRs.
- Provider::Anthropic uses Messages API for sync and streaming responses
- ChatConfig builds requests with ephemeral prompt-cache markers on the
system prompt and the last tool definition
- MessageFormatter reconstructs multi-turn history (text + tool_use +
tool_result blocks) from raw Message records, including the paired
user-role tool_result turn Anthropic requires after every tool_use
- ChatParser maps Anthropic Message into the shared ChatResponse Data
- Registry, Setting, User, Chat default model wired for ANTHROPIC_*
envs and Setting.anthropic_*; LLM_PROVIDER selects between providers
- Responder forwards raw conversation_history (Array<Message>) so
providers without hosted conversation state can rebuild context
- OpenAI provider accepts and ignores the new kwarg (no behavior change)
Tests cover provider init, model gating, MessageFormatter for all turn
shapes, ChatConfig request building (max_tokens, system cache, tool
conversion), ChatParser for text / tool_use / mixed blocks, Registry
discovery, and mocked chat_response success / error / function_request
paths. Live VCR cassettes recorded in a follow-up with a real key.
Stacked PRs: 2/5 batch ops + cost ledger, 3/5 PDF, 4/5 pgvector RAG,
5/5 settings UI + disclosure.
* fix(ai): address PR review on Anthropic provider foundation
Surface fixes raised by Codex + CodeRabbit on PR 1/5:
- Provider::Anthropic#chat_response now accepts (and ignores) a
`messages:` kwarg. Assistant::Responder passes both `messages:`
(OpenAI-shape) and `conversation_history:` (raw Message records) for
cross-provider parity, so the previous signature raised
ArgumentError on the first chat turn through the Anthropic provider.
- Provider::Anthropic#supports_model? bypasses the `claude` prefix
gate when a custom base_url is configured, mirroring the OpenAI
provider. Bedrock-shaped IDs like
`anthropic.claude-sonnet-4-5-20250929-v1:0` and
`claude-opus-4@20250514` are otherwise rejected by
Assistant::Provided#get_model_provider and the chat dies.
- Setting.anthropic_access_token is now in
EncryptedSettingFields::ENCRYPTED_FIELDS so the Anthropic API key
is encrypted at rest like every other provider secret. Previously
plaintext while siblings (openai_access_token, twelve_data_api_key,
external_assistant_token) were ciphertext.
- Chat.default_model falls back to whichever provider is actually
configured. Previously, with LLM_PROVIDER=anthropic but no
Anthropic credentials, the default model resolved to a Claude ID
that no registered provider supported, so chats failed even when
OpenAI was fully configured. Adds Provider::{Anthropic,Openai}#configured?
class methods for the readable callsite.
- Provider::Anthropic.effective_model uses
`ENV["ANTHROPIC_MODEL"].presence || Setting.anthropic_model` so the
Setting lookup is only performed when the env var is absent — the
previous `ENV.fetch(KEY, default)` evaluated the default arg
eagerly on every call.
- Provider::Anthropic::ChatConfig#anthropic_input_schema strips both
`:strict` and `"strict"` keys so JSON-decoded schemas with string
keys cannot leak the OpenAI-only flag through to Anthropic.
Test coverage added: supports_model? bypass on custom endpoints,
chat_response messages: kwarg compatibility, default_model fallback
in the three credential combinations, configured? against ENV +
Setting, strict-flag stripping for both key types, and a
`Setting.expects(:anthropic_model).never` assertion proving the
ENV-precedence test now exercises the lazy path.
All 4365 tests pass (1 pre-existing libvips env error unrelated).
* test(chat): make default_model tests resilient to ENV model overrides
CodeRabbit flagged on PR review: the new default_model tests asserted
against Provider::*::DEFAULT_MODEL, but Chat.default_model actually
returns Provider::*.effective_model.presence (which reads
OPENAI_MODEL / ANTHROPIC_MODEL from the environment). With either env
var set, the tests would fail intermittently even though routing was
correct.
- New default_model tests now assert against the provider's
effective_model directly, so they verify the routing decision
(which provider's value wins) without coupling to the constant.
- Pre-existing "creates with default model" assertions had the same
brittleness; switch them to compare against Chat.default_model so
the chosen model is whatever the env / Setting cascade resolves to.
Verified by running `ANTHROPIC_MODEL=claude-haiku-4-5 OPENAI_MODEL=gpt-4o
bin/rails test test/models/chat_test.rb` — 16 runs, 0 failures
(previously 2 pre-existing failures + 0 from the new tests).
* fix(ai): address local review on Anthropic foundation
- Provider::Anthropic#supports_pdf_processing? bypasses prefix gate for
custom endpoints, mirroring supports_model?
- Provider::Anthropic#initialize raises Error when custom_endpoint? AND
model.blank?, parity with Provider::Openai
- stream_chat_response captures partial usage on mid-stream errors and
records it via the new on_partial callback so chat_response can skip
the duplicate error row in the outer rescue
- safe_accumulated_message swallows the secondary failure when the SDK
cannot reconstruct a snapshot
- langfuse_client memoizes properly (||= instead of =) so repeated calls
don't churn Langfuse instances
- MessageFormatter sorts tool_calls by created_at then id so the
message array is deterministic across replays; skips tool_calls
missing both provider_call_id and provider_id rather than sending
`id: nil` and getting rejected by Anthropic
- Setting.anthropic_access_token default falls back through
ENV["ANTHROPIC_API_KEY"].presence (was missing .presence, so an
empty-string env value bled through)
- User#openai_configured? / #anthropic_configured? delegate to the
Provider::* class methods — single source of truth
- Assistant::Responder renames the OpenAI-shape history builder
conversation_history → openai_messages_payload so the kwarg name
matches the local method name (messages: openai_messages_payload,
conversation_history: chat_message_records)
- Assistant::Builtin stale-history comment updated to reference both
builders
Adds a streaming chat_response test using ad-hoc subclasses of the
SDK event types so the case/when dispatch matches via is_a? without
stubbing class-level === behavior.
* test(ai): add Anthropic tool_use round-trip + multi-tool turn coverage
Addresses @jjmata's "worth confirming" note on PR #1983: tool-use turns
from prior assistant messages must round-trip correctly when retrieved
from the database.
- New `ChatParser → ToolCall::Function → MessageFormatter` test walks
the full path: Anthropic response with a tool_use block →
ChatFunctionRequest → ToolCall::Function.from_function_request →
persisted on the AssistantMessage → MessageFormatter rebuild on the
next turn. Asserts the original `tool_use.id` is preserved end-to-end
as both `tool_use.id` and the paired `tool_result.tool_use_id`, and
that the original `input` hash and serialized result content survive.
- New multi-tool assistant turn test confirms two tool_use blocks on a
single assistant message render as two tool_use blocks followed by
two paired tool_result blocks in a single user-role follow-up,
matching Anthropic's required alternation.
Both tests exercise the existing PR1 code without behavior changes.
* test(ai): require "ostruct" explicitly in Anthropic provider tests
OpenStruct is moving out of Ruby's default load path (warning in 3.4+,
removed in 3.5+). Tests work today because ActiveSupport transitively
loads it, but that's incidental. Match the existing convention in
test/controllers/settings/hostings_controller_test.rb which explicitly
requires ostruct for the same reason.
* fix(ai): sanitize Langfuse warn logs, normalize tool_use.input, dedup history fetch
Addresses three open CodeRabbit findings on PR #1983.
- Provider::Anthropic Langfuse rescue branches no longer include
`e.full_message` in `Rails.logger.warn`. `full_message` bundles the
backtrace + cause chain and on some SDK error types includes the
serialized request/response payload (prompt, model output). Logs
now report `#{e.class}: #{e.message}` only. Three sites:
create_langfuse_trace, log_langfuse_generation, upsert_langfuse_trace.
Note: Provider::Openai has the same pattern (copy-pasted source) —
harmonization deferred to a follow-up cleanup PR; this commit fixes
only the Anthropic provider to keep PR scope tight.
- MessageFormatter#parse_arguments now coerces any non-Hash parsed
result to `{}`. Anthropic's Messages API requires `tool_use.input`
to be a JSON object (map); a stored ToolCall::Function record whose
arguments parse to a scalar, bool, or array (corrupt row, legacy
data, cross-provider bleed) would otherwise produce a payload the
API rejects. Normal flow stores Hash arguments end-to-end so the
fix is defensive — adds 2 tests covering scalar/array JSON strings
and non-String non-Hash inputs.
- Assistant::Responder dedups the chat-history fetch. The previous
layout fired two near-identical `chat.messages.where(...).includes(
:tool_calls).ordered` queries per LLM turn (one for the OpenAI-shape
payload, one for the raw-records kwarg). A new memoized
`complete_chat_messages` fetches once; `chat_message_records` filters
out the current message via `Array#reject`, `openai_messages_payload`
iterates the cached array unchanged. One SQL query per turn instead
of two. Memoization scope = single Responder instance (per LLM call),
so cache invalidation is not a concern.
All 4370 tests pass (1 pre-existing libvips env error unrelated).
Rubocop + brakeman clean.
* fix(ci): replace sk-ant- prefixed test placeholders
Pipelock secret scanner pattern-matches `sk-ant-*` as a real Anthropic
API key and fails the PR security-scan check. Test stubs and
ClimateControl env values used `sk-ant-test`, `sk-ant-from-setting`,
`sk-ant-x`, `sk-ant-y` as obvious placeholders, but the scanner does
not care about value entropy.
Switched to `fake-anthropic-key-*` / `fake-token-*` strings so the
scanner stops flagging them. No production code touched, no behavior
change — Provider::Anthropic still accepts any non-blank token.
* feat(ai): add Anthropic batch ops + LLM cost ledger (2/5)
Implements auto_categorize, auto_detect_merchants, and
enhance_provider_merchants on Provider::Anthropic via forced tool calls,
plus the cost-ledger plumbing they need.
- Provider::Anthropic::AutoCategorizer, AutoMerchantDetector,
ProviderMerchantEnhancer each define a single output tool whose
input_schema mirrors the desired output, then force the model to call
it via tool_choice: { type: "tool", name: ..., disable_parallel_tool_use: true }.
Anthropic guarantees the tool_use.input matches the schema, so there
is no JSON parsing fragility, no <think> tag stripping, and no
json_object/json_schema fallback ladders.
- Concerns::UsageRecorder mirrors the OpenAI sibling but persists
cache_creation_input_tokens / cache_read_input_tokens to dedicated
columns instead of metadata.
- Migration adds cache_creation_tokens, cache_read_tokens (nullable
integers) to llm_usages. OpenAI rows leave them null.
- LlmUsage::PRICING gains Claude 4.x rows (opus-4-7 $15/$75, sonnet-4-6
$3/$15, haiku-4-5 $1/$5 per MTok). infer_provider returns "anthropic"
for claude-* via the existing exact/prefix lookup.
- Provider::Anthropic#chat_response now persists cache columns directly
rather than stashing them in metadata.
- 25-transaction batch cap mirrors the OpenAI provider so the cost
ledger sees the same shape regardless of which provider ran a batch.
Tests cover the forced-tool-call path, null/None normalization,
case-insensitive merchant matching, the missing-tool_use error path,
and Anthropic-specific pricing + provider inference on LlmUsage.
Stacked on #1983 (PR 1/5). 3/5 PDF + vision next.
* fix(ai): attribute Bedrock model IDs to anthropic + clean nil enum
- LlmUsage.infer_provider now returns "anthropic" for Bedrock /
Vertex shaped IDs (anthropic.* and anthropic/*), so cost-ledger
filtering by provider stays correct even when no per-MTok rate is
stored. Previously these IDs fell through to the "openai" default.
- AutoCategorizer drops the redundant nil sentinel from the
category_name enum — the union type [string, null] already permits
null, and some JSON Schema validators reject nil literals inside
enum arrays.
* test(ai): require "ostruct" in Anthropic batch op tests
Same rationale as the PR1 ostruct fix — explicit require so the tests
don't depend on ActiveSupport's transitive load when Ruby 3.5+ removes
OpenStruct from the default load path.
* feat(ai): Anthropic native PDF processing (3/5)
Implements process_pdf and extract_bank_statement on Provider::Anthropic
using the native `document` content block — no rasterization, no text
pre-extraction.
- Provider::Anthropic::PdfProcessor classifies the document, summarizes
it, and extracts statement metadata via a forced report_document_analysis
tool whose input_schema mirrors the existing Provider::Openai output
(document_type from Import::DOCUMENT_TYPES, summary, extracted_data).
- Provider::Anthropic::BankStatementExtractor returns the same
{ transactions, period, account_holder, account_number, bank_name,
opening_balance, closing_balance } shape via report_bank_statement so
downstream pdf_import code is provider-agnostic.
- Both attach the PDF as
{ type: "document", source: { type: "base64", media_type: "application/pdf", data: <b64> } }
— Claude 3.5+ / 4.x accept this natively (up to 32MB / 100 pages).
No pdf-reader, no pdftoppm, no chunking for typical statements.
- supports_pdf_processing? (introduced in PR 1) already returns true for
claude-* models, gating process_pdf with a clear error otherwise.
- Cost ledger rows are persisted via the shared UsageRecorder concern,
including cache_creation/cache_read tokens.
Tests verify the document block shape, tool_choice forcing, normalized
document_type for unknown classifications, transaction normalization
(date / amount / reference → notes), and the missing-tool_use error
path. Blank pdf_content raises before any client call.
Stacked on #1984 (PR 2/5). 4/5 pgvector RAG next.
* fix(ai): guard PDF size + surface bank-statement truncation
- PdfProcessor and BankStatementExtractor raise upfront when
pdf_content.bytesize exceeds MAX_PDF_BYTES (32 MB, matching
Anthropic's hard limit). Previously a 100 MB PDF would be
base64-encoded (~133 MB) and packed into the JSON body before
the API rejected it — peak heap ~270 MB per Sidekiq worker.
- BankStatementExtractor inspects response.stop_reason; when the
model hit max_tokens it logs a warning and flags result[:truncated]
so downstream callers know the transaction list may be incomplete.
- ISO date pattern added to statement_period_start/end schema in
PdfProcessor so the model can't return "March 2026" — Anthropic
enforces the regex via the tool's input_schema.
Tests cover the size guard (raises before any client.messages call),
truncated-result flagging, and the warning log path.
* test(ai): require "ostruct" in Anthropic PDF tests
Match the explicit ostruct require added in PR1/PR2 — same Ruby 3.5+
load-path reason.
* feat(ai): default Anthropic installs to pgvector RAG (4/5)
The provider-agnostic vector store stack (VectorStore::Pgvector + the
Embeddable concern) already shipped to main. This PR closes the
Anthropic loop:
- VectorStore::Registry.adapter_name now returns :pgvector when
Setting.llm_provider == "anthropic" and no explicit
VECTOR_STORE_PROVIDER override is set. Anthropic has no hosted vector
store, so falling back to the local pgvector adapter is the only
correct default. Explicit VECTOR_STORE_PROVIDER still wins.
- SearchFamilyFiles surfaces a longer message when no adapter is wired
up — calling out pgvector + EMBEDDING_URI_BASE as the supported
Anthropic-only path so the user is not stuck with an "OpenAI required"
hint that is no longer accurate.
The Embeddable concern already pulls embeddings from
EMBEDDING_URI_BASE / EMBEDDING_ACCESS_TOKEN (with OpenAI as fallback),
so Anthropic installs point this at Voyage AI, a local Ollama instance,
or OpenAI embeddings — independent of the chat provider.
Tests cover the new default routing, the existing OpenAI default
staying intact, and explicit VECTOR_STORE_PROVIDER overriding the
Anthropic default.
Stacked on #1985 (PR 3/5). 5/5 settings UI + retention disclosure next.
* feat(ai): self-host settings UI for Anthropic provider (5/5)
Adds the Anthropic panel and the install-wide LLM provider selector to
the self-hosting settings page, plus a shared data-retention
disclosure that covers both OpenAI and Anthropic.
- New _llm_provider_selector partial: select for Setting.llm_provider
(openai | anthropic), respects the LLM_PROVIDER env var (disables the
control + shows the "configured through environment variables" hint
when set, mirroring the existing OpenAI panel behaviour), and renders
a compact data-handling block with one-line retention statements for
each provider.
- New _anthropic_settings partial mirrors _openai_settings exactly:
password-field for the API key with **** redaction, optional
base_url (for AWS Bedrock / GCP Vertex), optional default model. All
three fields disable when their ENV var is set.
- show.html.erb renders provider selector + OpenAI panel + Anthropic
panel under the same "General" section so users can configure either
(or both) without switching pages.
- Settings::HostingsController#update now permits and persists
anthropic_access_token (ignoring the **** placeholder, same pattern
as OpenAI), anthropic_base_url, anthropic_model, and llm_provider
(validated against %w[openai anthropic]). On Setting::ValidationError
the rescue branch preserves anthropic_base_url / anthropic_model
input so the form re-renders with the user's typed values intact —
parity with the issue #1824 fix for OpenAI.
- Locale keys added under settings.hostings.{llm_provider_selector,
anthropic_settings}.
Tests cover token update + placeholder redaction, base_url + model
update, llm_provider switch to anthropic, and rejection of unknown
provider values. The existing GET render test still passes, exercising
all three new partials.
Closes the 5/5 Anthropic series stacked on #1986.
* fix(ai): valid Tailwind token + base_url URL validation
- Data-handling block in _llm_provider_selector swaps the invalid
bg-surface-secondary token for bg-container-inset, matching the
inset-card pattern used elsewhere in sure-design-system/components.css.
bg-surface-secondary is not defined anywhere in the design system —
Tailwind treated it as a no-op, so the block rendered with no
background contrast.
- Settings::HostingsController validates anthropic_base_url as a
URI::HTTP (catches https too) and raises Setting::ValidationError
with a localized message when the input is not parseable.
Previously any string was persisted, surfacing as an opaque
connection error at request time instead of an immediate UX failure.
- Blank base_url now clears the setting (was already the case but
exercised explicitly in tests now).
* fix(ci): replace sk-ant- prefixed token in hostings controller test
Same pipelock secret-scan trigger as PR1 fix on registry/anthropic
tests. The sk-ant-* prefix is matched verbatim by the scanner
regardless of value entropy.
* fix(ai): provision pgvector table when it is the default store
#1986 makes pgvector the default vector store for Anthropic installs, but
CreateVectorStoreChunks only ran when VECTOR_STORE_PROVIDER=pgvector was set
explicitly — so a fresh Anthropic-only install migrated without the
vector_store_chunks table and failed on uploads/searches.
Add VectorStore::Registry.pgvector_effective? as the single source of truth
for "is pgvector active?" (explicit env OR the Anthropic default), and a new
idempotent migration that enables the extension + creates the table whenever
pgvector is effective and the table is missing — covering fresh and
already-migrated installs without drift. Addresses Codex P1.
* fix(ai): provision pgvector table for Anthropic-default installs
Migration gated on raw VECTOR_STORE_PROVIDER==pgvector, so an
Anthropic-default install (which selects pgvector implicitly via
Setting.llm_provider without setting VECTOR_STORE_PROVIDER) skipped
table creation and failed later on a missing vector_store_chunks
relation. Route through VectorStore::Registry.pgvector_effective? —
the single source of truth already shared by the adapter selection.
Addresses Codex P1 review finding.
* fix(ai): provision pgvector chunks table on schema-load installs
The ensure-migration only helps db:migrate upgraders. Fresh installs go
through bin/docker-entrypoint's db:prepare, which loads schema.rb (the
conditional table can't be dumped there — it needs the vector extension)
and marks every migration applied without running it. An Anthropic-only
fresh install therefore selected the pgvector adapter but had no table,
failing with raw PG errors on first upload or search.
Two layers close it:
- VectorStore::Pgvector#ensure_schema! provisions the table idempotently
on first use (mirrors CreateVectorStoreChunks; memoized; failures wrap
in VectorStore::Error, which with_response turns into a clean failed
response).
- VectorStore::Registry#build_pgvector now gates on
VectorStore::Pgvector.available? (table exists, or extension present),
so installs whose Postgres lacks pgvector entirely degrade to the
assistant's provider_not_configured message instead of raising
mid-chat.
Also resolves the schema.rb version conflict against main (keep the
branch's 2026_06_01_120000, on top of main's current tables).
* fix(ai): address review nitpicks on pgvector provisioning
- Registry: update the adapter doc comment to mention the
Anthropic-to-pgvector default alongside the openai fallback.
- ensure_schema!: guard the DDL with if_not_exists instead of a Mutex.
Adapter instances are built per call and never shared across threads,
so the realistic race is two processes (web + Sidekiq) provisioning
concurrently; IF NOT EXISTS makes the loser a no-op where a Mutex
would only serialize threads inside one process.
* fix(ai): address review on Anthropic settings UI
- Require an Anthropic model when a custom base URL is saved, mirroring the
OpenAI branch. Auto-submit-on-blur could persist a base URL with no model,
making Provider::Anthropic raise "Model is required..." on every LLM call.
- Narrow the LLM provider selector copy: only chat honors Setting.llm_provider;
categorization, merchant detection and PDF processing still always use OpenAI.
Stop advertising provider switching for those flows until they are wired.
- Reset global Setting.* in test teardown to prevent state leakage, and add a
test covering the new base-URL-requires-model validation.
* feat(ds): conditional LLM provider settings + merged copy
The self-hosting AI section showed both providers' credential blocks at once
and duplicated near-identical copy. Tidy it:
- Replace the provider <select> with a DS::SegmentedControl driving a new
provider-settings Stimulus controller: only the active provider's panel is
shown; switching reveals the other instantly and persists Setting.llm_provider.
- Merge the two byte-identical data-retention lines into one provider-neutral
Data handling note.
- Scope the token-budget copy to OpenAI-compatible calls (read only by
Provider::Openai) and add an inline 'add a key to activate' hint when the
active provider is unconfigured.
UI-only; no provider behavior change.
* feat(ds): responsive LLM provider picker (tabs >=sm, select on mobile)
The segmented tabs overflow a phone viewport once there are 3+ providers
(measured: 4 labels want ~409px in a 319px column at 390px wide). Below sm,
fall back to a native <select> -- which doubles as the submitted field -- while
keeping the segmented tabs at sm and up.
Both controls bind to the same provider-settings Stimulus controller (the
select reads its value, the tabs read data-provider), so adding a 3rd/4th
provider scales on mobile with no layout math.
* fix(hostings): sanitize llm provider selector
* test(hostings): avoid brittle provider hint assertion
---------
Co-authored-by: sure-admin <sure-admin@splashblot.com>
* fix(settings): preserve OpenAI form input on validation failure
Fixes#1824.
The OpenAI settings form auto-submits on blur, so typing the URI base
before the model triggers cross-field validation. The rescue re-renders
the page with values read from Setting.openai_*, which is still blank
because the failed save was rejected — so the user's input disappears
and they see 'OpenAI model is required' with no value to fix.
Stash the submitted uri_base and model on rescue and prefer them over
the saved Setting when rendering, so the user can finish typing the
missing field and re-submit.
* test(settings): cover openai_model preservation on validation fail (#1862)
jjmata asked for symmetric coverage of the model field. Add a test where
the user changes the URI base and clears the model in the same submit:
the cross-field validation fails and the re-rendered model input must
reflect the submitted (cleared) value rather than reverting to the saved
model. Complements the existing uri_base preservation test.
* Extract hardcoded strings to i18n
Replace numerous hardcoded English strings with I18n lookups (t / I18n.t) across controllers, views, helpers, and components, and convert model validation error messages to symbol keys. Added multiple locale files under config/locales for models and views. This centralizes user-facing notices/alerts, UI text, import/validation messages, and prepares the app for localization and easier translation maintenance.
* Update en.yml
* Update preview-cleanup.yml
* Revert "Update preview-cleanup.yml"
This reverts commit 1ba6d3c34c.
* test: align i18n assertions with translated messages
* Standardize balance error key and tweak locales
Replace SophtronAccount's :requires_balance error key with :no_balance and update related locale strings for sophtron, plaid, and simplefin accounts to use the new key and clearer copy. Also switch the QIF upload redirect notice to use a relative translation key (t('.qif_uploaded')), remove an unused SSO providers help line, and fix a trailing-newline/whitespace issue in the subscriptions locale. These changes standardize validation keys and improve translation consistency and messaging.
---------
Co-authored-by: KiloClaw <kiloclaw@openclaw.ai>
* feat(design-system): add info semantic color token
Mirrors success/warning/destructive: --color-info maps to blue-600 in
light mode, blue-500 in dark mode. Unblocks the DS::Alert info variant
from carrying a raw 'blue-600' literal in icon_color and lets surface
tokens use bg-info/N alpha modifiers like the rest of the system.
Refs #1715
* refactor(design-system): adopt semantic tokens and add body slot in DS::Alert
Replaces the bg-{blue,green,yellow,red}-50 / text-{...}-700 / border-{...}-200
palette block in DS::Alert with semantic alpha-modifier surfaces
(bg-{info,success,warning,destructive}/10 + matching /20 borders).
Drops the 'blue-600' literal that icon_color was returning for the
info variant; helpers#icon now accepts color: :info backed by the
new --color-info token.
Adds an optional title: kwarg and an opt-in block-content slot so
rich alerts (title + paragraph, lists, embedded actions) can render
without callers reaching for a hand-rolled flex layout. The existing
message: API stays backward-compatible — nothing in the codebase that
already calls DS::Alert.new(message: ..., variant: ...) needs to change.
Lookbook gains with_title and with_body_slot examples covering the
new shapes.
Refs #1715
* refactor(views): migrate api_keys, hostings, lunchflow alerts to DS::Alert
Cleans up nine bespoke alert blocks that hand-rolled the same
flex + icon + bordered-surface shape DS::Alert already provides:
- settings/api_keys/{new,created,created.turbo_stream}.html.erb — three
near-identical 'Security Warning' / 'Important Security Note' boxes
using the broken bg-warning-50 / text-warning-700 raw-palette pair.
- settings/hostings/{_alpha_vantage,_eodhd,_yahoo_finance,_twelve_data,_provider_selection}_settings.html.erb —
five amber-50 / amber-200 warning boxes covering rate-limit notes,
health-check failure messaging, and the env-configured override
banner. The twelve_data plan-restriction block keeps its bullet
list and pricing link inside the new DS::Alert body slot.
- lunchflow_items/{_api_error,_setup_required}.html.erb — two modal
alert headers whose flex+icon scaffolding now collapses onto
DS::Alert. The surrounding bg-surface 'Common issues' / 'Setup
steps' info cards stay as-is; this PR only touches the alert
shape itself.
No functional or behavioural changes. Locale keys preserved.
amber-* palette uses on the alerts disappear; remaining bg-amber-*
hits in the codebase live outside the alert pattern and stay for
follow-up sub-PRs of #1715.
Refs #1715
* chore(design-system): swap raw gray classes for semantic tokens in settings/
Pilot for the broader raw-color sweep. Maps 21 occurrences across 11
files to design-system equivalents:
- text-white bg-gray-900 hover:bg-gray-800 (CTA buttons)
-> text-inverse button-bg-primary hover:button-bg-primary-hover
- bg-gray-25 / bg-gray-50 / bg-gray-100 (subtle surface backgrounds)
-> bg-surface-inset
- bg-gray-800 (tooltip pills) -> bg-inverse
- text-white inside tooltips -> text-inverse
- text-gray-300 (muted tooltip labels) -> text-inverse opacity-70
- text-gray-600 (muted body text) -> text-secondary
- hover:text-gray-700 -> hover:text-primary
- focus:ring-gray-900 -> focus:ring-button-bg-primary
The 7 status-indicator dots (`bg-gray-400`) are intentionally left
as raw classes. Gray-400 against both light and dark container bgs
gives reasonable contrast either way, and there's no semantic token
that fits a "neutral inactive indicator" use case yet. Worth a
follow-up if a `bg-subdued` token would benefit other places.
* fix(design-system): use theme-aware focus ring on provider submit buttons
Two issues caught in code review:
1. focus:ring-button-bg-primary silently emits no CSS (CodeRabbit, Codex).
button-bg-primary is a custom @utility, not a theme color, so Tailwind's
ring-{name} resolution finds no --color-button-bg-primary and falls
back to the default. Replaces with focus:ring-gray-900
theme-dark:focus:ring-white — same color flip as the button bg, but
resolved through theme colors so ring-{name} actually generates CSS.
2. _enable_banking_panel.html.erb dropped focus-ring + transition entirely
in the original sweep (CodeRabbit). Restores parity with the other
provider panels using the corrected ring classes.
Long-term cleanup: tracked under issue #1653 (modifier-aware utilities)
to make button-bg-primary also a theme color so ring-button-bg-primary
becomes valid.
* refactor(design-system): migrate fg-* utilities to text-* and remove namespace
The design system carried two parallel namespaces for foreground colors:
text-* (canonical, ~2,000 uses) and fg-* (32 uses). Most fg-* tokens
were 1:1 duplicates of a text-* counterpart. fg-gray was nearly
identical to text-secondary, with a one-step shade difference in dark
mode.
This PR migrates all 32 usages to their text-* equivalents and removes
the fg-* block from the design tokens. Closes#1606.
Mapping:
- fg-inverse -> text-inverse (20 usages, identical light/dark values)
- fg-gray -> text-secondary (7 usages; light values match, dark is
one step lighter: gray-300 vs gray-400)
- fg-primary -> text-primary (3 usages, identical values)
- fg-subdued -> text-subdued (2 usages, identical values)
The four other fg-* tokens (fg-contrast, fg-primary-variant,
fg-secondary, fg-secondary-variant) had zero usages despite being
defined; they are removed without replacement.
JSON / build:
- design/tokens/sure.tokens.json: $version 1.0.0 -> 2.0.0 (breaking
schema change per the policy added in #1620). 8 fg-* token
definitions removed.
- button-bg-ghost-hover's dark value still references "fg-inverse"
internally; rewritten to "bg-gray-800 text-inverse" so the cleanup
doesn't break that utility.
- _generated.css regenerated. 42 utility blocks now (was 50).
Lookbook tokens preview:
- The Text & foregrounds section dropped its split between text-*
(canonical) and fg-* (legacy). Now a single section listing the
five text-* utilities. The "(legacy)" framing is gone since there's
no legacy left.
README:
- design/tokens/README.md's button-bg-ghost-hover edge-case example
updated to reflect the new "bg-gray-800 text-inverse" dark value.
Visual review needed in dark mode:
- Anywhere icons use the application_helper#icon helper with
color: "default" (most icons in the app). The default class moved
from fg-gray (gray-400 dark) to text-secondary (gray-300 dark), so
default-color icons render slightly lighter in dark mode.
- DS::Buttonish icons in secondary buttons (same shade shift).
- DS::Link icons (same).
- Time series chart axes (same).
- All tooltips, account add flow, settings hostings buttons,
invitations, AI consent, family export, danger-zone buttons --
these used fg-inverse, which is identical to text-inverse, so no
visual change expected.
* fix(design-system): use inverse pair on tooltips for readable dark mode
* fix(lookbook): use semantic tokens in menu preview header text
* fix(lookbook): set text-primary on layout body so previews inherit theme
* fix(design-system): keep shadows dark-toned in dark mode
Inverting shadows to white|8% on dark surfaces produces a halo
effect rather than an elevation cue, and stacks redundantly with
the alpha-white 1px ring already in shadow-border-*.
Switch dark-mode shadows to black at progressively higher alpha
(25%/30%/35%/40%/50% for xs..xl) so they read as actual cast
shadows on near-black surfaces. Surface-tint differences and the
existing alpha-white border ring continue to handle elevation
hierarchy and edge definition.
Approach matches Material 3, Apple HIG, IBM Carbon, Refactoring UI,
and the dark-mode shadows used in Linear/Vercel/Stripe.
* fix(design-system): set text-primary on DS::Dialog element
Browser UA stylesheets apply color: black directly to <dialog>,
which overrides ancestor inheritance even when a body or html
ancestor sets a theme-aware color. Unstyled child content then
renders black regardless of theme.
Setting text-primary on the dialog element itself defeats the UA
override and lets descendants inherit the semantic token.
* fix(lookbook): use shadow css vars in effects preview so dark theme renders
* Revert "fix(design-system): keep shadows dark-toned in dark mode"
This reverts commit 3e9d76ed0beb5ac5f2acbad61e4d1c39eadc9ac2.
* fix(design-system): use opacity-70 instead of text-inverse/70 in value tooltip
The custom @utility text-inverse expands to @apply text-white and
isn't modifier-aware, so text-inverse/70 produced no CSS at all and
the muted labels fell through to inherited color (invisible on the
white pill in dark mode).
Replace with text-inverse + opacity-70. Same visual effect, works
with the existing utility definition.
* Binance as securities provider
* Disable twelve data crypto results
* Add logo support and new currency pairs
* FIX importer fallback
* Add price clamping and optiimize retrieval
* Review
* Update adding-a-securities-provider.md
* day gap miss fix
* New fixes
* Brandfetch doesn't support crypto. add new CDN
* Update _investment_performance.html.erb
* Document admin-only reset auth in OpenAPI docs
The DELETE /api/v1/users/reset endpoint now requires admin role
(ensure_admin). Update the rswag spec to:
- Set default user role to admin so the 200 test passes
- Add a 403 response case for non-admin users with read_write scope
- Clarify the description notes admin requirement
- Add SuccessMessage schema and users paths to openapi.yaml
https://claude.ai/code/session_01Tj8ToLRmVg5HLmHwq9KKDY
* Consolidate duplicate 403 responses for reset endpoint
OpenAPI keys responses by status code, so two 403 blocks caused the
first (insufficient scope) to be silently overwritten by the second
(non-admin). Merge into a single 403 whose description covers both
causes: requires read_write scope and admin role. The test exercises
the read-only key path which hits 403 via scope check.
https://claude.ai/code/session_01Tj8ToLRmVg5HLmHwq9KKDY
* Em-dash out of messages.
* Fix tests
* Fix tests
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Add default family selection for invite-only onboarding mode
When onboarding is set to invite-only, admins can now choose a default
family that new users without an invitation are automatically placed into
as members, instead of creating a new family for each signup.
https://claude.ai/code/session_01U9KgikKjV6xbyBZ5wMYsYx
* Restrict invite codes and onboarding settings to super_admin only
The Invite Codes section on /settings/hosting was visible to any
authenticated user via the show action, leaking all family names/IDs
through the default-family dropdown. This tightens access:
- Hide the entire Invite Codes section in the view behind super_admin?
- Add before_action :ensure_super_admin to InviteCodesController for
all actions (index, create, destroy), replacing the inline admin? check
- Add ensure_super_admin_for_onboarding filter on hostings#update that
blocks non-super_admin users from changing onboarding_state or
invite_only_default_family_id
https://claude.ai/code/session_01U9KgikKjV6xbyBZ5wMYsYx
* Fix tests for super_admin-only invite codes and onboarding settings
- Hostings controller test: sign in as sure_support_staff (super_admin)
for the onboarding_state update test, since ensure_super_admin_for_onboarding
now requires super_admin role
- Invite codes tests: use super_admin fixture for the success case and
verify that a regular admin gets redirected instead of raising StandardError
https://claude.ai/code/session_01U9KgikKjV6xbyBZ5wMYsYx
* Fix system test to use super_admin for self-hosting settings
The invite codes section is now only visible to super_admin users,
so the system test needs to sign in as sure_support_staff to find
the onboarding_state select element.
https://claude.ai/code/session_01U9KgikKjV6xbyBZ5wMYsYx
* Skip invite code requirement when a default family is configured
When onboarding is invite-only but a default family is set, the
claim_invite_code before_action was blocking registration before
the create action could assign the user to the default family.
Now invite_code_required? returns false when
invite_only_default_family_id is present, allowing codeless
signups to land in the configured default family.
https://claude.ai/code/session_01U9KgikKjV6xbyBZ5wMYsYx
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat(helm): add Pipelock ConfigMap, scanning config, and consolidate compose
- Add ConfigMap template rendering DLP, response scanning, MCP input/tool
scanning, and forward proxy settings from values
- Mount ConfigMap as /etc/pipelock/pipelock.yaml volume in deployment
- Add checksum/config annotation for automatic pod restart on config change
- Gate HTTPS_PROXY/HTTP_PROXY env injection on forwardProxy.enabled (skip
in MCP-only mode)
- Use hasKey for all boolean values to prevent Helm default swallowing false
- Single source of truth for ports (forwardProxy.port/mcpProxy.port)
- Pipelock-specific imagePullSecrets with fallback to app secrets
- Merge standalone compose.example.pipelock.yml into compose.example.ai.yml
- Add pipelock.example.yaml for Docker Compose users
- Add exclude-paths to CI workflow for locale file false positives
* Add external assistant support (OpenAI-compatible SSE proxy)
Allow self-hosted instances to delegate chat to an external AI agent
via an OpenAI-compatible streaming endpoint. Configurable per-family
through Settings UI or ASSISTANT_TYPE env override.
- Assistant::External::Client: SSE streaming HTTP client (no new gems)
- Settings UI with type selector, env lock indicator, config status
- Helm chart and Docker Compose env var support
- 45 tests covering client, config, routing, controller, integration
* Add session key routing, email allowlist, and config plumbing
Route to the actual OpenClaw session via x-openclaw-session-key header
instead of creating isolated sessions. Gate external assistant access
behind an email allowlist (EXTERNAL_ASSISTANT_ALLOWED_EMAILS env var).
Plumb session_key and allowedEmails through Helm chart, compose, and
env template.
* Add HTTPS_PROXY support to External::Client for Pipelock integration
Net::HTTP does not auto-read HTTPS_PROXY/HTTP_PROXY env vars (unlike
Faraday). Explicitly resolve proxy from environment in build_http so
outbound traffic to the external assistant routes through Pipelock's
forward proxy when enabled. Respects NO_PROXY for internal hosts.
* Add UI fields for external assistant config (Setting-backed with env fallback)
Follow the same pattern as OpenAI settings: database-backed Setting
fields with env var defaults. Self-hosters can now configure the
external assistant URL, token, and agent ID from the browser
(Settings > Self-Hosting > AI Assistant) instead of requiring env vars.
Fields disable when the corresponding env var is set.
* Improve external assistant UI labels and add help text
Change placeholder to generic OpenAI-compatible URL pattern. Add help
text under each field explaining where the values come from: URL from
agent provider, token for authentication, agent ID for multi-agent
routing.
* Add external assistant docs and fix URL help text
Add External AI Assistant section to docs/hosting/ai.md covering setup
(UI and env vars), how it works, Pipelock security scanning, access
control, and Docker Compose example. Drop "chat completions" jargon
from URL help text.
* Harden external assistant: retry logic, disconnect UI, error handling, and test coverage
- Add retry with backoff for transient network errors (no retry after streaming starts)
- Add disconnect button with confirmation modal in self-hosting settings
- Narrow rescue scope with fallback logging for unexpected errors
- Safe cleanup of partial responses on stream interruption
- Gate ai_available? on family assistant_type instead of OR-ing all providers
- Truncate conversation history to last 20 messages
- Proxy-aware HTTP client with NO_PROXY support
- Sanitize protocol to use generic headers (X-Agent-Id, X-Session-Key)
- Full test coverage for streaming, retries, proxy routing, config, and disconnect
* Exclude external assistant client from Pipelock scan-diff
False positive: `@token` instance variable flagged as "Credential in URL".
Temporary workaround until Pipelock supports inline suppression.
* Address review feedback: NO_PROXY boundary fix, SSE done flag, design tokens
- Fix NO_PROXY matching to require domain boundary (exact match or .suffix),
case-insensitive. Prevents badexample.com matching example.com.
- Add done flag to SSE streaming so read_body stops after [DONE]
- Move MAX_CONVERSATION_MESSAGES to class level
- Use bg-success/bg-destructive design tokens for status indicators
- Add rationale comment for pipelock scan exclusion
- Update docs last-updated date
* Address second round of review feedback
- Allowlist email comparison is now case-insensitive and nil-safe
- Cap SSE buffer at 1 MB to prevent memory blowup from malformed streams
- Don't expose upstream HTTP response body in user-facing errors (log it instead)
- Fix frozen string warning on buffer initialization
- Fix "builtin" typo in docs (should be "built-in")
* Protect completed responses from cleanup, sanitize error messages
- Don't destroy a fully streamed assistant message if post-stream
metadata update fails (only cleanup partial responses)
- Log raw connection/HTTP errors internally, show generic messages
to users to avoid leaking network/proxy details
- Update test assertions for new error message wording
* Fix SSE content guard and NO_PROXY test correctness
Use nil check instead of present? for SSE delta content to preserve
whitespace-only chunks (newlines, spaces) that can occur in code output.
Fix NO_PROXY test to use HTTP_PROXY matching the http:// client URL so
the proxy resolution and NO_PROXY bypass logic are actually exercised.
* Forward proxy credentials to Net::HTTP
Pass proxy_uri.user and proxy_uri.password to Net::HTTP.new so
authenticated proxies (http://user:pass@host:port) work correctly.
Without this, credentials parsed from the proxy URL were silently
dropped. Nil values are safe as positional args when no creds exist.
* Update pipelock integration to v0.3.1 with full scanning config
Bump Helm image tag from 0.2.7 to 0.3.1. Add missing security
sections to both the Helm ConfigMap and compose example config:
mcp_tool_policy, mcp_session_binding, and tool_chain_detection.
These protect the /mcp endpoint against tool injection, session
hijacking, and multi-step exfiltration chains.
Add version and mode fields to config files. Enable include_defaults
for DLP and response scanning to merge user patterns with the 35
built-in patterns. Remove redundant --mode CLI flag from the Helm
deployment template since mode is now in the config file.
* Add warning for TwelveData plan-restricted tickers
Fixes#800
- Add Security::PlanRestrictionTracker concern using Rails cache
- Detect plan upgrade errors during Security::Price::Importer sync
- Display amber warning on /settings/hosting with affected tickers
- Include unit tests for the new functionality
* Scope plan restriction cache by provider
Addresses review feedback:
- Cache key now includes provider name to support multiple data providers
- Methods now require provider parameter for proper scoping
- Added tests for provider-scoped restrictions
- Added documentation explaining instance-level API key architecture
* Fix RuboCop array bracket spacing
* Fix empty array bracket spacing
* Move plan upgrade detection to Provider::TwelveData
* Fix provider scoping tests to use direct cache writes
---------
Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
* Implement a setting to retrieve high res logos
* Update _brand_fetch_settings.html.erb
* Add fallback for stock tickers also to use Brandfetch
* Update security.rb
* Update toggle logic for high-res logos setting
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
* Update security.rb
* Update security.rb
---------
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
* Add pending transaction handling and duplicate reconciliation logic
- Implemented logic to exclude pending transactions from budgets and analytics calculations.
- Introduced mechanisms for reconciling pending transactions with posted versions.
- Added duplicate detection with support for merging or dismissing matches.
- Updated transaction search filters to include a `status_filter` for pending/confirmed transactions.
- Introduced UI elements for reviewing and resolving duplicates.
- Enhanced `ProviderSyncSummary` with stats for reconciled and stale pending transactions.
* Refactor translation handling and enhance transaction and sync logic
- Moved hardcoded strings to locale files for improved translation support.
- Refined styling for duplicate transaction indicators and sync summaries.
- Improved logic for excluding stale pending transactions and updating timestamps on batch exclusion.
- Added unique IDs to status filters for better element targeting in UI.
- Optimized database queries to avoid N+1 issues in stale pending calculations.
* Add sync settings and enhance pending transaction handling
- Introduced a new "Sync Settings" section in hosting settings with UI to toggle inclusion of pending transactions.
- Updated handling of pending transactions with improved inference logic for `posted=0` and `transacted_at` in processors.
- Added priority order for pending transaction inclusion: explicit argument > environment variable > runtime configurable setting.
- Refactored settings and controllers to store updated sync preferences.
* Refactor sync settings and pending transaction reconciliation
- Extracted logic for pending transaction reconciliation, stale exclusion, and unmatched tracking into dedicated methods for better maintainability.
- Updated sync settings to infer defaults from multiple provider environment variables (`SIMPLEFIN_INCLUDE_PENDING`, `PLAID_INCLUDE_PENDING`).
- Refined UI and messaging to handle multi-provider configurations in sync settings.
# Conflicts:
# app/models/simplefin_item/importer.rb
* Debounce transaction reconciliation during imports
- Added per-run reconciliation debouncing to prevent repeated scans for the same account during chunked history imports.
- Trimmed size of reconciliation stats to retain recent details only.
- Introduced error tracking for reconciliation steps to improve UI visibility of issues.
* Apply ABS() in pending transaction queries and improve error handling
- Updated pending transaction logic to use ABS() for consistent handling of negative amounts.
- Adjusted amount bounds calculations to ensure accuracy for both positive and negative values.
- Refined exception handling in `merge_duplicate` to log failures and update user alert.
- Replaced `Date.today` with `Date.current` in tests to ensure timezone consistency.
- Minor optimization to avoid COUNT queries by loading limited records directly.
* Improve error handling in duplicate suggestion and dismissal logic
- Added exception handling for `store_duplicate_suggestion` to log failures and prevent crashes during fuzzy/low-confidence matches.
- Enhanced `dismiss_duplicate` action to handle `ActiveRecord::RecordInvalid` and display appropriate user alerts.
---------
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
* Initial implementation
* FIX keys
* Add langfuse evals support
* FIX trace upload
* Delete .claude/settings.local.json
Signed-off-by: soky srm <sokysrm@gmail.com>
* Update client.rb
* Small LLMs improvements
* Keep batch size normal
* Update categorizer
* FIX json mode
* Add reasonable alternative to matching
* FIX thinking blocks for llms
* Implement json mode support with AUTO mode
* Make auto default for everyone
* FIX linter
* Address review
* Allow export manual categories
* FIX user export
* FIX oneshot example pollution
* Update categorization_golden_v1.yml
* Update categorization_golden_v1.yml
* Trim to 100 items
* Update auto_categorizer.rb
* FIX for auto retry in auto mode
* Separate the Eval Logic from the Auto-Categorizer
The expected_null_count parameter conflates eval-specific logic with production categorization logic.
* Force json mode on evals
* Introduce a more mixed dataset
150 items, performance from a local model:
By Difficulty:
easy: 93.22% accuracy (55/59)
medium: 93.33% accuracy (42/45)
hard: 92.86% accuracy (26/28)
edge_case: 100.0% accuracy (18/18)
* Improve datasets
Remove Data leakage from prompts
* Create eval runs as "pending"
---------
Signed-off-by: soky srm <sokysrm@gmail.com>
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Implement Yahoo Finance
* Added tests
* Updated hosting controller to check for managed app_mode instead of env_override
* Suggestions from CodeRabbit and Fixes on tests
* Remove Css changes
* Fix yahoo finance impl and i18n
* Updated view to use healthy method
* remove usage
* Updated env example
* keep usage on class just to keep same format
* Ci test
* Remove some useless validations
* Remove logs
* Linter fixes
* Broke this in my conflict merge
* Wrong indentation level
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Remove orphan function
* Add centralized branding helpers and update locales
* Remove _plus and add (proper) brand
* No longer Sure, configurable
* Consistency with compose file naming
* Missed `product_name` mapping
* Fix brand/product name in mailers
* Product name in email reset flow
* Fix i18n errors/tests
* Fix password mailer brand/product name (again)
* Missed hardcoded `Sure` in onboarding goals
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
* PR nitpick on documentation
* Missing interpolation key for invited UI
* Orphan assets
* New logos
---------
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Implement support for generic OpenAI api
- Implements support to route requests to any openAI capable provider ( Deepsek, Qwen, VLLM, LM Studio, Ollama ).
- Keeps support for pure OpenAI and uses the new better responses api
- Uses the /chat/completions api for the generic providers
- If uri_base is not set, uses default implementation.
* Fix json handling and indentation
* Fix linter error indent
* Fix tests to set env vars
* Fix updating settings
* Change to prefix checking for OAI models
* FIX check model if custom uri is set
* Change chat to sync calls
Some local models don't support streaming. Revert to sync calls for generic OAI api
* Fix tests
* Fix tests
* Fix for gpt5 message extraction
- Finds the message output by filtering for "type" == "message" instead of assuming it's at index 0
- Safely extracts the text using safe navigation operators (&.)
- Raises a clear error if no message content is found
- Parses the JSON as before
* Add more langfuse logging
- Add Langfuse to auto categorizer and merchant detector
- Fix monitoring on streaming chat responses
- Add Langfuse traces also for model errors now
* Update app/models/provider/openai.rb
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: soky srm <sokysrm@gmail.com>
* handle nil function results explicitly
* Exposing some config vars.
* Linter and nitpick comments
* Drop back to `gpt-4.1` as default for now
* Linter
* Fix for strict tool schema in Gemini
- This fixes tool calling in Gemini OpenAI api
- Fix for getTransactions function, page size is not used.
---------
Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Add ability to delete invite codes
Implemented destroy action in InviteCodesController and updated routes to support invite code deletion. Updated invite code partial to include a delete button and improved styling. Also refactored the generate tokens button in invite code settings to use DS::Button.
* Show advanced settings only to admin users
Updated the settings navigation to display the advanced section only for admin users. Also improved handling of hidden elements in the invite code CSS.
* feat: Add Brand Fetch logo link for logos
* docs: brand fetch integration docs
* Document CLIENT_ID location?
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
---------
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: Vincent Teo <vinteo@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* Remove Synth Finance integration
* Linter noise
* Fix failing (old) test, use it for Twelve Data
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
* feat: Add Twelve Data provider for exchange rates and securities
* test: fix hosting controller test, linting
* fix: add countries gem to handle country codes in Twelve Data provider
* fix: allow security search combobox to have no logo
* refactor: update Twelve Data provider use time series endpoint
* fix: set twelve data as default provider
* fix: replace hardcoded bg-white with bg-container in notification notice
* fix: replace hardcoded text-white with fg-inverse in notification CTA
* fix: replace hardcoded text-white with fg-inverse in text tooltip
* fix: replace hardcoded bg-gray-900 text-white with bg-inverse fg-inverse in invitations form
* fix: replace hardcoded bg-gray-800 text-white with bg-inverse fg-inverse in AI consent form
* fix: replace hardcoded text-white with fg-inverse in changelog page
* fix: replace hardcoded text-white and border-gray-500 with fg-inverse and border-secondary in investment tooltip
* fix: replace hardcoded text-white with fg-inverse in holdings missing price tooltip
* fix: replace hardcoded text-white and bg-gray-400 with fg-inverse and bg-surface-inset in settings profiles
* fix: replace hardcoded bg-orange-500 text-white with bg-yellow-600 fg-inverse in settings hosting danger zone
---------
Co-authored-by: Eran Avidor <eavidor@Eran-Avidor-MBP.lan>
* Add lookbook + viewcomponent, organize design system file
* Build menu component
* Button updates
* More button fixes
* Replace all menus with new ViewComponent
* Checkpoint: fix tests, all buttons and menus converted
* Split into Link and Button components for clarity
* Button cleanup
* Simplify custom confirmation configuration in views
* Finalize button, link component API
* Add toggle field to custom form builder + Component
* Basic tabs component
* Custom tabs, convert all menu / tab instances in app
* Gem updates
* Centralized icon helper
* Update all icon usage to central helper
* Lint fixes
* Centralize all disclosure instances
* Dialog replacements
* Consolidation of all dialog styles
* Test fixes
* Fix app layout issues, move to component with slots
* Layout simplification
* Flakey test fix
* Fix dashboard mobile issues
* Finalize homepage
* Lint fixes
* Fix shadows and borders in dark mode
* Fix tests
* Remove stale class
* Fix filled icon logic
* Move transparent? to public interface
Since the very first 0.1.0-alpha.1 release, we've been moving quickly to add new features to the Maybe app. In doing so, some parts of the codebase have become outdated, unnecessary, or overly-complex as a natural result of this feature prioritization.
Now that "core" Maybe is complete, we're moving into a second phase of development where we'll be working hard to improve the accuracy of existing features and build additional features on top of "core". This PR is a quick overhaul of the existing codebase aimed to:
- Establish the brand new and simplified dashboard view (pictured above)
- Establish and move towards the conventions introduced in Cursor rules and project design overview #1788
- Consolidate layouts and improve the performance of layout queries
- Organize the core models of the Maybe domain (i.e. Account::Entry, Account::Transaction, etc.) and break out specific traits of each model into dedicated concerns for better readability
- Remove stale / dead code from codebase
- Remove overly complex code paths in favor of simpler ones
