* fix(transfers): prevent duplicate creation on double-submit
TransfersController#create -> Transfer::Creator had no protection
against a repeated form submission - a double-click, a browser retry,
or two near-simultaneous requests could each create a separate,
identical transfer (and its 2-4 underlying Entry/Transaction rows).
Adds a per-form idempotency key, the same approach already used for
TransactionsController#create: a UUID hidden field generated fresh on
page load, tagging the outflow/inflow (and fee, with a distinguishing
suffix since a fee leg shares its account with its primary leg) entries
via the existing entries(account_id, source, external_id) partial
unique index. A pre-check handles the sequential double-submit case;
rescue ActiveRecord::RecordNotUnique is the authoritative backstop for
genuine concurrent requests - the whole Transfer.transaction block
rolls back cleanly on conflict, so there's no risk of a half-created
transfer.
A same-day duplicate transfer can be legitimate (unlike a duplicate
valuation, see #3339/PR #3340), so this uses the same per-submission
token approach as #3334/PR #3338 rather than a natural-key DB
constraint.
Fixes#3341.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(transfers): store the idempotency key in its own column, isolate the retry with a savepoint
Same two review findings as PR #3338 (transactions) and #3340
(valuations), applied here since this branch shares the same
mechanism:
- Reusing external_id/source for the web-form idempotency token made
every leg of a manually-created transfer satisfy Entry#linked?,
incorrectly making it look provider-synced. Uses the same dedicated
entries.idempotency_key column added in
db/migrate/20260902180400_add_idempotency_key_to_entries.rb (cherry-picked
identically from PR #3338 - this branch depends on that migration;
please merge #3338 first, or merge this after it lands so the
duplicate migration file is a no-op).
- Transfer::Creator now wraps the actual save in
Transfer.transaction(requires_new: true) so a RecordNotUnique only
rolls back to a savepoint rather than aborting any transaction the
caller might already be in, keeping the rescue's retry lookup usable
(mirrors the fix already applied to Account::ReconciliationManager
in PR #3340).
Added a regression test asserting neither leg of a transfer created
via this path is linked? or has external_id/source set.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(transfers): distinct idempotency key per leg, rebuild invalid index on retry
Two more review findings:
- CodeRabbit: the form doesn't prevent selecting the same account as
both source and destination. The outflow and inflow legs shared the
bare idempotency key, so on that same-account path they'd collide
with each other under the same account-scoped unique index (as
would both fee legs, which shared a single "-fee" suffix). Every
leg now gets a distinct, role-specific suffix (outflow stays bare -
that's what find_existing_transfer looks up by - inflow/source_fee/
destination_fee each get their own).
- Codex (same finding already fixed once for entries.idempotency_key's
sibling migration, recurring here since this branch carries an
identical copy): index_exists? alone doesn't distinguish a valid
index from an INVALID one left behind by an interrupted CREATE INDEX
CONCURRENTLY, so a retry after a failed build would short-circuit
and record the migration as applied while the constraint was still
missing. Now checks pg_index.indisvalid directly before deciding to
skip.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(transfers): clear idempotency key on destroy so a retry doesn't 500
Codex flagged that Transfer#destroy! (used by reject!) preserves the
outflow/inflow entries but not the Transfer join row - a retried
create request with the same idempotency_key would find no Transfer
via find_existing_transfer, attempt another insert, hit the stale
entry's unique key, and re-raise RecordNotUnique instead of finding
a match. Clear the key on the surviving entries when a transfer is
destroyed.
Also adds a regression test for the CodeRabbit-flagged per-leg key
collision concern (already fixed by role-specific suffixes in the
prior commit) to lock in that fee legs never share a key with their
primary leg.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(transfers): verify idempotency key matches the request, fix stale doc comment
jjmata review on #3342:
- find_existing_transfer matched on idempotency_key + source_account only,
so a stale key from a cached form could silently return a different,
older transfer instead of creating the one actually requested. Now
verifies destination account, date, and amount before treating a key
match as the same request; a genuine mismatch surfaces as a new
StaleIdempotencyKeyError (422 + message) instead of a false success or
a raw 500.
- Removed a comment claiming parity with a
TransactionsController#new_transaction_idempotency_key method that
doesn't exist in the codebase.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(transfers): preserve from_account_id on error, match fees/exchange rate in idempotency check
coderabbitai review on #3342:
- All three create rescue blocks (exchange rate unavailable, invalid date,
stale idempotency key) failed to set @from_account_id, so the re-rendered
form lost the user's selected source account.
- matches_request? only compared accounts/date/outflow amount, so a retry
with the same key but a different exchange_rate or fee would be reported
as success while silently keeping the old inflow amount and fee entries.
Now recomputes the request's effective inflow amount and compares derived
fee totals too; a mismatch raises StaleIdempotencyKeyError like other
stale-key mismatches instead of silently returning the old transfer.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Gerald <248542187+gfr-free@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* fix(transactions): prevent duplicate creation on double-submit
TransactionsController#create had no protection against a repeated
form submission - a double-click, a browser retry, or two
near-simultaneous requests could all create a separate identical
transaction. Adds a per-form idempotency key (a UUID hidden field,
generated fresh on page load) that reuses the existing
entries(account_id, source, external_id) partial unique index, with a
pre-check for the sequential case and a RecordNotUnique rescue as the
authoritative backstop for genuine concurrent requests - the same
pattern already used by mark_as_recurring and the public API's
idempotency-key support.
Fixes#3334.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(transactions): store the idempotency key in its own column, not external_id
Codex review finding: reusing external_id/source for the web-form
idempotency token made every manually-created transaction satisfy
Entry#linked? (external_id.present?), since the form always supplies
a key. That incorrectly made manual entries look provider-synced -
disabling their date/nature/amount/currency fields in the editor
(app/views/transactions/show.html.erb), and hiding them from future
provider dedup matching (which filters to external_id: nil).
Adds a dedicated entries.idempotency_key column with its own partial
unique index scoped by account_id, used only for this de-duplication
and with no meaning anywhere else in the app, so it can't collide with
provider-linkage semantics. TransactionsController now tags/looks up
entries by this column instead of source/external_id.
Added a regression test asserting a transaction created via this path
is not linked? and has no external_id/source set.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(migration): rebuild an invalid index left by an interrupted CONCURRENTLY build
Codex review finding: index_exists? alone doesn't distinguish a valid
index from an INVALID one left behind by an interrupted CREATE INDEX
CONCURRENTLY (e.g. a deploy killed mid-build). A retry after such a
failure would short-circuit on the early-return and record this
migration as applied, while the actual uniqueness constraint stays
missing/broken. Checks pg_index.indisvalid directly before deciding
whether to skip the rebuild.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix(transactions): rotate idempotency token on bfcache/Turbo restore, keep index removal concurrent
Codex flagged that a page restored from the browser bfcache or Turbo's
snapshot cache (back button, duplicated tab) keeps the already-consumed
idempotency token in the hidden field. Submitting a different, edited
transaction from that restored page would then match the old committed
entry and silently redirect onto it instead of creating the new one.
transaction_form_controller now rotates the token on turbo:before-cache
so any later restore starts from a fresh, unconsumed value.
Also address CodeRabbit's note that the migration's down block did a
blocking DROP INDEX instead of DROP INDEX CONCURRENTLY.
* fix(transactions): also rotate idempotency token on native bfcache restore
CodeRabbit noted turbo:before-cache only covers Turbo's own snapshot
cache, not the browser's native bfcache (e.g. a full navigation away
and back, not through Turbo drive). Add a persisted-pageshow handler
alongside it, and wire both through declarative data-action bindings
on the form per this repo's Stimulus convention instead of manual
addEventListener/connect/disconnect.
* fix(transactions): fall back to manual UUID when crypto.randomUUID is unavailable
crypto.randomUUID() requires a secure context, but this app's self-hosted
mode is commonly reached over plain HTTP (LAN, reverse proxy without TLS).
On such a deployment, calling it inside the cache-restore rotation handlers
throws, leaving the stale, already-consumed idempotency token in the hidden
field — a later edited resubmission would then silently match the old entry
via find_duplicate_manual_entry and drop the user's edits. Build a v4 UUID
manually from crypto.getRandomValues (which has no secure-context
restriction) when randomUUID is missing.
Also drops a stale comment reference to a MANUAL_FORM_SOURCE constant that
doesn't exist anywhere in the codebase, and corrects a rescue comment that
still described the old (account_id, source, external_id) index instead of
the (account_id, idempotency_key) index actually backing this constraint.
---------
Co-authored-by: Gerald <248542187+gfr-free@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Adds a 'Running Sure on small (512 MB) hosts' section to the Docker
self-hosting guide: what fits in 512 MB (boot, onboarding, daily use,
small syncs/imports, cron - ~352 MB steady state), what does not
(demo-data generator deterministic OOM, very large first imports, AI
flavors), the tuning already baked into the image (jemalloc, YJIT,
Puma 1x3), and the operational steps for loading sample data on a
small host (raise the worker limit, fresh deploy - plan changes do
not apply on restart).
Co-authored-by: Instinct agent <agent@sure.am>
The account activity tab rendered split transaction children as flat,
ungrouped rows, unlike /transactions which collapses them into a
parent row with indented children when "Group split transactions" is
enabled. Wire the same EntriesHelper.group_split_entries logic into
the account activity feed (UI::Account::ActivityDate, the actual
render path since the ViewComponent refactor superseded the old
accounts/show/_activity partial), sourcing split parents via the same
single-query batched lookup pattern already used by
TransactionsController#index.
Also forward view_ctx through entries/_split_group so split children
render correctly regardless of which page renders the group.
Fixeswe-promise/sure#3227
Co-authored-by: Gerald <248542187+gfr-free@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* feat(ai): support OPENAI_EXTRA_HEADERS on OpenAI-compatible provider
Adds a fail-closed parser for the OPENAI_EXTRA_HEADERS env var (a JSON
object of header names to values) as a new Provider::Openai.extra_headers
class method. Malformed, non-object, blank, or unset values yield {}
with an error log and never the raw value, so chat keeps working on bad
config. Parsed headers are passed to the ruby-openai client at
construction, attaching them to every request the provider's client
makes (chat and batch flows alike).
ENV-only by design: no Setting fallback or settings-UI entry. Values are
stringified (nested JSON becomes Ruby-inspect strings) and blank values
are dropped.
Adds hosting docs and commented examples in .env.example and
.env.local.example, plus Minitest coverage mirroring the request_timeout
tests, including a docs-consistency test binding the knob to its docs.
* feat(ai): substitute {session_id} in OPENAI_EXTRA_HEADERS per chat request
Header values containing the literal {session_id} are now withheld at
client construction and merged onto the client at request time, with the
placeholder replaced by the chat's UUID. This identifies requests per
conversation rather than per install, for gateways that key sessions
(e.g. OpenCode Zen's x-opencode-session).
A session header is only merged when a session_id is present, so batch
flows (auto-categorize, merchant detection, PDF processing) — which
bypass chat_response — never send it; they receive static headers only.
The merge adds/overwrites without deleting managed headers.
Docs updated to cover both static and session-valued usage.
* fix(ai): keep OPENAI_EXTRA_HEADERS session values request-scoped
client.add_headers persists headers on the shared client in
ruby-openai 8.1.0, so a chat's resolved session header could survive
onto later requests made through the same provider instance. Session
headers are now merged onto a request-scoped dup of the client; the
shared client is never mutated. Batch flows and session-less chats
cannot observe another chat's session id.
Also updates the CodeRabbit-flagged tests to assert the shared client
stays untouched and the scoped copy is what issues the chat request.
* docs(ai): add YARD tags to OPENAI_EXTRA_HEADERS method docs
Converts the comment blocks on the four methods touched by this
feature (extra_headers, initialize, request_timeout, and
with_session_headers) into YARD docstrings with @param/@return tags,
satisfying CodeRabbit's docstring-coverage pre-merge check.
* Add Trade Republic provider integration
Introduce authenticated web and QR login, resilient account synchronization, deterministic financial imports, account discovery, and provider diagnostics. Keep login state encrypted, PINs transient, and incomplete provider responses non-destructive.
* Address Trade Republic review findings
Keep QR-authenticated sessions syncable, preserve historical holding snapshots, correct dividend direction, handle unpriced positions safely, localize repair feedback, and align provider controls with the design system.
* Add Trade Republic translations for supported locales
* Restore German Trade Republic account labels
* Resolve remaining Trade Republic review findings
* Resolve remaining Trade Republic review findings
* Address latest Trade Republic review feedback
* Refactor Trade Republic panel buttons to use DS::Button component and add integration tests
* Fix 100x money inflation and missing positions locale key in TR views
Money.new takes major units, so multiplying by 100 displayed EUR 12.34
as EUR 1234 in the holdings category cards and expense summary. Also
add the pluralized holdings.index.positions key that t(".positions")
resolves to (previously only defined at the unused holdings.positions
root level), across all 18 locales.
* fix(db): repair merge artifacts in schema and migrations
- Remove duplicated icon/progress_basis columns on goals in schema.rb
- Renumber Trade Republic migrations to unique versions (clashed with
main's 20260824120000_add_lifecycle_to_goals)
- Bump schema version to match latest migration
* Address remaining Trade Republic review feedback
* fix(trade-republic): address open PR #3168 review findings\n\n- Reject authenticated sessions without a securities account number so a\n blank account does not mark the item connected on a broken session.\n- Derive a missing trade amount from |quantity| x price, and a missing\n price from the resolved amount, without changing the signed import amount.\n- Regenerate db/schema.rb so the Trade Republic item/account tables and\n indexes are present; a fresh test database was otherwise missing the\n tables even though the migrations were marked up.\n
* fix(trade-republic): localize activity labels in ActivitiesProcessor (i18n)
* fix(trade-republic): localize activity labels in ActivitiesProcessor (i18n)
* fix(trade-republic): add activity labels i18n keys to all locale files
* Fix Trade Republic PR review follow-ups
* fix(trade-republic): i18n-aware category guard and ignore generated graphify cache
- Category matcher skipped core deposit/withdrawal labels; guard now compares
against translated values so German etc skip correctly
- Remove committed graphify-out cache and ignore dir
* Protect holdings from malformed snapshots
* Consolidate Trade Republic migrations
* Address final Trade Republic review comments
* Address final Trade Republic review comments
- Remove hard-coded category matcher (merchant keyword taxonomy) and leave Trade Republic transactions uncategorized when no structured category exists; rely on Sure rules/AI
- Revert shared ProviderImportAdapter# import_trade extra: param; handle Trade Republic trade metadata locally in ActivitiesProcessor via post-import Trade extra merge (preserve existing extra, deep_merge)
- Preserve Trade Republic product distinctions (cash, brokerage/private_markets/interest_products/crypto_wallet via portfolio categories) without collapsing account kinds
---------
Co-authored-by: Aland Baban <snow@iBananaMac.fritz.box>
* fix(hosting): consistent provider-block visibility + fix Twelve Data toggle bug (#3089)
T-Invest was the only provider block always rendered regardless of its
checkbox state; now it follows the same pattern as every other provider
(shown when tinkoff_invest or moex_public is enabled, since T-Invest also
serves as a brand-logo fallback for MOEX-priced securities).
Also fixes a related functional bug: unchecking every securities provider
tried to clear the legacy securities_provider setting by assigning nil,
but rails-settings-cached treats nil as "delete override", which silently
reverted the field to its own default ("twelve_data") — re-enabling Twelve
Data right after the user disabled it. Assigning "" instead persists the
cleared state.
Twelve Data and Yahoo Finance settings blocks can still be shown purely
because they're the selected FX/exchange-rate provider even when unchecked
for securities pricing; added an info notice explaining that instead of
leaving it unexplained.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* fix(hosting): address review feedback on #3333
- Drop the FX-only notice for Twelve Data/Yahoo Finance per feedback —
the block staying visible while unchecked (because it's still the FX
provider) doesn't need extra UI explanation.
- Fix Codex finding: T-Invest settings must stay visible/manageable
whenever a token is already configured, not just when tinkoff_invest
or moex_public is checked. Security::Provided#import_brand_logo calls
the T-Invest provider unconditionally for every non-crypto security
once a token exists, regardless of price provider — hiding the field
in that case would leave an active credential impossible to see,
rotate, or clear through the UI. Reworded the notice to reflect the
real, provider-independent reason instead of the narrower "MOEX only"
framing.
- Fix CodeRabbit finding: setting_test.rb's default-fallback test now
isolates against SECURITIES_PROVIDER(S) env vars, and the
explicit-clear test captures and restores the pre-test values instead
of hardcoding a restore target.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* No overexplaining in code
---------
Co-authored-by: Gerald <248542187+gfr-free@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
Translate the remaining provider-owned setup and status copy while reusing Accountable subtype labels before legacy English fallbacks. Cover German rendering, error handling, and pluralized English and German status summaries.
Co-authored-by: Johns <19662585+Rowdy@users.noreply.github.com>
* feat(bills): assistant and MCP tools for bills
Last of three chunks carved out of #3083, stacked on the UI bundle. Exposes
bills to the builtin assistant and to MCP clients. Everything here is gated
behind preview features, so the tools are absent from tools/list until a user
opts in.
Seven tools:
- get_bills, get_bill_details and get_paycheck_plan for reads
- get_bill_audit, a deterministic review that surfaces likely duplicates, price
changes, trials about to convert, upcoming renewals and long-overdue bills
- create_bill, update_bill and record_bill_payment for writes
Shared argument parsing, permission checks and error shapes live in
BillsSupport, so every tool answers with the same {error, hint} contract the
existing tools use, and a bad argument never aborts the turn.
The write tools mutate financial records on a model's say-so, so they refuse
rather than guess: a payment cannot exceed what its cycle still owes, a repeated
settle will not quietly close next month, an unrecognized frequency is an error
instead of a silent monthly default, and non-finite or negative amounts are
rejected before they reach the database.
The read tools say what they filtered. An empty result names the statuses that
do hold matches, the paycheck plan discloses the unconfirmed series it excluded
from spending headroom, and history and price-change windows report their real
totals rather than letting a caller sum a truncated list.
A not-found no longer returns the scoped relation's SQL, which handed any MCP
client the access-control schema for the cost of a guessed id.
The in-page AI helpers are not here. Smart fill and smart configuration are
buttons on the bills pages, so they ship with the UI bundle along with the
provider-side suggester they call.
Suite 7,854 runs, 0 failures. Rubocop clean, eager loading verified.
* Address the ready-review round
* Reject an out-of-range audit lookback out loud
* Speak the cycle remainder guard through the allocator locale
* feat(rules): add not-equal, does-not-contain, is-not-empty condition operators
Extend transaction rule conditions beyond "equal to" / "is empty":
- text: add "does not contain" (not_like), "not equal to" (!=), "is not empty" (is_not_null)
- number: add "not equal to" (!=)
- select: add "not equal to" (!=), "is not empty" (is_not_null)
NULL handling is inclusive so the operators match user intent:
- "!=" uses IS DISTINCT FROM, so e.g. "category not equal to X" also matches
uncategorized (NULL) transactions
- "does not contain" also matches rows where the field is NULL
transaction_type keeps its custom operator set, and transaction_details is
pinned to the original operators since its JSONB apply only supports
contains/equals/empty semantics.
The conditions Stimulus controller hides the value field for both valueless
operators (is_null and is_not_null).
* refactor(rules): address PR review feedback on condition operators
- Pass VALUELESS_OPERATORS from Ruby to JS via Stimulus value attribute
instead of duplicating the list as a static class property, so there
is a single source of truth for which operators suppress the value field
- Clarify IS DISTINCT FROM comment to note the NULL-inclusion behaviour
is intentional for select-type fields (merchant_id, category_id) and
not applicable to number fields where NULL is impossible at the DB level
- Add test that exercises the OR IS NULL branch of not_like by using
transaction_notes (entries.notes is nullable, unlike entries.name)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor(rules): localize condition operator labels via i18n
Moves all Rule::ConditionFilter operator labels (including ones that
predate this PR) out of OPERATORS_MAP and into config/locales, so
operators() resolves them through t() per request instead of hardcoded
English strings.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GvxjdTgH34cPoJenQAqnpN
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(bills): the bills pages, calendar feed and in-page AI helpers
Second of three chunks carved out of #3083, stacked on the schema and domain
core. This is everything a user sees and clicks. The whole surface sits behind
the preview flag, so it is unreachable until someone opts in.
Pages, all under one nav entry:
- the pay run, a month calendar, the full bills table, and the paycheck planner
- a detail drawer per bill, with payment history, price changes and cost
analytics
- create and edit flows for bills, subscriptions, installment plans and income
The overview marks pay periods inside the month, so a weekly paycheck no longer
reads as one undifferentiated month of bills. Markers appear only when income
actually subdivides the month, which means monthly and undeclared income render
exactly as before and there is no new setting to configure.
Navigation and design system:
- one preview-gated nav item shared by the desktop rail and the mobile bar
- DS::Sparkline for payment-history charts, replacing raw SVG in views
- status badges render through DS::Pill rather than hand-rolled spans
- the suggestions panel is a disclosure that remembers being collapsed, per
device, the way privacy mode and the sidebar width already do
- every surface reflows to phone widths without horizontal scroll
Calendar feed: a signed ICS feed per family, served sessionless by token, with a
reset that revokes previously shared URLs.
In-page AI helpers: smart fill on the bill form and a smart configuration
proposal on an existing bill, each reading a bounded slice of charge history.
Provider-side prompt assembly sits behind the existing LlmConcept interface,
with an implementation for each of the two providers. These belong here rather
than with the assistant tools because they are buttons on these pages and lean
on the provider suggester, not on the tool registry.
Suite 7,776 runs green apart from the pre-existing passkey-session flake, which passes standalone. Rubocop clean, eager loading verified. The hosting guide for the feature ships here rather than with the schema, since its instructions walk pages this PR introduces.
* Render the suggested strip through DS::Disclosure
The hand-rolled details pair predates the component. The card_inset
variant is the same shape, so the strip now inherits the design system
chrome, and the persisted-disclosure controller rides along unchanged.
* Route the remaining hand-rolled chips through the design system
The subscription-state chips, rule-match chips and match-reason chips
become DS::Pill, with the state chips extracted to one shared partial so
the drawer and the summary tab stop carrying copy-pasted markup. The AI
prompt chips become DS::Button and the bills-index filter becomes
DS::SearchInput, both of which this PR already uses elsewhere for the
same shapes.
* Fix erb_lint whitespace offenses in bills views
* Address the post-ready review round
* Require a writable destination account and gate the feed on preview
* Reject an unresolvable declared account out loud
* fix(securities): fall back to a direct chart lookup when Yahoo search has no results
Fixes#3312.
Yahoo Finance's /v1/finance/search autosuggest endpoint has gaps for some
instruments its own chart/quote backend still serves correctly -- notably
Australian managed funds identified by APIR codes (e.g. VAN0111AU, shown
working at finance.yahoo.com/quote/VAN0111AU.AX). Since Sure's "Add
security" combobox relies solely on that search endpoint
(Provider::YahooFinance#search_securities), affected securities can never
be found or added with live pricing -- and the combobox's manual-ticker
fallback creates a permanently offline security instead, since
Security::Resolver has no exchange/provider match to work with.
Investigated whether another provider already in the registry (EODHD,
Twelve Data, Tiingo, Alpha Vantage, MFAPI) or a new one could cover this
instead: none do. AU managed fund/unit-trust data is a paid-enterprise
niche (Morningstar, FE fundinfo, APIR's own reference-data service) with
no free or self-hostable API. The data already exists in the Yahoo
integration Sure has -- the gap was in how search_securities looks it up.
Adds a fallback: when the search endpoint returns zero results for a
ticker-shaped query (no spaces, plausible length), try it as a literal
symbol against the same chart endpoint fetch_security_prices already uses,
and synthesize a single search result from the chart's `meta` block if it
resolves. Guards against Yahoo's "YHD" generic placeholder exchange (used
for many non-US instruments like managed funds) being mapped to XNAS/NASDAQ
by map_exchange_mic's existing guess for that code -- leaves
exchange_operating_mic nil instead, which Security::Resolver already
treats as "unknown" rather than a mismatch. Best-effort: any failure in the
fallback (auth, rate limit, network, malformed response) degrades to no
extra results rather than failing the whole search.
Known remaining gap, not fixed here: a bare APIR code with no exchange
suffix (e.g. "VAN0111AU" without ".AX") still won't resolve, since Sure has
no way to guess which Yahoo suffix to try without an exchange hint. This
fix covers the case a user pastes the full symbol shown on Yahoo's own
quote page URL.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9494Pxh4LZKnNLTGfFXPw
* fix(securities): restrict Yahoo direct-symbol fallback to exchange-qualified symbols
Triaged review feedback on #3313. Verified against live data: a bare ticker
like "XYZ" resolves via Yahoo's chart endpoint to a real, unrelated security
(NYSE-listed Block, Inc.) even though Yahoo's own search index has no
suggestion for it. The fallback's existing guard (no spaces, plausible
length) let this through, meaning a plain acronym search with no real
search-index match could surface a surprising, spurious result instead of
"no match."
Adds a stricter guard requiring the symbol be exchange-qualified (contains
a literal ".", e.g. "VAN0111AU.AX") before attempting the chart lookup. A
bare, indexable ticker that's actually valid already resolves through the
primary search above it; this fallback exists specifically for suffixed,
non-US-style symbols the search index has gaps for, so requiring the
suffix narrows it to that case without losing the fix's actual target.
Updated the existing "symbol doesn't exist" test to use an
exchange-qualified symbol so it still exercises the not-found chart path,
and added a regression test asserting "XYZ" no longer reaches
fetch_cookie_and_crumb at all.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9494Pxh4LZKnNLTGfFXPw
---------
Co-authored-by: Jonathan Kaiser <jaysbeekay@users.noreply.github.com>
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
* fix(demo): stop the goal-seeding matrix crashing rake demo_data:default
Several goals in the demo coverage matrix claimed the same account as a
100%-whole-account link (GoalAccount#whole_account_link_must_be_exclusive),
which only ever worked because they saved one at a time with nothing to
conflict with yet — the moment a second active goal wanted the same
account, save! raised and the whole task aborted.
Give every goal but one per account an explicit partial earmark instead of
a whole-account claim ("Long-term portfolio" keeps the whole of `primary`,
since its on_track pace needs most of that balance).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016GTNba5qE5NwzaHzbp27ye
* No need to overexplain inside the code.
---------
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
create_and_sync anchors only the opening balance. For an account whose
opening balance differs from today's — a loan entered with its original
principal alongside its current remaining balance — that opening anchor is
the account's only entry, so the initial sync walks forward from it and
overwrites the balance the user just typed.
Anchor today's balance too, as an explicit same-day reconciliation, whenever
the two differ and the opening anchor sits on an earlier day. The date guard
matters because opening_balance_date is user-editable and can be today:
ReconciliationManager matches an existing valuation by date alone, kind is
not part of the lookup, so it would reuse the opening anchor and reassign its
amount — discarding the opening balance and leaving a kind: "opening_anchor"
row holding the current balance. On its own date the opening balance wins.
A direct reconciliation rather than CurrentBalanceManager keeps the anchor
type-agnostic: the manager's cash-account strategy computes a zero delta at
creation and would only rewrite the opening anchor, leaving today's balance
unanchored for the first sync.
initial_balance's presence is read from the raw value, because "".to_d is 0.
A blank field would otherwise set an opening balance of 0 and, since 0
differs from the entered balance, write a reconciliation on top. An explicit
0 is still a real opening balance.
initial_balance is a Loan attribute today, so the loan regression tests cover
the reachable path.
* Surface probe timeout separately from LLM request timeout in admin System Health
- Expose probe_request_timeout in AiHealth (mirrors Probe#timeout)
- Split the ambiguous 'Request timeout' row into 'LLM request timeout'
and a new 'Health-check probe timeout' row
- Forward AI_HEALTH_PROBE_TIMEOUT / AI_HEALTH_PROBE_CACHE_TTL in
compose.example.yml and compose.example.ai.yml
- Remove now-orphaned labels.request_timeout locale key (i18n-tasks)
- Add regression test asserting both values surface distinctly
* Address PR #3276 review feedback
- Restore the credential-redaction assertion on the PDF probe status test
to match the token the test actually configures (it had been changed to a
placeholder that appears in no code path, making the assertion a no-op and
silently removing its coverage). Sanitize the new distinct-timeouts test to
a non-secret token and assert it is absent.
- Consolidate the probe timeout into a single public AiHealth::Probe.timeout
class method and have AiHealth#probe_request_timeout_value delegate to it
so the two can never drift (rather than re-implementing the same
ENV.fetch + positive-check logic).
- Add unit coverage for Probe.timeout exercising the env var and the fallback
path for missing/zero/negative/non-numeric values.
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: jaysbeekay <jaysbeekay@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* feat(bills): schema and domain core for the bills subsystem
First of three chunks carved out of #3083. This one carries the schema and the
domain layer: no bills pages, no calendar feed, no assistant tools. Nothing here
is reachable from the UI yet, so it changes no user-visible behavior on its own.
Schema, in a single migration with a full down:
- recurrence_rules, recurring_occurrences, recurring_allocations,
recurring_price_changes and recurring_match_rejections
- bill columns on recurring_transactions (bill_type, payment_url, autopay,
notes, anchor and end conditions, weekend adjustment, dedup scope)
- the four data backfills, in their original order
Domain layer:
- Schedule, the pure date PORO every cadence resolves through, and
FrequencyPreset for the labels
- OccurrenceGenerator, Matcher, Allocator, PriceChangeDetector, Classifier,
DeclaredBill, HistoryBackfiller and PaycheckPlanner
- Pipeline, tying detection to generation, plus the nightly job and rake task
Existing detection code changed in three places, each a bug this schema exposes:
- Cleaner used a flat two-month staleness threshold, which silently retired
every quarterly and annual series
- SubscriptionAuditGenerator used a flat 45-day overdue threshold, meaningless
at both ends of the frequency range
- CashFlowWarningGenerator read one projected entry per series, which only
equalled the monthly amount because every series was monthly; weekly bills
were under-counted fourfold in its 30-day projection
The JSON API travels with the model rather than the UI, because the status enum
widens here. The API accepts only active and inactive on write; suggested,
paused and ended are lifecycle states owned by detection, so the documented
enum stays truthful.
Uniqueness keys gain dedup_scope alongside amount, never instead of it: a
series that is not price-forked carries a blank scope, so amount is what keeps
two different prices apart.
Suite 7,550 runs, 0 failures. Rubocop and brakeman clean. Eager loading
verified, and the migration reverses and re-applies. Includes the first review round: orphan repair matches income and refuses coincidental twins, session imports persist occurrence mappings across chunks, semimonthly anchors canonicalize, classifier keywords match whole words, and the down refuses rather than failing when price-forked rows exist.
* Address second review round
Bound the cross-currency default allocation by the entry leftover and the
occurrence remainder, matching the same-currency path. Let keyword stems
carry a suffix again after the word-boundary fix silenced them. Skip an
incoherent recurrence rule row instead of rolling back the whole import.
Check rollback collisions per restored index so a refusal cannot land
after the bills tables are dropped. Replay the closed_at test through a
real second import. Preload the orphan repair associations and move the
allocator errors to locale keys.
* Match index NULL semantics in the rollback collision checks
GROUP BY treats NULLs as equal but the restored unique indexes do not:
account_id is nullable and indexed, so two accountless rows can never
collide under any of them. Excluding NULL accounts keeps the guard from
refusing a rollback PostgreSQL can perform. Verified live both ways:
accountless duplicates roll back, a real collision still refuses.
* Address maintainer review
Scope the payable debt-destination subquery to the row and its family
instead of scanning every account in the installation. Batch the cash
flow generator remaining-amount sums into one grouped query, matching
the two sibling sites. Enforce both window bounds in the after_count
branch so a future-anchored plan cannot leak past the requested end
date. Skip the explicit regeneration when the day column change will
fire the model callback anyway. Add the missing locale entry for the
allocation currency validation.
* Fix PDF vision path failing when poppler-utils is missing
The Docker image omitted poppler-utils, so the OpenAI PDF vision path
(which renders pages with pdftoppm before sending them upstream) always
failed and the admin AI status page surfaced a generic "request failed"
code rather than a useful reason.
- install poppler-utils in the Docker base image (pdftoppm for the
vision render path)
- add an optional failure_code to Provider::Error, so probe errors can
carry a machine-readable reason
- in Provider::Openai::PdfProcessor#convert_pdf_to_images, check
pdftoppm's return value and -- when the binary is genuinely missing,
raise a Provider::Openai::Error with failure_code :render_missing_binary
while preserving the existing [] fallback for other render failures
- register the :render_missing_binary code in the admin locale so the AI
status page shows a concrete, actionable message instead of "request
failed"
- AiHealth::Probe#failure_code now falls through to the default codes
when an error's failure_code is nil, instead of returning nil
- regression tests covering both the missing-binary and the
present-but-fails cases
Fixes the "PDF vision/native path" system check on instances where the
image is built from the checked-in Dockerfile.
* Fix missing-binary detection and preserve failure_code across the error boundary
- convert_pdf_to_images: Kernel#system returns nil (not false) when the executable is absent; raise the coded error on rendered.nil? so a missing pdftoppm yields :render_missing_binary instead of a blank conversion. - Provider::Error#as_json + default_error_transformer: carry failure_code through serialization and error re-wrapping. - Drop the binary_missing? helper (nil result is the authoritative signal) and update regression tests to stub the real nil return value. - Add coverage for failure_code serialization/transformation.
* Fix Provider::Error transformer syntax error and cover Faraday branch
Addresses jjmata's blocking change-request: app/models/provider.rb:54
used a postfix `if` modifier inside a hash-argument/method-call argument
list, which is not valid Ruby. `ruby -c` failed to parse the file, so the
base class every provider inherits from could not autoload and the whole
app (boot, requests, jobs, tests) was down.
Rewrite default_error_transformer to build the optional failure_code
keyword once (only when the error exposes a truthy code) and splat it, so
both the Faraday::Error branch and the generic branch carry the code with
no syntax error and no nil kwarg.
Also add the two Faraday-branch regression tests that were missing
(failure_code preservation + response-body-to-details extraction),
guarding this exact class of broken-argument bug with real assertions.
Verification: ruby -c passes on provider.rb, pdf_processor.rb, probe.rb,
and both test files; plus a behavioral harness against the real
provider.rb source covering the coded/plain/nil-code, Faraday-coded,
Faraday-no-code, nil-response, and generic-error paths (19/19 pass).
Ref: we-promise/sure#3275
* Document the methods added or changed by this PR
* Add poppler-utils to devcontainer image
---------
Co-authored-by: hermes-on-behalf-of-jon <hermes@nousresearch.com>
Co-authored-by: jaysbeekay <jaysbeekay@users.noreply.github.com>
Co-authored-by: sure-admin <sure-admin@splashblot.com>
* fix(transactions): dedupe tag filter for correct summary totals (#3174)
The Transactions page summary box (COUNT/SUM) was using
`.joins(:tags).where(tags: { name: tags })`, an INNER JOIN that fans out
to one row per matching tag. A transaction tagged with two of the filtered
tags produced two rows; the list rendered it once (deduped by Postgres),
but the totals query summed and counted both rows.
Switch the filter to the same `IN (subquery)` idiom already used in
`Api::V1::TransactionsController#index` (line 280-284): the subquery
selects the deduplicated transaction ids, and the outer query keeps them
via `WHERE id IN (...)`. COUNT and SUM now see exactly one row per
matching transaction.
Closes#3174.
* docs: add docstrings to satisfy CodeRabbit coverage check on PR #3290
CodeRabbit's pre-merge check flagged 0% docstring coverage (threshold
80%) on the two functions touched by the tag-filter dedupe fix.
Adds a one-line docstring to Transaction::Search#apply_tag_filter and
explanatory comments above the two new regression tests.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9494Pxh4LZKnNLTGfFXPw
* fix: correct tag association call in regression tests (CI failure)
CI's test_unit job failed on all three new regression tests with
NoMethodError: undefined method '<<' for an instance of Transaction.
`entryable << tag` doesn't exist -- `entryable` is the Transaction record
itself; tags attach through its `has_many :tags, through: :taggings`
association, so it needs to be `entryable.tags << tag`.
I hadn't actually run this suite before -- the environment I authored it
in had no working Ruby/Bundler, so this shipped based on tracing rather
than execution. Now verified for real: 26 runs, 206 assertions, 0
failures, 0 errors for this file; rubocop clean.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A9494Pxh4LZKnNLTGfFXPw
---------
Co-authored-by: jaysbeekay <jaysbeekay@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* feat(ai-health): name the missing function calling behind an opaque chat error
The assistant reads accounts, transactions and holdings through function
calls, so every chat request carries a `tools` payload. A model without
function-calling support rejects it — OpenRouter answers a bare 404 — and
the operator sees only that status code, with nothing pointing at the
model. Both earlier attempts at this guessed from the chat-time error;
the AI status page already runs live probes, so let it answer the
question directly instead.
`AiHealth::Probe#function_calling` asks the configured model for one
trivial tool call the way the assistant asks for its own: chat
completions with `tools` for OpenAI-compatible endpoints, the Responses
API for hosted OpenAI, and `messages.create` with `tools` for Anthropic,
carrying the same strict schema `Provider::Openai` sends. Reading it
against the plain LLM probe is what makes the verdict sound rather than
a guess at 404s: plain chat passing while the same request with tools
fails means the model has no function calling; a response that carries
no tool call means the endpoint took the tools but the model ignored
them; both failing, or a timeout, stays an ordinary probe failure.
The AI status card gains a Function calling (tools) row, an alert
naming the fix for each of the two bad outcomes, and a failure reason.
The hosting settings model field now says the assistant needs a
tools-capable model and links super admins to the check.
Refs #830
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DWCmRQ1ry26JnhA79s1ZKH
* fix(ai-health): only call a refusal a refusal, and probe the route chat takes
Two findings from review of the function-calling probe.
A tools request can fail for reasons that say nothing about tool support:
a 429, a 500, a dropped connection, an unreadable body. Reading any
non-timeout failure as `:unsupported` sent the operator hunting for a new
model over a transient blip. Only a 4xx the service chose to answer with
— excluding the ones that mean "not now" or "not you" — is a refusal of
the tools payload; everything else stays an ordinary probe failure. The
bare 404 from OpenRouter that this page exists to explain still reads as
missing function calling.
`Provider::Openai#supports_responses_endpoint?` is the real routing
decision and `OPENAI_SUPPORTS_RESPONSES_ENDPOINT` can flip it either way,
so choosing the API from "is the endpoint custom" could probe Chat
Completions while chat uses Responses, or the reverse — reporting on a
path the assistant never takes. Ask the provider instead, and cache the
two routes under separate keys.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DWCmRQ1ry26JnhA79s1ZKH
* fix(ai-health): confirm a tools refusal before blaming the model
A client error on the tools request can mean "your tools payload" or "your
request, tools or not" — an invalid schema, a route the endpoint does not
serve, a model it will not run. Splitting those on the status code alone
still put a 422 from an endpoint contract on the model's account and told
the operator to go find another one.
The probe now confirms it: when the tools request comes back a client
error, it asks again with the tools taken off. Only if that lands is the
tools payload what was turned down, and the probe says so with its own
failure code — provider-agnostic, and no reading of error text for the
word "tool", which would only ever fit the provider it was written
against. Statuses that mean "not now" or "not you" (401, 402, 403, 408,
429) never get a second ask. `AiHealth` now just reports the probe's
verdict instead of inferring one from the status.
The troubleshooting fix no longer points at an OpenRouter free tier:
those providers commonly log prompts and completions for training, and
every assistant tool call carries accounts, transactions, and holdings.
It points at the model recommendations already in this doc, and says why
free tiers are the wrong place to look.
Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DWCmRQ1ry26JnhA79s1ZKH
---------
Co-authored-by: Claude <noreply@anthropic.com>
* security: throttle every credential-guessing endpoint, fix duplicate Rack::Attack middleware
Follow-up on #1087 (Findings H4, M7). PR 4 of the 6-PR series.
Enumerated every endpoint that checks a password, TOTP code, or backup
code (grepped for User.authenticate_by/#authenticate/#verify_otp? across
app/controllers, not just the ones named in the issue) — six in total,
none previously throttled:
- POST /sessions (SessionsController#create) — web login
- POST /mfa/verify (MfaController#verify_code) — TOTP + backup codes
(#verify_otp? handles both internally, so no separate endpoint to add)
- POST /password_reset (PasswordResetsController#create) — also M7
- POST /api/v1/auth/login (Api::V1::AuthController#login) — mobile/API
- POST /oidc_account/create_link (OidcAccountsController#create_link) —
password check gating SSO-identity linking, not sign-in; easy to miss
grepping routes.rb for "session"/"login"
- POST /api/v1/auth/sso_link (Api::V1::AuthController#sso_link) — same
as above for the mobile app
Each gets two throttles (ip AND normalized email, or ip AND the MFA
step-up's session-bound user id where there's no email param) so an
attacker can't bypass by rotating IPs against one target, nor by
spraying many emails from one IP — Rack::Attack requires every matching
throttle to pass. limit: 10/minute, matching the existing oauth/token
and admin/ip throttles already in this file.
Also fixed a latent, unrelated-but-adjacent bug found while confirming
these throttles would actually enforce the limits documented in their
own comments: config/application.rb had an explicit `config.middleware.use
Rack::Attack` alongside the gem's own Railtie doing the same thing (`bin/rails
middleware` listed it twice) — every throttle's counter was incrementing
twice per request, so all of them, old and new, were silently firing at
half their documented limit. Removed the redundant explicit registration.
Race-condition check (per standing instruction): Rack::Attack's counter
increments are atomic within its cache store, so concurrent requests at
the threshold don't undercount. No new race introduced.
New tests in test/integration/rack_attack_test.rb:
- Registration checks for all 6 new throttle keys (existing convention
in this file).
- Direct block-level tests for the discriminator logic (right path
matched, right value extracted, blank/missing input produces nil
rather than a bogus key) — Rack::Attack's cache backs onto Rails.cache,
which is :null_store in the test environment, so no amount of request
volume in a normal integration test can ever actually trip a throttle
here; calling the registered block directly against a constructed
Rack::Attack::Request is what makes the assertions meaningful instead
of just checking string keys exist.
- Regression test asserting Rack::Attack appears exactly once in the
middleware stack.
Verified against the NAS sure_test_web container: full restart, bin/rails
test (8/8 rack_attack tests green; ran the full test/integration suite
plus sessions/mfa/password_resets/api-auth/oidc_accounts controller tests
too — 6 pre-existing failures, confirmed identical on the unmodified
baseline before concluding they're the known WebAuthn-RP-ID-mismatch and
AI-disabled environmental categories, not a regression), bin/rubocop,
bin/brakeman. Also did a live demonstration against the running container
(which runs RAILS_ENV=production, where Rack::Attack is actually enabled):
12 rapid POSTs to /sessions with bad credentials — requests 1-10 got 422,
11 and 12 got 429, exactly matching limit: 10. Container restored to its
original state and restarted afterward.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* security: extract email from JSON bodies for credential-guess throttles
Rack::Attack runs before Rails' JSON parameter parsing, so request.params
only exposed query/form fields. The documented api/v1/auth/login and
.../sso_link JSON format bypassed the per-email throttle entirely,
letting an attacker rotate IPs against one target's account.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* security: guard JSON email peek against non-rewindable input and non-object payloads
Rack 3 no longer requires rack.input to be rewindable, and a bare
JSON.parse(body)["email"] raises NoMethodError on valid non-Hash JSON
(null, arrays, scalars) — either would 500 the request instead of just
skipping the email throttle.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* security: assert non-rewindable JSON bodies stay readable by the controller
Only checking that the throttle discriminator returned nil left a gap: an
implementation that read the body and then discarded the result on error
would pass the same assertion while leaving the controller with an
exhausted stream.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* security: close credential-guessing throttle bypass via format-suffixed paths
request.path == "/sessions" (etc.) never matched "/sessions.json", which
Rails still routes to the same controller action since none of these
routes are declared format: false. Match the optional format suffix
explicitly instead, per jjmata's review on PR #3263.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
* security: match Rails' actual format-segment charset in credential_guess_path
\w excludes hyphens, but Rails' default (.:format) segment matches
[^./?]+, which does include them — e.g. "/api/v1/auth/login.rate-limit"
still routed and bypassed the throttle. Match the real charset instead,
per CodeRabbit's follow-up on PR #3263.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
---------
Co-authored-by: Gerald <248542187+gfr-free@users.noreply.github.com>
Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
* fix(plaid): don't request liabilities on non-liability links
Plaid's Link flow filters out any institution that doesn't support every
product in the link token's requested + additional-consented set. Sure always
requested 'liabilities' as an additional consented product for every account
type, which hides investment-only institutions like E*TRADE (ins_129473) that
don't offer Plaid's liabilities product.
Only include 'liabilities' when the account being linked is itself a liability
(CreditCard/Loan). Investment and Depository links no longer force it, so
E*TRADE and similar institutions become linkable.
* docs(plaid): document product-selection methods
---------
Co-authored-by: terafin <terafin@users.noreply.github.com>
