mirror of
https://github.com/we-promise/sure.git
synced 2026-07-27 04:02:15 +00:00
375dd060dcabcd3e54fd91b7b7a2eacdab9ecb6f
343 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
RealDiligent
|
98bf563458 |
fix: prevent AddAmountToTransfers migration aborting on legacy transfer data (#2795)
The backfill computed amount = outflow_entry.amount - source_fee_amount and then added a check constraint requiring amount >= 0. Historical rows that predate the modern sign convention (negative outflow amounts) or carry fees larger than the entry amount produce a negative principal, so the constraint aborts the migration with PG::CheckViolation — blocking db:prepare and the entire upgrade for affected self-hosters. Normalize with ABS and clamp at zero in the backfill. Rows that already satisfied the old expression are unchanged (ABS is a no-op for positive amounts); only previously-failing rows now migrate instead of killing the upgrade. Fixes #2653 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
sure-admin
|
3d40d93bba | fix(schema): restore enable banking account columns | ||
|
Guillem Arias Fauste
|
9d4b4bc27c |
feat(jobs): reap imports and exports stuck by lost background jobs (#2681)
* feat(jobs): reap imports and exports stuck by lost background jobs A hard-killed worker (OOM, SIGKILL during deploy) loses its in-flight Sidekiq job permanently. Sync is the only model with a stale sweep; everything else wedges in a non-terminal status forever: - Import stuck "importing"/"reverting" (e.g. #2274) - ImportSession stuck "importing" — unrecoverable, publish_later refuses to re-publish while importing - FamilyExport stuck "pending"/"processing" — exports index polls every 3s indefinitely - PdfImport's AI-processing claim held forever (ProcessPdfJob's reclaim only runs if the job is redelivered) - Provider activities_fetch_pending flags stranded when a self-rescheduling fetch chain loses a link SyncCleanerJob (existing hourly cron) now sweeps all of them, each isolated so one failure doesn't block the rest, and records every reaped record as a DebugLogEntry with the family attached. Idempotency guards so a stray or redelivered job cannot corrupt a record the reaper (or a user retry) has since moved on: - Import#publish skips complete/reverting/revert_failed imports — a redelivered ImportJob after completion double-applies data for import types without row dedup. Failed imports deliberately stay publishable: their transaction rolled back, so a re-run is a retry. - FamilyDataExportJob refuses terminal exports but still allows processing ones through, since graceful-shutdown redelivery is what completes them. Reaper thresholds (6h imports, 2h exports) key off updated_at and dwarf legitimate runtimes, so live jobs are not swept in practice. * fix(jobs): make the reapers race-safe and commit-aware Review feedback on #2681 (jjmata, Codex, CodeRabbit): - Every sweep now mutates under record.with_lock with a staleness re-check, mirroring the guard Sync#perform gained in #2680 — a job finishing between the sweep query and the write can no longer be clobbered mid-flight - Import.clean distinguishes which side of import!'s single transaction the worker died on: rows attached means the data committed and only the status write was lost, so the record is finalized as complete (marking it failed invited a re-publish that double-imports types without row dedup, e.g. TradeImport); no rows means a clean rollback and the failed/try-again path stays - PdfImport.clean applies the same split: no rows → the AI-extraction claim died, reclaim to pending; rows → the publish died post-commit, finalize complete instead of letting the same extracted rows be published twice. Stuck PdfImport reverts (previously unswept by either clean) now go to revert_failed like every other import - ImportSession.clean reconciles chunks whose import! committed but never got the complete-status write before failing the session, so re-publish skips them instead of duplicating their rows via SureImport's split path - Import#publish redelivery skip is captured via DebugLogEntry instead of Rails.logger so support can see it in /settings/debug - The interrupted-error copy is i18n-backed (imports.errors.interrupted) * fix(jobs): round-2 review feedback on the reapers - Import.clean excludes session-owned chunks (import_session_id: nil) — SyncCleanerJob runs it before ImportSession.clean, so it could finalize or fail a SureImport chunk outside the session flow that owns its lifecycle. Regression test added (CodeRabbit major) - family.sync_later moved outside the row-lock transaction in both reap paths — Rails doesn't defer enqueues to after-commit by default, so Sidekiq could pick the job up before the status write was visible - Per-record rescue in Import.clean / PdfImport.clean so one bad record doesn't abort the rest of the hourly sweep - ImportSession.clean uses the importing? enum predicate; reconcile_committed_chunks! iterates with each (default-ordered association made find_each warn, and chunk counts are tiny) * fix(jobs): address round-3 reaper review (isolation + commit signal) Per-record rescue isolation for the three sweeps that lacked it — FamilyExport.clean, ImportSession.clean, and SyncCleanerJob's activity-flag loop — mirroring Import.clean/PdfImport.clean. One bad record (validation error, DB blip) no longer aborts the rest of that sweep for the hour; the activity-flag guard is per-record so a failure also stops skipping the models that follow. data_committed? now covers Category/Rule/Merchant imports, whose records hang off the family rather than the import (no entries or accounts), via the new Import#committed_by_named_records? helper. A committed one is reaped to complete instead of a retryable failed; nameless RuleImport rows carry no stable key, so a nameless-only file has no commit signal and stays retryable. * fix(schema): drop duplicate enable_banking_accounts columns from merge The merge of main into this branch re-appended product, credit_limit, and identification_hashes after updated_at, so schema.rb declared each twice and db:schema:load raised "you can't define an already defined column 'product'". Removed the duplicate declarations (kept the new treat_balance_as_available_credit column); the test database loads again. |
||
|
jdcdp
|
849578a84a |
Add support for trading212 integration in investments sync (#2513)
* Add support for trading212 integration in investments sync * Respect Trading 212 history rate limits * Replace hex colors with design tokens * Changed withdrawal to withdraw * Address PR review feedback for Trading 212 integration - Replace all Rails.logger calls with DebugLogEntry.capture across syncer, importer, processor, provider, controller, and unlinking - Fix destroy action to surface unlink errors via DebugLogEntry - Replace hardcoded 'GBP' currency fallback with Current.family.currency - Replace hardcoded English status strings in Syncer with i18n - Add config/initializers/trading212.rb with DEBUG_RAW ENV toggle - Add comprehensive test coverage: item, account, data_helpers, holdings_processor, activities_processor, importer, syncer, and controller tests - Rewrite syncer and controller tests to match actual interfaces * Fix tests and bugs found during local test execution - Fix standard_ticker: empty string caused nil.upcase via [].first - Fix parse_date: DateTime < Date, so when Date caught DateTime first - Fix ActivitiesProcessor/HoldingsProcessor: missing instruments_map in DataHelpers caused NameError when processing dividends - Fix test: security fixture ticker collision with test data - Fix test: sync_status_summary i18n matching in assertions - Fix test: trading212_provider ConfigurationError test path - Fix test: controller invalid params needs Turbo-Frame header - Fix test: syncer test uses stubs instead of strict expects 116 tests, 257 assertions, 0 failures, 0 errors * Keep raw provider response bodies out of exception messages. * Fix Secrets leak into page HTML, prevents the API key/secret from appearing in the HTML source while keeping the "leave blank to keep existing" UX. * Address review findings: env gate, sync test, uniqueness test, destroy flow - Gate TRADING212_DEBUG_RAW behind Rails.env.local? so staging/production cannot accidentally enable raw payload logging - Assert SyncJob enqueue in sync controller test, not just redirect - Fix cross-item uniqueness test to actually use two different items - Remove rescue in destroy so unlink failures stop the flow (matching Brex/Akahu pattern) instead of proceeding to destroy_later silently * Add Trading212 tables to db/schema.rb for CI test database CI runs db:test:prepare which loads db/schema.rb, not migrations. Without these table definitions, fixture loading fails with PG::UndefinedTable: relation "trading212_accounts" does not exist. * Removed lint complaint * Register Trading212 in ProviderConnectionStatus::PROVIDERS Fixes CI failure: test_provider_registry_covers_syncable_family_provider_item_associations * Fixed bad merge --------- Signed-off-by: jdcdp <47483528+jdcdp@users.noreply.github.com> Co-authored-by: jdcdp <jdcdp@cdm4.net> |
||
|
kianrafiee
|
d267288a45 |
Add RentCast & Realie integration for automatic property data and valuations (#2727)
* Add RentCast and Realie AVM providers for property valuation Adds Automated Valuation Model (AVM) provider support so self-hosted users can create property accounts from a US address lookup instead of entering details manually: - Provider::Rentcast and Provider::Realie clients, each fetching the property record (type, year built, square footage) and value estimate in a single API request, registered under a new :property_valuations registry concept - API key fields (encrypted, ENV-overridable) with monthly usage display in Settings > Self-Hosting under "Property Valuation Providers" - New property flow: when a provider key is configured, the method selector offers "Add via RentCast/Realie" alongside manual entry; the lookup form reuses the manual flow's localized address fields and creates the account active with fetched attributes, valuation as balance, and the address saved - SyncPropertyValuationsJob refreshes linked property valuations once a day (never hourly) via config/schedule.yml; each provider enforces its monthly request cap (RentCast 50, Realie 25) with a calendar-month counter shared between creation lookups and refreshes Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Allow ENV override of AVM provider monthly request caps The RentCast (50/month) and Realie (25/month) budgets default to the free tier limits but can now be raised for paid plans via RENTCAST_MAX_REQUESTS_PER_MONTH / REALIE_MAX_REQUESTS_PER_MONTH, mirroring the AlphaVantage and Tiingo request limit overrides. The settings descriptions and usage display reflect the effective cap. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address review feedback: durable request counters, safer refresh job - Move monthly AVM request counters from Rails.cache to a new provider_request_counts table with atomic upsert increments, so the hard budget caps survive cache eviction, restarts, and Redis flushes - SyncPropertyValuationsJob: only mark a property synced when the balance update succeeds (wrapped in a transaction), process stalest valuations first so a tight budget goes where it matters, and report failures via DebugLogEntry.capture instead of Rails.logger - Realie: reject lookups whose returned city/ZIP contradict the entered address (street+state queries can match the wrong city); document that numeric use codes are unpublished and leave the subtype unset - Add Faraday open/request timeouts to both provider clients - Localize all user-facing provider error messages (config/locales/models/provider/en.yml) - Add a check constraint restricting properties.avm_provider to known providers - Use the min-h-80 scale token instead of min-h-[320px] - Tests: exact-argument expectations on the lookup stub, provider stubs in the no-provider test, RentCast/Realie API key update tests, ProviderRequestCount unit tests, failed-balance regression test Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Stub AVM provider registry lookups in settings system test Settings::HostingsController#show now resolves the RentCast and Realie providers for usage display; the system test's partial get_provider stubbing treats the new lookups as unexpected invocations without these. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Address second-round review: candidate scan, provider reuse, shared row partial - Realie: when the address lookup returns multiple candidates, pick the first one consistent with the entered city/ZIP instead of judging only the first array element; the location-mismatch error now only fires when no candidate matches - SyncPropertyValuationsJob: resolve one provider instance per key for the whole run, so the per-instance request throttle actually spaces requests across properties instead of resetting on each iteration - Extract the AVM method selector row into a shared partial so the manual and provider options render one shape Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add retry middleware and sync index from review feedback - Both AVM clients now retry transient connection failures (same Faraday retry config as Tiingo) so a network blip doesn't burn one of the tight monthly budget's requests - Partial index on properties(avm_provider, avm_last_synced_on) backing the daily sync job's filter and stalest-first ordering Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Key the AVM sync index on avm_last_synced_on The daily job orders by avm_last_synced_on ASC NULLS FIRST across all AVM-linked properties, so leading the partial index with avm_provider prevented it from serving the sort. Rekeyed on avm_last_synced_on with matching null ordering; the partial predicate still covers the filter. Amended in place since the migration is unmerged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Map Realie numeric use codes; skip incomplete addresses in sync job - Realie documents its numeric use codes at docs.realie.ai/api-reference/feature-key (earlier skip reason was wrong — the table exists). Map the residential/land/agricultural codes to Property subtypes, leaving codes without a subtype equivalent (e.g. 1006 mobile/manufactured) unset like the RentCast mapping. - The daily job now skips properties whose address is missing a street or state before resolving a provider, logging a warn-level DebugLogEntry, so a malformed address can't burn a monthly-budget request every day. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Guard currency mismatches in refresh; reset AVM keys in test teardown - The daily job now skips (with a warn-level DebugLogEntry) properties whose account currency no longer matches the provider's valuation currency, checked via the concept's valuation_currency before spending a request — writing a USD valuation into a re-currencied account would corrupt the balance - Hostings controller test teardown now clears rentcast_api_key and realie_api_key alongside the other cached global settings Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Validate AVM lookup inputs before spending a provider request The form marks name and address fields required, but a forged or JS-less submission bypasses that and would burn one of the tight monthly-budget requests on a lookup that can't produce a property. Property::AvmImport now validates name and the full address locally (localized error) before calling the provider. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add preview-and-confirm step to the AVM property lookup RentCast's AVM endpoint is location-based: a plausible but nonexistent address still geocodes and returns an area-derived estimate with no property record behind it. Instead of silently creating an account from that, the lookup now shows what the provider returned — type, year built, area (each "Unknown" when missing), and the estimated value — with an explicit notice when no property record was found, and only the user's confirmation creates the account. The confirm step reuses the fetched data via the form (sanitized server-side), so the flow still costs exactly one provider request. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Require a complete US address before daily valuation refresh The job's guard only required street and state, but a later-blanked city or ZIP silently disables the Realie wrong-city check (blank entered fields count as "no mismatch"), so a refresh could accept another property's valuation. The guard now mirrors the import-time completeness check and also skips addresses edited to a non-US country. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Sign the AVM preview payload; blur valuation in privacy mode - The confirm step now rebuilds everything from a signed, 1-hour message-verifier token generated at lookup time, proving the lookup (and its counted provider request) actually ran — a direct confirm POST with fabricated data can no longer create provider-linked properties that the daily refresh job would then spend quota on. Forged/expired tokens re-render the lookup form with a clear error. - The preview's market value now carries the privacy-sensitive class so privacy mode blurs it like other account amounts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Treat zero Realie valuations as absent; bind preview token to family Review follow-ups from PR #2727: - Realie returns modelValue: 0 when it can't produce an AVM estimate; zero now falls back to the assessed market value instead of syncing a $0 balance, and an all-zero record errors out. - Monetary values now parse via BigDecimal(value.to_s) rather than Float#to_d, avoiding binary float rounding artifacts in balances. - The signed AVM preview token's purpose is scoped to the family that ran the lookup, so a leaked token can't be replayed cross-family. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Refresh AVM property valuations monthly instead of daily Per @jjmata's review on #2727: valuations change slowly and providers enforce tight monthly request caps, so a daily cron adds no value. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
William Wei Ming
|
f210d1ca4c |
Perf/accounts controller index optimization (#1926)
* fix Prism issue - assigned but unused variable plaid_item * measurement to improve accounts_controller__index performance based on skylight * fix FEEDBACK - Stop eager-loading full sync histories on accounts index * fix FEEDBACK - Remove sticky memoization from SimplefinItem#accounts * fix lint error in PR review * fix FEEDBACK - Avoid returning map misses as authoritative results * fix test error * fix FEEDBACK - Avoid redundant query by using already-loaded @manual_accounts * Address maintainer review on accounts index sync preloading Memoize SimplefinItem#accounts, align syncing? with Sync#visible?, and add fallback regression tests. * Add composite index for sync DISTINCT ON queries Support latest_by_syncable ordering with syncable_type, syncable_id, created_at DESC, and id DESC. * fix error in dockerfile.preview file * Suppress Pipelock false positives on CI database fixtures. Pipelock 2.7.0 full-repo audit flags ephemeral postgres:// URLs in workflow env blocks. Re-apply inline suppressions lost in the main merge. * Address review: guard partial Current sync maps and drop unrelated diff Add key? checks so partially populated Current sync maps fall back to DB queries. Revert Dockerfile.preview and pipelock.yml changes unrelated to the accounts index N+1 optimization. * fix(sync): fully populate Current sync maps for all preloaded syncables * fix(ci): skip scheduled preview cleanup on forks Only run the hourly Cloudflare preview cleanup on we-promise/sure, where the required secrets exist. * Remove schema.rb Postgres-version churn, keep only new syncs index Reset db/schema.rb to upstream/main and re-add only the index_syncs_on_syncable_and_created_at_and_id index. The prior diff included ~30 lines of noise (check-constraint reformatting, virtual column reformat, and column reorderings) caused by dumping under a different PostgreSQL version, which obscured the single intended schema change and invited merge conflicts. Co-authored-by: Cursor <cursoragent@cursor.com> --------- Co-authored-by: Cursor <cursoragent@cursor.com> |
||
|
Max Barbare
|
51c93649da |
feat(snaptrade): replace device-flow OAuth with authorization-code + PKCE flow (#2747)
* feat(snaptrade): replace device-flow OAuth with authorization-code + PKCE flow Squashed from 16 commits on snaptrade-oauth-apps for a clean rebase onto current upstream/main ahead of opening a PR. * fix(snaptrade): address PR #2747 review feedback on OAuth PKCE flow - Remove unreachable dead-code guard in import_latest_snaptrade_data - Guard apply_oauth_tokens! against a malformed payload missing access_token - Wrap token endpoint network errors in ApiError and retry like data calls - Remove unused Provider::Snaptrade#revoke_token! instance method - Preserve return_to/accountable_type through the SnapTrade portal callback so the account-linking flow no longer drops users back to accounts_path - Show the real absolute OAuth callback URL in self-hosted setup instructions - Refresh brakeman.ignore fingerprint for the connect redirect after the return_to/accountable_type params were added Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y8SCCmKX6RphB5E73WSUQQ * fix(snaptrade): don't retry non-idempotent OAuth/API requests CodeRabbit flagged that Provider::Snaptrade retried OAuth token exchanges/refreshes and all API POST/DELETE calls (get_connection_url, delete_connection) after timeouts/connection failures. If the response is lost after SnapTrade already consumed a single-use auth code, rotated the refresh token, or applied a POST/DELETE, replaying the request either fails with invalid_grant on a token that actually succeeded, or risks duplicate side effects. Retries are now limited to GET requests; OAuth token requests and non-GET API calls translate a network failure straight into an ApiError without replay. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrrGkgSBEqhjjBmmH1fcXL * fix(snaptrade): stop querying non-deterministically encrypted token via empty-string compare CodeRabbit flagged that the syncable scope's where.not(oauth_access_token: [nil, ""]) re-encrypts "" with a random IV on every query, so the "" comparison can never match a stored ciphertext and is a silent no-op. No code path ever persists oauth_access_token as "" (only nil or a real token via apply_oauth_tokens!), so the exclusion is unnecessary -- narrowed the scope to a plain NULL check, which encryption handles transparently since nil is never encrypted. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrrGkgSBEqhjjBmmH1fcXL --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> |
||
|
Juan José Mata
|
c6eb7cdeed |
Revert "Refactor application workflows and update test coverage"
This reverts commit
|
||
|
Juan José Mata
|
565e049f89 | Refactor application workflows and update test coverage | ||
|
Markus Laaksonen
|
c4ac6a365f |
feat(sync): add toggle to parse Enable Banking CC balance as available credit (#2512)
* feat(sync): allow EnableBanking credit card balances to be parsed as available credit * feat(ui): expose Enable Banking balance interpretation toggle on credit card form Adds a toggle to the credit card edit dialog, shown only when the account is linked to an Enable Banking provider account. Toggling persists treat_balance_as_available_credit and enqueues an item sync so the balance is reinterpreted immediately. * fix(sync): keep existing balance when credit limit is missing for available-credit cards When treat_balance_as_available_credit is on and the API omits credit_limit, the reported balance is known to be available credit, so recording it as debt fabricates a liability. Skip the balance write in that case and only update the available credit metadata. Also extract the credit card branching into a helper and include provider_key, account_provider and metadata in the debug log entries so support can filter /settings/debug by the affected connection. * refactor(ui): move provider lookup to Account and label the toggle for assistive tech Adds Account#provider_account_for so the view no longer queries account_providers directly, and wires aria-labelledby from the toggle to its visible label. * fix(ui): apply Enable Banking setting only after a successful account update Runs the provider flag update after super and only on the redirect path, so a failed account save no longer persists the flag or enqueues a sync. Also permits the enable_banking params so malformed input is filtered instead of raising. * test(sync): extract relink helper in Enable Banking processor test * feat(sync): fall back to manually set available credit as the credit limit When the toggle is on, the accountable's available_credit field now holds the credit limit (API-provided, or user-entered when the API omits it) and is never overwritten with the reported balance. This lets users whose bank reports available credit without a credit limit compute the outstanding debt by entering their limit in the Available credit field, while keeping the field stable across syncs so a stale reported balance can never be mistaken for a limit. --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
kianrafiee
|
c54369bd2d |
Add send_email_notification rule action (#2527)
* Add send_email_notification rule action Adds a new rule action that emails a digest of transactions matching a rule. Re-syncs re-apply every active rule to all in-window matches, so a notification_deliveries table (unique on rule_id + transaction_id) backs deduplication and a per-rule watermark: - Rule::ActionExecutor::SendEmailNotification plucks candidate ids, drops ones already in notification_deliveries, records the remainder BEFORE enqueuing (fail-safe: a crash suppresses rather than double-sends), and returns the count of newly-notified transactions. - RuleEmailNotificationJob loads the rule + transactions and delivers the digest via RuleNotificationMailer. - Creating the action pre-seeds all currently-matching transactions as already-delivered (after_create_commit), so the rule only emails about transactions appearing after the action exists. Dedup keys on the DB row id, not provider identity, so a re-ingested transaction (new id) may re-notify; accepted as benign. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Add view transactions link to rule notification digest email Include a "View transactions" CTA (APP_DOMAIN/transactions) in both the HTML and text versions of the rule notification digest email. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address review feedback on rule email notifications - Restrict digest delivery to family admins only (drop non-admin fallback) - Make NotificationDelivery.record_for return only inserted ids and enqueue off that result, preventing duplicate digests under concurrent rule runs - Seed the notification baseline when an existing action is changed to send_email_notification, so historical matches are not emailed - Strengthen tests: assert the transactions CTA/link in the digest and the enqueued job's transaction-id args Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Include super_admin owner as rule digest recipient The admin-only recipient lookup used find_by(role: :admin), which excluded super_admin owners. A self-hosted family is commonly a single super_admin, so the digest was silently skipped (NullMail no-op) and no email was sent. Match the recipient on %w[admin super_admin] (the same pattern used elsewhere, and consistent with User#admin?), while still excluding regular members/guests. Add tests for the super_admin recipient and the no-admin skip path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Add mixed-role digest recipient test Cover the case where a family has both an admin and a super_admin. The recipient lookup uses find_by(role: %w[admin super_admin]) with no ORDER BY, so which one is returned is non-deterministic; the contract is only that the recipient is an admin-level user (never a member). Assert recipient.admin? rather than a brittle precedence between the two roles. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Deliver rule digest via deliver_later and order in DB Use deliver_later so a slow/flaky SMTP connection doesn't tie up the Sidekiq worker, and push the entry-date sort into the query instead of materializing and sorting the result set in Ruby. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Replace raw hex colors with email-safe design tokens in digest template The rule digest email hardcoded Tailwind slate-100/200 hex values for table borders, which aren't part of this project's design system. Resolve to the actual border-primary/border-secondary token values and centralize them as a reusable .email-table class in the mailer layout. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Mike Lloyd
|
fc0581fba3 |
Add per-account toggle to disable automatic transaction categorization (#2636)
* Add per-account toggle to disable automatic transaction categorization Adds an `enable_category_matcher` boolean (default: true) to accounts so users can opt out of Plaid's automatic category suggestions on a per-account basis. When disabled, newly synced transactions arrive uncategorized so rules or manual assignment take precedence. - New migration adds `enable_category_matcher` column (default true, null: false) - `PlaidEntry::Processor#matched_category` gates the CategoryMatcher call on the account flag - Toggle rendered in the account edit modal for linked accounts (saves via main form submit) - `AccountableResource#account_params` permits the new field - `AccountsController#toggle_category_matcher` action added for potential API use - i18n strings added for label and hint text - Unit test covers the disabled-matcher path Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Only show category matcher toggle for Plaid-linked accounts Only PlaidEntry::Processor honors enable_category_matcher, but the toggle rendered for every linked account, silently doing nothing for other providers. Adds Account::Linkable#supports_category_matcher? (covering both the legacy plaid_account_id link and AccountProvider rows) and gates the form toggle on it. The SimpleFIN TODO now also points at this helper so the toggle appears once SimpleFIN matching lands. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Add controller tests for category matcher toggle persistence and rendering Covers the two paths flagged in review as untested: the flag persisting through the shared AccountableResource#update action via account_params (both disable and re-enable), and the edit form rendering the toggle only for accounts where supports_category_matcher? is true. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Signed-off-by: Mike Lloyd <49411532+mike-lloyd03@users.noreply.github.com> Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
Guillem Arias Fauste
|
5105752bbd |
feat(sync): family-facing cancellation for syncs, imports, and exports (#2685)
* feat(sync): family-facing cancellation for syncs, imports, and exports Users had no way to stop or recover any background operation: a mistaken "Sync all" runs to completion, and an import or export whose job died (hard worker kills lose in-flight Sidekiq jobs) wedges with a spinner forever. Sync cancellation (cooperative — nothing is ever killed): - New syncs.cancel_requested_at column. Only the cancelled sync carries the flag: pending descendants are marked stale immediately (their queued jobs no-op via the existing may_start? guard), while descendants whose jobs are already executing finish their work honestly. - Family::Syncer stops fanning out child syncs once the flag is set (fresh read per iteration — the flag comes from the web process). - Finalization resolves a cancel-requested sync to stale instead of completed, which also skips post-sync (transfer matching, rules, broadcasts) via the existing stale gate. - The `visible` scope excludes cancel-requested syncs, so spinners clear immediately and — fixing a latent bug this feature would have amplified — sync_later no longer piggybacks a new sync request onto a dying sync it would silently swallow. - Cancel button appears next to "Sync all" on the accounts page while a family sync is visible. SyncsController#cancel scopes through Sync.for_family with resource_owner, so cross-family ids 404 and account-level syncs respect per-user account access. Stuck import/export self-service: - Import#force_fail! / FamilyExport#force_fail!: allowed only once the record has been idle past PRESUMED_LOST_AFTER (1 hour — dwarfs any legitimate run), and applied inside with_lock with a status re-check, so a job finishing between page render and button click wins. Imports fail into the existing retry path (reverting -> revert_failed keeps the revert retryable); PdfImports release their processing claim back to pending; exports fail so a new one can be created. - "Mark as failed" buttons appear on the imports/exports index rows only when a record is presumed lost, behind the pages' existing permission gates (statement-import permission for imports, admin for exports). * fix(sync-cancel): cascade pending cancels, guard late finalizers, scope provider syncs Review feedback on #2685 (CodeRabbit, Codex): - request_cancel! now cascades finalization for pending syncs too: a pending child resolved to stale never runs its job, so nothing else would ever call finalize_if_all_children_finalized — its waiting parent hung in syncing until the 24h sweep (CodeRabbit critical) - SimplefinItem::Syncer#mark_completed re-reads the sync under a row lock and skips finalization once cancellation was requested or the row went terminal — its in-memory copy predates the cancel, and the unguarded complete! (plus the raw status fallback) resurrected a cancelled sync and re-ran post-sync (Codex) - Cancelling provider-item syncs now requires admin: for_family's resource_owner only scopes the Account branch, so a restricted member could cancel admin-managed provider syncs spanning accounts they cannot see. Family- and account-level syncs stay member-cancellable, matching the buttons the UI shows (Codex) - Lost-import error copy moved behind i18n (imports.errors.presumed_lost), resolved at call time (CodeRabbit) - Tests: pending-child cancel finalizes the parent; late provider complete! cannot resurrect a cancelled sync; provider-sync cancellation is admin-only * fix(sync-cancel): capture the skipped-finalization case via DebugLogEntry CodeRabbit round-2: the mark_completed skip (cancelled/terminal sync) is support-relevant — record it in the super-admin debug UI with the family and provider attached instead of a raw Rails.logger line. category: provider_sync, matching the other provider syncers. |
||
|
Blaž Dular
|
826c4a356e |
feat(bank-sync): Wise integration (#2433)
* feat(wise): add Wise integration with JAR savings account and activity support
- Add WiseItem/WiseAccount models with full sync pipeline (importer, syncer, processor)
- Detect income vs expense using targetAccount == recipientId from borderless accounts API
- Support JAR (SAVINGS) accounts with totalWorth balance and savings subtype
- Fetch JAR activity via profile activities API (INTERBALANCE, BALANCE_CASHBACK, BALANCE_ASSET_FEE)
- Route INTERBALANCE activities to both JAR and STANDARD accounts and link as Transfer records
- Add provider connection status registration, routes, views, and i18n
- Add migration for wise_items and wise_accounts tables
- Add tests for WiseAccount, WiseEntry::Processor, WiseActivity::Processor, WiseItem::Importer, and WiseItem#link_jar_transfers!
* chore(lint): add ignore to security scan (false positive)
* fix(wise): address PR review feedback on activity routing, HTML stripping, rate limiting, and scope extraction
- Replace title string matching in activity_for_account? with resource.id vs balance_id comparison to avoid breakage when users rename JAR accounts on Wise
- Replace gsub(/<[^>]+>/, "") with ActionController::Base.helpers.strip_tags to safely handle user-controlled HTML-like content
- Wrap paginated API calls in with_rate_limit_retry (up to 3 attempts, exponential backoff) to handle 429 responses during fetch_jar_activities and fetch_transfers
- Extract WiseAccount.unlinked scope and remove duplicated left_joins query from controller
* fix(wise): address PR #2433 review feedback
- Wire @wise_items into AccountsController#index so linked accounts appear on /accounts
- Fix find_wise_account_for_linking to preserve .active scope via .merge instead of .then
- Fix cross-currency incoming transfer amount to use targetValue instead of sourceValue
- Add missing select_profiles.session_expired locale key
- Add missing account_name interpolation to link_existing_account success notice
- Use i18n for profile type labels in profile_display_name
- Trigger wise_item.sync_later after account linking in all three linking actions
- Isolate fee transaction failure so it no longer aborts the main transfer import
- Use bare raise in WiseActivity/WiseEntry processors to preserve original backtrace
- Re-raise in WiseItem::Unlinking to prevent silently orphaned Holdings
- Fix avatar badge to use design system tokens (bg-container-inset, text-primary)
* fix(wise): fix INTERBALANCE routing and encrypt pending token in session
* fix(wise): replace hand-rolled buttons/links with DS::Button and DS::Link
Address repeated sure-design DS drift findings: migrate all manual
Tailwind button_to and link_to calls in Wise views to DS::Button
(outline/outline_destructive variants) and DS::Link (secondary variant).
Add missing provider_panel.disconnect locale key for the disconnect button text.
* fix(wise): use render_provider_panel_error in create instead of missing new template
* fix(wise): add missing comma in PROVIDERS array
CI failed with a SyntaxError because the questrade entry wasn't
comma-terminated before the wise entry.
* chore(wise): re-add pipelock:ignore for pending token param
Lost when the token source moved from session to an encrypted params
field in
|
||
|
Juan José Mata
|
66cf9e7f0b |
feat(insights): proactive financial intelligence feed (#2550)
* feat(insights): proactive financial intelligence feed Adds a nightly job that analyzes each family's finances in pure Ruby and surfaces typed, stateful insights on the dashboard and a new /insights page. - Insight model (active/read/dismissed) with a per-family dedup_key unique index so nightly re-runs refresh rows instead of duplicating them - Seven generators (spending anomaly, cash-flow warning, net worth milestone, subscription audit, savings rate change, idle cash, budget health) built on IncomeStatement, BalanceSheet, RecurringTransaction, and BudgetCategory - LLM used as a writer, not a reasoner: Insight::BodyWriter narrates pre-computed facts via the configured provider, with an i18n template fallback so self-hosted installs without API keys work identically; bodies are only (re)written when an insight is new or its numbers changed - GenerateInsightsJob: cron fan-out per family, per-family advisory lock, metadata-diff upsert that preserves read/dismissed state for unchanged signals and reactivates on material change - Dashboard insights_feed section (top 3, collapsible/reorderable) and an /insights feed page with turbo-stream dismissal; viewing the feed marks insights read - Tests for the model, job upsert semantics, and controller flows Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6ZRA6cgCRM4UdFKct3wm4 * fix(insights): address Codex review — expiry, budget metadata, LLM usage - Expire visible insights whose condition cleared: generators declare the insight types they produce, the registry reports which generators ran to completion, and the job expires visible insights of those types whose dedup_key was not regenerated. A crashing generator can't wipe out its healthy insights, and an expired insight reactivates when its condition returns — unlike a user-dismissed one, which stays dismissed. - Include a bucketed budget-spent percent in budget_at_risk metadata so the body refreshes when overall usage moves >=10 points even if the same categories remain flagged. - Pass family to chat_response so LLM narration is recorded in llm_usages. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6ZRA6cgCRM4UdFKct3wm4 * fix(insights): address CodeRabbit review - Skip the mark-as-read write for Turbo hover-prefetch requests (X-Sec-Purpose) so unread badges don't clear before a real visit - Show the New pill on the dashboard feed (active = unread there; the feed never marks insights read) - Filter idle accounts in SQL instead of a per-account exists? loop - Eager-load merchants in the subscription audit query - Widen the advisory-lock key to the signed-bigint range and log when acquisition fails so a skipped nightly run is observable - Mirror the dedup_key unique index as a model validation - Assert the refresh action enqueues for the signed-in family Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6ZRA6cgCRM4UdFKct3wm4 * fix(insights): clear stale lifecycle timestamps on reactivation When an insight resurfaces, the row now leaves no contradictory state behind: the material-change path clears both read_at and dismissed_at, and the expired-recovery path clears read_at. Tests assert the contract, and the dashboard feed test now also locks in the unread "New" badge. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6ZRA6cgCRM4UdFKct3wm4 * fix(insights): drop redundant standalone family_id index Every composite index on insights already leads with family_id, so the auto-created single-column index was pure write overhead. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Y6ZRA6cgCRM4UdFKct3wm4 * fix(insights): stop nightly drift from resurrecting dismissed insights Three generators stored continuously drifting values (projected amounts, starting balance, current net worth) in metadata, and any metadata diff counts as a material change: the body is rewritten (an LLM call when a provider is configured) and read/dismissed state is cleared. Dismissing a cash-flow warning was undone at the next 6:00 run. Metadata now carries only the signal identity plus a coarse bucket, the same damping the budget generators already use; display values live in facts. Also orders the idle-cash pick by balance so the selected accounts don't flip between runs, and documents the dollar-scale threshold assumption on the relevant constants. * fix(ds): map DS::Button aria_label into the aria hash A bare aria_label: option reaches the tag helpers as a literal aria_label attribute (Rails only dasherizes the nested aria: hash), so the icon-only fallback overrode it and screen readers announced the insight dismiss button and the popover trigger as "X". * fix(insights): gate LLM narration behind AI consent The nightly job runs unprompted for every family, so narration now requires someone in the family to have AI enabled: consent to share financial facts with the provider, and a cost cap in managed mode. The template fallback keeps behavior identical otherwise. Narration failures are captured via DebugLogEntry so support can see them. Adds BodyWriter coverage, including a template-interpolation test for every generator template key. * feat(insights): rework the dashboard feed The feed rendered full insight cards inside the section shell — the only widget nesting card-on-card — and appended below the fold for every family with a saved section order. It now mirrors the outflows and balance-sheet list idiom (inset well with an uppercase mini-header, white row block, 28px sentiment-tinted icon circles via color-mix on the DS CSS variables, right-aligned key figures) and leads the dashboard for saved orders that predate it. Icon color comes from sentiment, not priority — a savings-rate improvement is high priority AND good news, and must not render red; red is reserved for a projected-negative balance. Rows have no hover wash (cursor plus a gentle icon scale, like the sibling widgets), links to /insights disable Turbo prefetch so the mark-as-read actually fires for mouse users, and the standard widget-size popover offers Half/Full. Full stays the default: the feed is far shorter than any other single-width widget, so a half default leaves a grid hole the masonry cannot backfill. * feat(insights): actionable cards, dismiss undo, live refresh Each row now persists its display facts (new jsonb column) alongside the change-detection metadata. Facts refresh every run without touching the body or user state, which is exactly why they are not part of the material-change comparison. The card gains what the stored data always supported: a type-and-period meta line replacing "x minutes ago", a right-aligned key figure (green only for good news), and a contextual link resolved from the subject ids in metadata — category, account, recurring transaction, budget month — omitted when the subject no longer exists. Dismiss is forgiving: a toast in the notification tray offers undo, and undismissing restores the row as read rather than re-badging it. Milestone insights can never regenerate once dismissed, so this closes a real loss path. Upsert failures are captured via DebugLogEntry. Manual refresh gets feedback: the button swaps to a disabled checking state, the page subscribes to a family-scoped stream, and the job broadcasts the refreshed list and the idle button when it finishes (also after a lock-skipped run, so the button cannot stay stuck). Savings copy is sign-aware: a negative rate reads "you spent more than you earned" with true minus signs. The empty state swaps Lucide sparkles for the brand assistant glyph (DS::EmptyState learns icon_custom:). * feat(insights): top-bar entry with unread count The dashboard feed hides at zero insights and nothing else linked to /insights, so the manual refresh (and the empty state) were unreachable for exactly the families who need them: fresh setups before the first 6:00 run. The sticky top bar travels to every screen and its right cluster had room, so insights get an icon entry there with a monochrome unread badge. Prefetch is disabled on the link for the same mark-as-read reason as the feed. --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Guillem Arias <accounts@gariasf.com> |
||
|
Jestin Palamuttam
|
5803023fa7 |
feat(provider): add native Questrade brokerage provider integration (#2534)
* Add native Questrade brokerage provider integration Adds a per-family Questrade provider so users can sync their Questrade investment accounts (TFSA, FHSA, RRSP, margin, etc.) directly via Questrade's free personal API, with no paid aggregator. - OAuth2 refresh-token flow with single-use token rotation, persisted under a row lock. Tokens self-renew on each sync; the connected panel lets users paste a fresh token if a connection goes stale (no need to disconnect and re-link). - Imports accounts, balances, positions and activities; multi-currency holdings with per-currency cash holdings; Norbert's Gambit journals. - New-account and link-existing-account flows, settings card with desktop-only setup steps, and connect/update/disconnect. - Restricted to Investment account types. Registered in the provider connection-status registry with a syncable scope so it participates in nightly family sync. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix: linting error * fix: refresh token encrypted The OAuth token exchange passed the single-use refresh token as a GET query parameter, so it could leak into URL-based logs (Sentry breadcrumbs, APM spans, debug output). Switch to POST with a form-encoded body (RFC 6749 3.2) so the credential stays out of URLs. Verified Questrade's token endpoint accepts POST (returns 400 for a bad token, not 405). Adds a test asserting the token travels in the body. * Address PR review: authz, data integrity, retries, logging Batch of fixes from the automated PR review: - Require admin for all mutating/linking Questrade actions, and gate existing-account linking through accessible_accounts + write permission (was only Current.family scoped). - Clear requires_update when a fresh token is accepted; use a real 302 redirect (not 422) on full-page failures. - Require refresh_token on all saves (not just create) unless the item is scheduled for deletion. - Migrations target Rails 7.2; questrade_items state columns are NOT NULL. - Background activity dedup keys on Questrade fields (matches the importer) so multiple activities no longer collapse to one. - Persist the normalized account payload; date-scope synthetic cash holdings so daily history is not overwritten. - Retry 429/5xx via a RetryableResponseError instead of hard-failing. - Route provider error bodies to DebugLogEntry instead of Rails.logger / exception messages, so payloads do not leak into application logs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address PR review: atomic linking, sync health, retry loop, USD cash - Wrap account creation + provider linking (+ sync_start_date) in a transaction in both link paths so a link failure rolls back the orphan account. - Surface per-account process/schedule failures in the item sync health instead of always reporting healthy. - Always stamp last_activities_sync once the background fetch completes, so legitimately empty accounts stop being re-queued every sync. - Treat only the account-currency (CAD) balance as primary cash; other currencies (e.g. USD) now surface as separate cash holdings instead of being hidden as primary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address PR review: serialize token exchange, real processor tests - Single-use token race: the SDK now wraps every token exchange (initial and 401 re-auth) in a model-supplied lock that reloads and spends the freshest persisted token (provided.rb#synchronize_exchange). Two concurrent syncs/jobs can no longer double-spend the same refresh token. Adds a test asserting the exchange runs inside the lock with the fresh token. - Replace the all-skipped QuestradeAccount processor test stubs with real fixture-backed tests covering balance anchoring, holdings import, and Buy-trade import (plus blank-symbol / blank-type guards). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Fix indentation of spliced Questrade schema blocks The manually added questrade_accounts/questrade_items create_table blocks sat at column 0 instead of the file 2-space indent, so rubocop flagged them as inconsistent. Re-indent to match the rest of the schema. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Include currency and type in the Questrade activity merge key Two activities that differ only by currency or type could collapse to a single row in merge_activities. Add both fields to activity_key in the importer and the background fetch job so multi-currency imports dedup correctly. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Address review: infer account currency, Encryptable, safer flag clear From @jjmata's review: - Currency: QuestradeAccount#upsert_from_questrade! no longer hardcodes CAD for every account. upsert_balances! now infers the home currency from the per-currency balances (the currency holding the cash wins, ties broken by total equity, default CAD) so USD-denominated accounts are labelled USD and match the right combinedBalances anchor. Adds tests for USD and CAD cases. - QuestradeItem now includes the shared Encryptable concern instead of reimplementing encryption_ready? inline. - QuestradeActivitiesFetchJob#clear_pending_flag is now best-effort so it can never mask (and swallow) the original error in perform's rescue. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * Fix review issues: safe_return_to_path, DebugLogEntry, financial reset, turbo_prefetch, N+1 counts - Add safe_return_to_path to QuestradeItemsController (blocks //evil.com protocol-relative open redirect; same 3-check guard as PR #2591 Wise provider) - Pass return_to through select_accounts and complete_account_setup so users land back on the account they were linking from - Replace Rails.logger.error/warn with DebugLogEntry.capture in controller and unlinking concern (surface errors in the app debug log UI) - Add questrade_items to Family::FinancialDataReset::PROVIDER_ITEM_ASSOCIATIONS so Reset Financial Data actually removes Questrade data - Add when "questrade" case to load_provider_items in providers_controller so the settings panel lazy-load refresh works - Fix turbo_prefetch: false on non-lunchflow provider links in _method_selector.html.erb and select_provider.html.erb (prevents prefetch-cache blank-modal bug for all generic sync providers) - Preload questrade_accounts: :account_provider and build @questrade_account_counts_map in AccountsController; read from map in partial instead of calling .count on associations (eliminates N+1) - Localize default connection name via I18n.t(questrade_items.default_name) - Add default_name key to questrade_items locale Patterns and bugs surfaced during review of PR #2591 (Wise provider). * Cross-apply Wise learnings to Questrade provider Encryption (matched convention from Wise/jjmata review): - Add deterministic: true to QuestradeItem#refresh_token - Add encrypts :raw_payload + :raw_institution_payload to QuestradeItem - Add Encryptable + encrypts :raw_payload, :raw_holdings_payload, :raw_activities_payload, :raw_balances_payload to QuestradeAccount (brokerage-specific columns; matches MercuryAccount/UpAccount pattern) Bug fix: - Add missing RetryableResponseError class to Provider::Questrade (used in with_retries rescue clause but never defined — would cause NameError on any rate-limited or 5xx response) Logging: - Replace Rails.logger.error with DebugLogEntry.capture in QuestradeItem#import_latest_questrade_data, #process_accounts, and #schedule_account_syncs to surface errors in the support UI Consistency: - Extract update_sync_status(sync, key, **i18n_options) helper in QuestradeItem::Syncer, replacing 5 inline sync.update! guard calls - Use blank? instead of ||= for default name fallback in create action Tests: - Add QuestradeItemsControllerTest (18 tests: CRUD, sync, account linking/setup flows, admin guard enforcement) - Add questrade fixtures: questrade_items.yml, questrade_accounts.yml - Add retry/backoff tests to Provider::QuestradeTest (network error, 429, 5xx — all verify MAX_RETRIES exhaustion raises Error) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Signed-off-by: Jestin Palamuttam <34907800+jestinjoshi@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Shibu M
|
8ccea36517 | Merge branch 'we-promise:main' into Transfer-charges | ||
|
Guillem Arias Fauste
|
c5ca0431c9 |
feat(goals): investment-backed goals (Phase 2) (#2491)
* feat(goals): earmark a portion of an account toward a goal Goals currently count each linked account's whole balance, so an account shared across goals double-counts and one account can't fund several goals in distinct slices. Add a per-account earmark — the "GoalBacking" the v1 model already foreshadowed (goal.rb). - goal_accounts.allocated_amount (nullable). NULL = "dedicate the whole balance" (the v1 default: no backfill, existing goals unchanged); a set amount reserves a fixed slice. - Goal#current_balance is now the single chokepoint computing each account's backing under a family-wide shared pool: fixed earmarks take their slice, an unallocated link takes the remainder, and when fixed earmarks exceed the balance every slice is scaled down pro-rata so the goals' shares can never sum past the account balance (no double-counting). - Account#free_to_earmark / #goal_earmarked_total (mirror Budget's available_to_allocate) back a soft, non-blocking over-allocation hint. - GoalsController threads a goal[allocations] hash through create/update. Phase 1 of the goals earmarking work; investment-backed goals follow. * feat(goals): earmark UI on the goal form + backing-aware funding breakdown - Goal form: a per-account "earmark amount" input (blank = whole balance) next to each funding-account checkbox, prefilled from the saved allocation on edit. - Goal#account_backing exposes a single linked account's share so the funding-accounts breakdown shows each account's earmarked contribution and percent instead of its whole balance — keeping the show page consistent with the (now allocation-aware) progress ring. - English strings for the earmark controls and the "earmarked of balance" breakdown line. * fix(goals): address review on the earmark shared-pool math - Overdrawn (<= 0 balance) accounts now back nothing on both the fixed and whole-balance paths. The fixed path previously produced negative backing and let a goal claim money the account doesn't hold. - An archived goal reads its OWN earmark from its own goal_accounts instead of the shared pool (which excludes archived goals), so it no longer mis-reports the whole account balance for itself. - goals#index injects one family-wide earmark pool into every card (Goal.pooled_allocations_for) instead of querying once per goal (N+1), and preloads goal_accounts. - The projection chart scales its whole-account historical series by the backing ratio so the saved line meets current_balance at "today" rather than dropping off a cliff for earmarked goals. - Honest comments: free_to_earmark no longer claims a form warning that doesn't exist yet; pace documents its deliberate whole-account basis. * fix(goals): widen the earmark input so the 'Whole balance' placeholder isn't clipped * fix(goals): address review on #2490 - autosave: true on goal_accounts so earmark edits to already-linked accounts persist through goal.save! (Rails only auto-saves newly built children, so changing/clearing an existing earmark was silently dropped). + test. - Reset the balance/progress memos on AASM transitions, not just the status memos, so a same-instance render after complete!/archive! isn't stale. + test. - backing_ratio is 0 (not 1) when the linked-account total is non-positive, so the projection saved series ends at 0 to match the forced-zero current_balance. - Localize the funding-row subtype label via goals.form.subtypes.*. - Add the earmark strings to zh-CN (the maintained second locale; goals has no ca locale, so Catalan keeps falling back to en like the rest of goals). * feat(goals): investment-backed goals (Phase 2) Goals can now be funded by investment accounts, not just depository. - Relax linked_accounts_must_be_depository -> _must_be_fundable (depository || investment); the funding picker + counts include investment accounts. - Add goals.progress_basis ('balance' | 'contributions', default 'balance'). Investment-backed goals default to 'contributions' so a market swing doesn't move the goal: current_balance = value - cumulative market gain (Sum of balances.net_market_flows); depository accounts have zero net_market_flows, so they're unchanged. Goal#market_value_money shows what it's worth today next to the contributed figure on the show page. - Pledge false-match guard: investment accounts never use manual_save / valuation-delta matching (a market move isn't a deposit) - they resolve on transfer (cash-inflow) entries only. Guarded in both Account#default_pledge_kind and GoalPledge#matches?. - Add a `reopen` AASM event (completed -> active) + route/action/menu item so a manually-completed goal whose value later dips can be reopened. Stacked on the earmarking branch (#2490). Full suite green; +6 goal tests. * fix(goals): address Phase 2 review — allocation-aware contributions, N+1, basis-on-update - Contributions basis now goes through the same earmark/shared-pool logic as the balance basis: backing_balance_for -> backing_share_for(account, base), where base is the live balance (balance basis) or net contributions (contributions basis). Earmarks are respected and shared accounts no longer double-count on contributions goals; market_value_money stays consistent. - Batch the per-account net_market_flows sum (Goal.market_flows_for) and inject it on index like pooled_allocations, killing the N+1 for contributions goals. - Default the basis on update too (not just create), so adding an investment account to an existing depository goal flips it to contributions instead of silently tracking market value. - Fix the stale reconciliation_manager comment (renamed validation) and the orphaned zh-CN must_be_depository key. * fix(goals): address review on #2491 - before_save (not before_validation) for the progress_basis default, so a goal can be inspected via valid? without its basis flipping as a side effect (jjmata). - Pledge copy keys off default_pledge_kind, not manual?, so a manual investment account — which pledges via transfer — shows the transfer prompt instead of the "update your manual balance" flow (codex). pledge_action_label_key and the pledge modal's per-account helper flag both use it. - Add the Phase 2 strings (reopen success/invalid_transition, show.reopen, ring.market_value) to zh-CN. --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
Shibu M
|
35f395b39f | Merge branch 'we-promise:main' into Transfer-charges | ||
|
Guillem Arias Fauste
|
1b403d64e5 |
feat(goals): earmark a portion of an account toward a goal (Phase 1) (#2490)
* feat(goals): earmark a portion of an account toward a goal Goals currently count each linked account's whole balance, so an account shared across goals double-counts and one account can't fund several goals in distinct slices. Add a per-account earmark — the "GoalBacking" the v1 model already foreshadowed (goal.rb). - goal_accounts.allocated_amount (nullable). NULL = "dedicate the whole balance" (the v1 default: no backfill, existing goals unchanged); a set amount reserves a fixed slice. - Goal#current_balance is now the single chokepoint computing each account's backing under a family-wide shared pool: fixed earmarks take their slice, an unallocated link takes the remainder, and when fixed earmarks exceed the balance every slice is scaled down pro-rata so the goals' shares can never sum past the account balance (no double-counting). - Account#free_to_earmark / #goal_earmarked_total (mirror Budget's available_to_allocate) back a soft, non-blocking over-allocation hint. - GoalsController threads a goal[allocations] hash through create/update. Phase 1 of the goals earmarking work; investment-backed goals follow. * feat(goals): earmark UI on the goal form + backing-aware funding breakdown - Goal form: a per-account "earmark amount" input (blank = whole balance) next to each funding-account checkbox, prefilled from the saved allocation on edit. - Goal#account_backing exposes a single linked account's share so the funding-accounts breakdown shows each account's earmarked contribution and percent instead of its whole balance — keeping the show page consistent with the (now allocation-aware) progress ring. - English strings for the earmark controls and the "earmarked of balance" breakdown line. * fix(goals): address review on the earmark shared-pool math - Overdrawn (<= 0 balance) accounts now back nothing on both the fixed and whole-balance paths. The fixed path previously produced negative backing and let a goal claim money the account doesn't hold. - An archived goal reads its OWN earmark from its own goal_accounts instead of the shared pool (which excludes archived goals), so it no longer mis-reports the whole account balance for itself. - goals#index injects one family-wide earmark pool into every card (Goal.pooled_allocations_for) instead of querying once per goal (N+1), and preloads goal_accounts. - The projection chart scales its whole-account historical series by the backing ratio so the saved line meets current_balance at "today" rather than dropping off a cliff for earmarked goals. - Honest comments: free_to_earmark no longer claims a form warning that doesn't exist yet; pace documents its deliberate whole-account basis. * fix(goals): widen the earmark input so the 'Whole balance' placeholder isn't clipped * fix(goals): address review on #2490 - autosave: true on goal_accounts so earmark edits to already-linked accounts persist through goal.save! (Rails only auto-saves newly built children, so changing/clearing an existing earmark was silently dropped). + test. - Reset the balance/progress memos on AASM transitions, not just the status memos, so a same-instance render after complete!/archive! isn't stale. + test. - backing_ratio is 0 (not 1) when the linked-account total is non-positive, so the projection saved series ends at 0 to match the forced-zero current_balance. - Localize the funding-row subtype label via goals.form.subtypes.*. - Add the earmark strings to zh-CN (the maintained second locale; goals has no ca locale, so Catalan keeps falling back to en like the rest of goals). --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
DataEnginr
|
76fe10798d | Update schema.rb version after migrate, preserve clean diff vs main | ||
|
DataEnginr
|
e75f2a0c78 |
Fix fee display consistency, derive fees from entries, clean schema churn
- Show principal-only transfer amounts on both sides with separate fee and total lines (fixes inconsistent gross/net convention) - Derive displayed fee amounts from fee_transactions entries (single source of truth) instead of stored columns - Remove stored source_fee_amount/destination_fee_amount columns from transfers table - Add foreign key for transactions.transfer_id -> transfers.id (replaces invalid CHECK subquery) - Move destination fee line inside destination side div for consistent layout - Remove orphaned view_fee_transaction locale keys from 7 locale files - Rebuild schema.rb from origin/main to eliminate unrelated column reordering churn |
||
|
DataEnginr
|
1b21c4dd7b |
Store fees as separate expense transactions with principal-only entries
Entries now hold principal only (no fee baked into amounts). Fee transactions created as standard kind with Fees category. Transfer#amount_abs returns principal from new amount column. Update handler recomputes entries and fee transactions on edit. Remove dead source_principal/destination_principal helpers. Schema regenerated cleanly with only transfer fee columns. |
||
|
Shibu M
|
eccf050521 | Merge branch 'we-promise:main' into Transfer-charges | ||
|
DataEnginr
|
eee0ca8975 | Fix migration: add missing add_column, keep schema on 7.2 with minimal diff | ||
|
DataEnginr
|
945a08b0c2 |
Merge remote-tracking branch 'upstream/main' into feature/exclude-from-reports
# Conflicts: # db/schema.rb |
||
|
Shibu M
|
354ece074d | Merge branch 'we-promise:main' into Transfer-charges | ||
|
Juan José Mata
|
b416618558 |
Add SnapTrade OAuth device flow (#2494)
* Add SnapTrade OAuth device flow * Add SnapTrade OAuth device flow ### Motivation - Integrate SnapTrade device-code OAuth so administrators can start a device authorization flow and poll for tokens using SnapTrade's well-known OAuth metadata endpoint. - Persist and encrypt device-flow token material on `SnaptradeItem` to support long-lived API calls and future refresh handling. ### Description - Add OAuth discovery and device-code support to the SnapTrade provider with `oauth_authorization_server_metadata`, `start_device_authorization`, and `poll_device_token` helper methods and shared `oauth_connection` / `parse_oauth_response` helpers. - Persist OAuth device-flow tokens on `snaptrade_items` via a migration that adds `oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, and `oauth_token_expires_at`, and update `db/schema.rb` accordingly. - Extend `SnaptradeItem` to encrypt stored `oauth_access_token` and `oauth_refresh_token` when ActiveRecord encryption is configured and add `oauth_token_active?` to check token validity. - Add high-level `start_oauth_device_flow` and `complete_oauth_device_flow!` helpers to the `SnaptradeItem::Provided` concern to start authorization and persist token responses. - Expose admin-only member routes and JSON controller actions `start_oauth_device_flow` and `complete_oauth_device_flow` on `SnaptradeItemsController` to drive the device flow from the UI. - Add focused Minitest coverage: `test/models/provider/snaptrade_oauth_test.rb` for provider requests and error handling and `test/models/snaptrade_item_oauth_test.rb` for token persistence. ### Testing - Ran the targeted test suite: `bin/rails test test/models/provider/snaptrade_oauth_test.rb test/models/snaptrade_item_oauth_test.rb test/controllers/snaptrade_items_controller_test.rb`, and all tests passed. - Ran `bin/rubocop` against modified files and no offenses were reported. * Add SnapTrade OAuth device flow ### Motivation - Integrate SnapTrade device-code OAuth so administrators can start a device authorization flow and poll for tokens using SnapTrade's well-known OAuth metadata endpoint. - Persist and encrypt device-flow token material on `SnaptradeItem` to support long-lived API calls and future refresh handling. ### Description - Add OAuth discovery and device-code support to the SnapTrade provider with `oauth_authorization_server_metadata`, `start_device_authorization`, and `poll_device_token` helper methods and shared `oauth_connection` / `parse_oauth_response` helpers. - Persist OAuth device-flow tokens on `snaptrade_items` via a migration that adds `oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, and `oauth_token_expires_at`, and update `db/schema.rb` accordingly. - Extend `SnaptradeItem` to encrypt stored `oauth_access_token` and `oauth_refresh_token` when ActiveRecord encryption is configured and add `oauth_token_active?` to check token validity. - Add high-level `start_oauth_device_flow` and `complete_oauth_device_flow!` helpers to the `SnaptradeItem::Provided` concern to start authorization and persist token responses. - Expose admin-only member routes and JSON controller actions `start_oauth_device_flow` and `complete_oauth_device_flow` on `SnaptradeItemsController` to drive the device flow from the UI. - Add focused Minitest coverage: `test/models/provider/snaptrade_oauth_test.rb` for provider requests and error handling and `test/models/snaptrade_item_oauth_test.rb` for token persistence. ### Testing - Ran the targeted test suite: `bin/rails test test/models/provider/snaptrade_oauth_test.rb test/models/snaptrade_item_oauth_test.rb test/controllers/snaptrade_items_controller_test.rb`, and all tests passed. - Ran `bin/rubocop` against modified files and no offenses were reported. * Add SnapTrade OAuth device flow ### Motivation - Integrate SnapTrade device-code OAuth so administrators can start a device authorization flow and poll for tokens using SnapTrade's well-known OAuth metadata endpoint. - Persist and encrypt device-flow token material on `SnaptradeItem` to support long-lived API calls and future refresh handling. ### Description - Add OAuth discovery and device-code support to the SnapTrade provider with `oauth_authorization_server_metadata`, `start_device_authorization`, and `poll_device_token` helper methods and shared `oauth_connection` / `parse_oauth_response` helpers. - Persist OAuth device-flow tokens on `snaptrade_items` via a migration that adds `oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, and `oauth_token_expires_at`, and update `db/schema.rb` accordingly. - Extend `SnaptradeItem` to encrypt stored `oauth_access_token` and `oauth_refresh_token` when ActiveRecord encryption is configured and add `oauth_token_active?` to check token validity. - Add high-level `start_oauth_device_flow` and `complete_oauth_device_flow!` helpers to the `SnaptradeItem::Provided` concern to start authorization and persist token responses. - Expose admin-only member routes and JSON controller actions `start_oauth_device_flow` and `complete_oauth_device_flow` on `SnaptradeItemsController` to drive the device flow from the UI. - Add focused Minitest coverage: `test/models/provider/snaptrade_oauth_test.rb` for provider requests and error handling and `test/models/snaptrade_item_oauth_test.rb` for token persistence. ### Testing - Ran the targeted test suite: `bin/rails test test/models/provider/snaptrade_oauth_test.rb test/models/snaptrade_item_oauth_test.rb test/controllers/snaptrade_items_controller_test.rb`, and all tests passed. - Ran `bin/rubocop` against modified files and no offenses were reported. * Add SnapTrade OAuth device flow ### Motivation - Integrate SnapTrade device-code OAuth so administrators can start a device authorization flow and poll for tokens using SnapTrade's well-known OAuth metadata endpoint. - Persist and encrypt device-flow token material on `SnaptradeItem` to support long-lived API calls and future refresh handling. ### Description - Add OAuth discovery and device-code support to the SnapTrade provider with `oauth_authorization_server_metadata`, `start_device_authorization`, and `poll_device_token` helper methods and shared `oauth_connection` / `parse_oauth_response` helpers. - Persist OAuth device-flow tokens on `snaptrade_items` via a migration that adds `oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, and `oauth_token_expires_at`, and update `db/schema.rb` accordingly. - Extend `SnaptradeItem` to encrypt stored `oauth_access_token` and `oauth_refresh_token` when ActiveRecord encryption is configured and add `oauth_token_active?` to check token validity. - Add high-level `start_oauth_device_flow` and `complete_oauth_device_flow!` helpers to the `SnaptradeItem::Provided` concern to start authorization and persist token responses. - Expose admin-only member routes and JSON controller actions `start_oauth_device_flow` and `complete_oauth_device_flow` on `SnaptradeItemsController` to drive the device flow from the UI. - Add focused Minitest coverage: `test/models/provider/snaptrade_oauth_test.rb` for provider requests and error handling and `test/models/snaptrade_item_oauth_test.rb` for token persistence. ### Testing - Ran the targeted test suite: `bin/rails test test/models/provider/snaptrade_oauth_test.rb test/models/snaptrade_item_oauth_test.rb test/controllers/snaptrade_items_controller_test.rb`, and all tests passed. - Ran `bin/rubocop` against modified files and no offenses were reported. * Add SnapTrade OAuth device flow (provider, model, controller, routes, migration, tests) ### Motivation - Add support for SnapTrade OAuth 2.0 device authorization flow so users can authorize SnapTrade via device codes in addition to the existing portal flow. - Persist OAuth token metadata on `SnaptradeItem` for subsequent polling and usage by the provider and UI. ### Description - Implemented OAuth device flow in the provider with `oauth_authorization_server_metadata`, `start_device_authorization`, and `poll_device_token`, plus HTTP helper methods `oauth_connection`, `oauth_client_id`, and `parse_oauth_response` in `Provider::Snaptrade`. - Added controller endpoints `start_oauth_device_flow` and `complete_oauth_device_flow` in `SnaptradeItemsController` with robust error handling and helpers `oauth_error_payload` and `parse_oauth_error_body` to surface OAuth error fields. - Extended `SnaptradeItem` with encrypted columns for `oauth_access_token`, `oauth_refresh_token`, token metadata, `oauth_token_active?`, and model methods `start_oauth_device_flow` and `complete_oauth_device_flow!` to store token metadata. - Added migration `AddOauthDeviceFlowToSnaptradeItems` and updated `db/schema.rb` to include the new columns, registered a new initializer `config/initializers/snaptrade.rb` to read `SNAPTRADE_OAUTH_CLIENT_ID`, and updated `.env*.example` files to document `SNAPTRADE_OAUTH_CLIENT_ID`. - Exposed new routes `start_oauth_device_flow` and `complete_oauth_device_flow` for `snaptrade_items`. ### Testing - Added unit tests for provider OAuth behavior in `test/models/provider/snaptrade_oauth_test.rb`, model token persistence in `test/models/snaptrade_item_oauth_test.rb`, and controller error propagation in `test/controllers/snaptrade_items_controller_test.rb`. - Ran the test suite with `bin/rails test` and the full test run (including the new SnapTrade OAuth tests) passed. * Add SnapTrade OAuth device flow (start/poll), store tokens, and tests ### Motivation - Add support for SnapTrade OAuth Device Authorization flow so administrators can start device authorization and poll for tokens without exposing provider internals. - Persist OAuth token metadata on `SnaptradeItem` so the app can reuse and surface token state for SnapTrade integrations. - Improve error handling and sanitization for OAuth API errors returned by SnapTrade to avoid leaking upstream internals. ### Description - Introduces new environment examples and initializer: adds `SNAPTRADE_OAUTH_CLIENT_ID` to `.env.local.example`/`.env.test.example` and configures `Rails.configuration.x.snaptrade.oauth_client_id` in `config/initializers/snaptrade.rb`. - Extends `Provider::Snaptrade` with OAuth device flow support, adding discovery URL, device grant constant, Faraday `oauth_connection`, and methods `oauth_authorization_server_metadata`, `start_device_authorization`, `poll_device_token`, `oauth_client_id`, and `parse_oauth_response`, plus improved retry and API error wrapping. - Adds DB migration and schema changes to store OAuth fields on `snaptrade_items` (`oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, `oauth_token_expires_at`) and marks those attributes encrypted when ActiveRecord encryption is enabled. - Adds `SnaptradeItem` helpers `start_oauth_device_flow`, `complete_oauth_device_flow!`, and `oauth_token_active?` to start/poll and persist token metadata. - Adds controller actions `start_oauth_device_flow` and `complete_oauth_device_flow` with sanitized error responses and helper methods `oauth_error_payload` and `parse_oauth_error_body`, and wires new member routes in `config/routes.rb`. - Updates `SnaptradeItemsController` before_action lists to include the new actions and adds user-facing error message helper `start_oauth_device_flow_error_message`. ### Testing - Added unit tests `test/models/provider/snaptrade_oauth_test.rb` to validate discovery, device authorization request, token polling, missing client ID handling, and OAuth error propagation, and all assertions passed. - Added `test/models/snaptrade_item_oauth_test.rb` to assert `complete_oauth_device_flow!` stores tokens and expiry and that `oauth_token_active?` reports correctly, and the test passed. - Extended `test/controllers/snaptrade_items_controller_test.rb` with controller-level tests confirming sanitized OAuth error payloads and behavior for start/complete endpoints, and these controller tests passed. * Add SnapTrade OAuth device flow support and token storage ### Motivation - Add support for the OAuth 2.0 device authorization flow for SnapTrade so users can link brokerages via a device-code flow without traditional browser-based client redirects. - Persist OAuth token metadata on the SnaptradeItem so tokens can be reused and expiration tracked. - Surface and sanitize provider error payloads to callers while avoiding leakage of internal configuration details. ### Description - Added new DB columns and a migration (`oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, `oauth_token_expires_at`) and updated `db/schema.rb` to reflect the change. - Encrypted the new token fields on `SnaptradeItem` and added `oauth_token_active?`, `start_oauth_device_flow`, and `complete_oauth_device_flow!` helpers to the model. - Implemented OAuth logic in `Provider::Snaptrade` including discovery (`OAUTH_DISCOVERY_URL`), `start_device_authorization`, `poll_device_token`, request parsing, retry handling, and a small Faraday connection wrapper; added configuration accessors for `oauth_client_id` via `config.x.snaptrade`. - Added controller endpoints `start_oauth_device_flow` and `complete_oauth_device_flow` with safe error handling and helpers to format OAuth error payloads, and registered new member routes for `snaptrade_items`. - Added an initializer to load `SNAPTRADE_OAUTH_CLIENT_ID` into `Rails.configuration.x.snaptrade`, and updated `.env.local.example` and `.env.test.example` to include the new env var. - Added unit tests for provider OAuth behavior (`test/models/provider/snaptrade_oauth_test.rb`), SnaptradeItem token persistence (`test/models/snaptrade_item_oauth_test.rb`), and controller error handling (`test/controllers/snaptrade_items_controller_test.rb`), and updated controller tests accordingly. ### Testing - Ran provider-level tests in `Provider::SnaptradeOauthTest` to validate discovery, device authorization, token polling, and error handling, which passed. - Ran model tests in `SnaptradeItemOauthTest` to verify token metadata is stored and `oauth_token_active?` works, which passed. - Ran controller tests in `SnaptradeItemsControllerTest` that exercise the start/complete endpoints and error sanitization, which passed. * Add SnapTrade OAuth device-flow support and token storage ### Motivation - Add support for SnapTrade OAuth device authorization so users can perform device-code based OAuth without embedding secrets in the browser. - Persist OAuth token metadata on `SnaptradeItem` and expose programmatic start/complete endpoints while avoiding leaking provider internals on errors. - Make OAuth client ID configurable via environment or credentials for Sure deployments. ### Description - Introduce `SNAPTRADE_OAUTH_CLIENT_ID` to `.env` examples and add `config.x.snaptrade.oauth_client_id` initializer for configuration. - Add migration `AddOauthDeviceFlowToSnaptradeItems` and update `schema.rb` to add `oauth_access_token`, `oauth_refresh_token`, `oauth_token_type`, `oauth_scope`, and `oauth_token_expires_at` to `snaptrade_items`. - Persist and encrypt new token fields in `SnaptradeItem` and add helper methods `oauth_token_active?`, `start_oauth_device_flow`, and `complete_oauth_device_flow!`. - Extend `Provider::Snaptrade` with OAuth discovery, device authorization (`start_device_authorization`), token polling (`poll_device_token`), Faraday-based `oauth_connection`, parsing and error-wrapping logic, and retry handling. - Add controller actions `start_oauth_device_flow` and `complete_oauth_device_flow` to `SnaptradeItemsController`, plus helper methods to format OAuth error payloads and user-facing error messages. - Wire up new routes (`post :start_oauth_device_flow`, `post :complete_oauth_device_flow`) and add `config/initializers/snaptrade.rb`. - Add comprehensive tests for the OAuth flow and controller error handling and update a system test stub to avoid provider leakage during UI tests. ### Testing - Added `Provider::SnaptradeOauthTest` which stubs discovery, device authorization and token endpoints and asserts request payloads and responses; tests passed. - Added `SnaptradeItemOauthTest` which verifies token metadata persistence and `oauth_token_active?`; test passed. - Extended `SnaptradeItemsControllerTest` with scenarios for successful and failing device-flow completion and start flow error handling; tests passed. - Ran the affected system test changes in `TradesTest` (provider stubbing and modal behavior); the updated tests passed. |
||
|
DataEnginr
|
4efc268ca8 | Fix RuboCop Layout/SpaceInsideArrayLiteralBrackets in migration | ||
|
DataEnginr
|
42a075a6db | Address all PR review comments | ||
|
panther
|
eb0866a678 |
Add exclude_from_reports option to accounts
Adds a toggle to mark accounts as excluded from all financial reports while keeping them active and visible individually. - Migration: add exclude_from_reports boolean column to accounts - Model: included_in_reports scope - BalanceSheet: filter excluded from ClassificationGroup and AccountGroup totals, HistoricalAccountScope, AccountRow flag - IncomeStatement: exclude via SQL fragments in Totals, FamilyStats, CategoryStats - InvestmentStatement/Budget: chain included_in_reports scope - ReportsController: filter in breakdown view, export queries, trades - AccountsController: toggle_exclude_from_reports action + route - UI: DS::Toggle in account form, eye-off indicator in sidebar, menu toggle items - Locale: all labels in en.yml - Tests: model scope, controller toggle, income statement/balance sheet filtering - 5070 tests pass (0 failures, 0 regressions) |
||
|
Shibu M
|
d058bcbdcc | Merge branch 'we-promise:main' into Transfer-charges | ||
|
Jake
|
dc2a565b6a |
feat(up): add Up Bank (AU) provider integration (#2391)
* feat(up): add Up Bank (AU) provider integration Adds Up Bank as a per-family, token-based bank sync provider, modelled on the existing Akahu integration. Up uses a JSON:API REST API with a personal access token (Bearer), cursor pagination via links.next, and returns both HELD (pending) and SETTLED transactions from one endpoint. New: - Provider::Up client (JSON:API unwrap, links.next pagination, retries, typed errors, /util/ping) + Provider::UpAdapter (Factory-registered, Depository + Loan). - UpItem / UpAccount models with Provided, Unlinking, Syncer, SyncCompleteEvent, Importer, Processor, Transactions::Processor, and UpEntry::Processor (amount sign flip, HELD->pending, foreignAmount FX, merchant from description, stale-pending pruning). - Family::UpConnectable, UpItemsController, routes, settings panel + connect flow views, accounts index wiring, initializer, en locale, and model tests. Core wiring: - "up" added to Transaction::PENDING_PROVIDERS, the three pending-match SQL blocks in Account::ProviderImportAdapter, Provider::Metadata::REGISTRY, ProviderMerchant/DataEnrichment source enums, ProviderConnectionStatus, settings provider panels, and financial data reset. Migration create_up_items_and_accounts must be run before use. No external API endpoints added (no OpenAPI changes). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(up): dump up tables to schema and make since filter TZ-safe The feature commit added the up_items/up_accounts migration but never re-dumped db/schema.rb, leaving the schema version and tables stale. Add the two table definitions and foreign keys and bump the schema version so a fresh DB load matches the migration. Also format a bare Date `since` as UTC midnight instead of the server's local zone, so `filter[since]` is deterministic regardless of where the app runs (previously shifted by the local UTC offset). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(up): address code review feedback Behavior/correctness: - Persist skipped accounts via a new up_accounts.ignored flag and a needs_setup scope, so skipped accounts stop resurfacing as "needs setup" on every sync. Linking clears the flag. - destroy now checks unlink_all! per-account results and aborts deletion (alert) if any unlink failed, instead of swallowing failures. - render_provider_panel_error redirect uses :see_other (was an invalid 4xx redirect status). - Up provider adapter falls back to item institution name/url when institution_metadata is absent (early return previously blocked it). Resilience/security: - fetch_all_resources guards against an API repeating the same links.next cursor (Set#add?), preventing infinite pagination. - HTTP client validates absolute URLs (from links.next) against Up's HTTPS host before sending the bearer token, preventing credential leakage to untrusted hosts. Diagnostics: - Route provider sync/import failures through DebugLogEntry.capture (controller, UpItem, syncer, unlinking) with family/account context. Low-level HTTP client and currency-normalization warnings keep Rails.logger to match existing provider conventions. Data integrity: - up_accounts.name and currency are NOT NULL (align with model presence validations); account_id stays nullable (allow_nil uniqueness). Forms: - select_existing_account radio is required; controller guards a blank/ unknown up_account_id with a friendly alert instead of RecordNotFound. Tests: - Add UpAccount needs_setup scope test, pagination loop guard test, untrusted-host rejection test; tighten filter[since] assertion to the exact UTC timestamp. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(up): address second-round review feedback - Capture sync/import failures via DebugLogEntry so swallowed errors in account/transaction fetching and transaction processing surface in /settings/debug instead of only Rails.logger. - Gate UP_DEBUG_RAW raw payload dump to local envs to avoid leaking PII (merchant names, amounts, account IDs) in managed/production logs. - Collapse linked/unlinked/total account counts into one memoized query instead of 3 separate COUNTs per rendered item. - Rename "Set Up Up Accounts" locale title to "Link Up Accounts". Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(up): add method docstrings and align failed_result keys Add docstrings to all Up provider source files (controller, models, providers, concerns) to satisfy the 80% docstring coverage threshold. Third-round review: failed_result now mirrors import's result shape (accounts_updated/created/failed, transactions_imported/failed) instead of the stale accounts_imported key, so failure results stay consistent. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Shibu M
|
0b037e2027 | Merge branch 'we-promise:main' into Transfer-charges | ||
|
ghost
|
894d705e83 |
perf(dashboard): streamline investment activity totals (#2257)
* perf(dashboard): streamline investment activity totals * fix(dashboard): skip empty investment totals cache writes Short-circuit empty investment account totals before cache fetch and move the new SQL query capture regression helper into test/support. * refactor(dashboard): address review on investment totals query - Drop defensive ArgumentError guards in InvestmentStatement::Totals#initialize. The sole caller (totals_query) always passes correct types, and the empty_result early-return already handles the zero-accounts case. - Remove the redundant accounts JOIN and its family_id/status filters from the aggregation SQL. account_ids is already scoped to the family's visible (draft/active) investment accounts, so the join back to accounts is unnecessary work in the query plan. Drop the now-unused family_id param. - Add a regression assertion that the aggregate no longer joins accounts. |
||
|
maverick
|
b4be6ca30d | Merge branch 'we-promise:main' into Transfer-charges | ||
|
Kelvinchen03
|
11bba13601 | Add ON DELETE CASCADE to rejected_transfers foreign keys (#2211) | ||
|
Shibu M
|
fe6b2f9872 | merged with main | ||
|
ghost
|
de8cd86f2f |
perf(sync): scope transfer matching after account sync (#2230)
* perf(sync): scope transfer matching after account sync * fix(sync): make transfer lookup index migration reversible |
||
|
maverick
|
016a1acf35 |
Merge branch 'main' into Transfer-charges
Signed-off-by: maverick <23173570+DataEnginr@users.noreply.github.com> |
||
|
Guillem Arias Fauste
|
d845e44ff8 |
feat(ai): default Anthropic installs to pgvector RAG (4/5) (#1986)
* feat(ai): add Anthropic provider with chat parity (1/5)
Introduces Provider::Anthropic alongside Provider::Openai, implementing
the LlmConcept chat_response contract over the official anthropic Ruby
SDK. Batch ops, PDF, and RAG land in follow-up PRs.
- Provider::Anthropic uses Messages API for sync and streaming responses
- ChatConfig builds requests with ephemeral prompt-cache markers on the
system prompt and the last tool definition
- MessageFormatter reconstructs multi-turn history (text + tool_use +
tool_result blocks) from raw Message records, including the paired
user-role tool_result turn Anthropic requires after every tool_use
- ChatParser maps Anthropic Message into the shared ChatResponse Data
- Registry, Setting, User, Chat default model wired for ANTHROPIC_*
envs and Setting.anthropic_*; LLM_PROVIDER selects between providers
- Responder forwards raw conversation_history (Array<Message>) so
providers without hosted conversation state can rebuild context
- OpenAI provider accepts and ignores the new kwarg (no behavior change)
Tests cover provider init, model gating, MessageFormatter for all turn
shapes, ChatConfig request building (max_tokens, system cache, tool
conversion), ChatParser for text / tool_use / mixed blocks, Registry
discovery, and mocked chat_response success / error / function_request
paths. Live VCR cassettes recorded in a follow-up with a real key.
Stacked PRs: 2/5 batch ops + cost ledger, 3/5 PDF, 4/5 pgvector RAG,
5/5 settings UI + disclosure.
* fix(ai): address PR review on Anthropic provider foundation
Surface fixes raised by Codex + CodeRabbit on PR 1/5:
- Provider::Anthropic#chat_response now accepts (and ignores) a
`messages:` kwarg. Assistant::Responder passes both `messages:`
(OpenAI-shape) and `conversation_history:` (raw Message records) for
cross-provider parity, so the previous signature raised
ArgumentError on the first chat turn through the Anthropic provider.
- Provider::Anthropic#supports_model? bypasses the `claude` prefix
gate when a custom base_url is configured, mirroring the OpenAI
provider. Bedrock-shaped IDs like
`anthropic.claude-sonnet-4-5-20250929-v1:0` and
`claude-opus-4@20250514` are otherwise rejected by
Assistant::Provided#get_model_provider and the chat dies.
- Setting.anthropic_access_token is now in
EncryptedSettingFields::ENCRYPTED_FIELDS so the Anthropic API key
is encrypted at rest like every other provider secret. Previously
plaintext while siblings (openai_access_token, twelve_data_api_key,
external_assistant_token) were ciphertext.
- Chat.default_model falls back to whichever provider is actually
configured. Previously, with LLM_PROVIDER=anthropic but no
Anthropic credentials, the default model resolved to a Claude ID
that no registered provider supported, so chats failed even when
OpenAI was fully configured. Adds Provider::{Anthropic,Openai}#configured?
class methods for the readable callsite.
- Provider::Anthropic.effective_model uses
`ENV["ANTHROPIC_MODEL"].presence || Setting.anthropic_model` so the
Setting lookup is only performed when the env var is absent — the
previous `ENV.fetch(KEY, default)` evaluated the default arg
eagerly on every call.
- Provider::Anthropic::ChatConfig#anthropic_input_schema strips both
`:strict` and `"strict"` keys so JSON-decoded schemas with string
keys cannot leak the OpenAI-only flag through to Anthropic.
Test coverage added: supports_model? bypass on custom endpoints,
chat_response messages: kwarg compatibility, default_model fallback
in the three credential combinations, configured? against ENV +
Setting, strict-flag stripping for both key types, and a
`Setting.expects(:anthropic_model).never` assertion proving the
ENV-precedence test now exercises the lazy path.
All 4365 tests pass (1 pre-existing libvips env error unrelated).
* test(chat): make default_model tests resilient to ENV model overrides
CodeRabbit flagged on PR review: the new default_model tests asserted
against Provider::*::DEFAULT_MODEL, but Chat.default_model actually
returns Provider::*.effective_model.presence (which reads
OPENAI_MODEL / ANTHROPIC_MODEL from the environment). With either env
var set, the tests would fail intermittently even though routing was
correct.
- New default_model tests now assert against the provider's
effective_model directly, so they verify the routing decision
(which provider's value wins) without coupling to the constant.
- Pre-existing "creates with default model" assertions had the same
brittleness; switch them to compare against Chat.default_model so
the chosen model is whatever the env / Setting cascade resolves to.
Verified by running `ANTHROPIC_MODEL=claude-haiku-4-5 OPENAI_MODEL=gpt-4o
bin/rails test test/models/chat_test.rb` — 16 runs, 0 failures
(previously 2 pre-existing failures + 0 from the new tests).
* fix(ai): address local review on Anthropic foundation
- Provider::Anthropic#supports_pdf_processing? bypasses prefix gate for
custom endpoints, mirroring supports_model?
- Provider::Anthropic#initialize raises Error when custom_endpoint? AND
model.blank?, parity with Provider::Openai
- stream_chat_response captures partial usage on mid-stream errors and
records it via the new on_partial callback so chat_response can skip
the duplicate error row in the outer rescue
- safe_accumulated_message swallows the secondary failure when the SDK
cannot reconstruct a snapshot
- langfuse_client memoizes properly (||= instead of =) so repeated calls
don't churn Langfuse instances
- MessageFormatter sorts tool_calls by created_at then id so the
message array is deterministic across replays; skips tool_calls
missing both provider_call_id and provider_id rather than sending
`id: nil` and getting rejected by Anthropic
- Setting.anthropic_access_token default falls back through
ENV["ANTHROPIC_API_KEY"].presence (was missing .presence, so an
empty-string env value bled through)
- User#openai_configured? / #anthropic_configured? delegate to the
Provider::* class methods — single source of truth
- Assistant::Responder renames the OpenAI-shape history builder
conversation_history → openai_messages_payload so the kwarg name
matches the local method name (messages: openai_messages_payload,
conversation_history: chat_message_records)
- Assistant::Builtin stale-history comment updated to reference both
builders
Adds a streaming chat_response test using ad-hoc subclasses of the
SDK event types so the case/when dispatch matches via is_a? without
stubbing class-level === behavior.
* test(ai): add Anthropic tool_use round-trip + multi-tool turn coverage
Addresses @jjmata's "worth confirming" note on PR #1983: tool-use turns
from prior assistant messages must round-trip correctly when retrieved
from the database.
- New `ChatParser → ToolCall::Function → MessageFormatter` test walks
the full path: Anthropic response with a tool_use block →
ChatFunctionRequest → ToolCall::Function.from_function_request →
persisted on the AssistantMessage → MessageFormatter rebuild on the
next turn. Asserts the original `tool_use.id` is preserved end-to-end
as both `tool_use.id` and the paired `tool_result.tool_use_id`, and
that the original `input` hash and serialized result content survive.
- New multi-tool assistant turn test confirms two tool_use blocks on a
single assistant message render as two tool_use blocks followed by
two paired tool_result blocks in a single user-role follow-up,
matching Anthropic's required alternation.
Both tests exercise the existing PR1 code without behavior changes.
* test(ai): require "ostruct" explicitly in Anthropic provider tests
OpenStruct is moving out of Ruby's default load path (warning in 3.4+,
removed in 3.5+). Tests work today because ActiveSupport transitively
loads it, but that's incidental. Match the existing convention in
test/controllers/settings/hostings_controller_test.rb which explicitly
requires ostruct for the same reason.
* fix(ai): sanitize Langfuse warn logs, normalize tool_use.input, dedup history fetch
Addresses three open CodeRabbit findings on PR #1983.
- Provider::Anthropic Langfuse rescue branches no longer include
`e.full_message` in `Rails.logger.warn`. `full_message` bundles the
backtrace + cause chain and on some SDK error types includes the
serialized request/response payload (prompt, model output). Logs
now report `#{e.class}: #{e.message}` only. Three sites:
create_langfuse_trace, log_langfuse_generation, upsert_langfuse_trace.
Note: Provider::Openai has the same pattern (copy-pasted source) —
harmonization deferred to a follow-up cleanup PR; this commit fixes
only the Anthropic provider to keep PR scope tight.
- MessageFormatter#parse_arguments now coerces any non-Hash parsed
result to `{}`. Anthropic's Messages API requires `tool_use.input`
to be a JSON object (map); a stored ToolCall::Function record whose
arguments parse to a scalar, bool, or array (corrupt row, legacy
data, cross-provider bleed) would otherwise produce a payload the
API rejects. Normal flow stores Hash arguments end-to-end so the
fix is defensive — adds 2 tests covering scalar/array JSON strings
and non-String non-Hash inputs.
- Assistant::Responder dedups the chat-history fetch. The previous
layout fired two near-identical `chat.messages.where(...).includes(
:tool_calls).ordered` queries per LLM turn (one for the OpenAI-shape
payload, one for the raw-records kwarg). A new memoized
`complete_chat_messages` fetches once; `chat_message_records` filters
out the current message via `Array#reject`, `openai_messages_payload`
iterates the cached array unchanged. One SQL query per turn instead
of two. Memoization scope = single Responder instance (per LLM call),
so cache invalidation is not a concern.
All 4370 tests pass (1 pre-existing libvips env error unrelated).
Rubocop + brakeman clean.
* fix(ci): replace sk-ant- prefixed test placeholders
Pipelock secret scanner pattern-matches `sk-ant-*` as a real Anthropic
API key and fails the PR security-scan check. Test stubs and
ClimateControl env values used `sk-ant-test`, `sk-ant-from-setting`,
`sk-ant-x`, `sk-ant-y` as obvious placeholders, but the scanner does
not care about value entropy.
Switched to `fake-anthropic-key-*` / `fake-token-*` strings so the
scanner stops flagging them. No production code touched, no behavior
change — Provider::Anthropic still accepts any non-blank token.
* feat(ai): add Anthropic batch ops + LLM cost ledger (2/5)
Implements auto_categorize, auto_detect_merchants, and
enhance_provider_merchants on Provider::Anthropic via forced tool calls,
plus the cost-ledger plumbing they need.
- Provider::Anthropic::AutoCategorizer, AutoMerchantDetector,
ProviderMerchantEnhancer each define a single output tool whose
input_schema mirrors the desired output, then force the model to call
it via tool_choice: { type: "tool", name: ..., disable_parallel_tool_use: true }.
Anthropic guarantees the tool_use.input matches the schema, so there
is no JSON parsing fragility, no <think> tag stripping, and no
json_object/json_schema fallback ladders.
- Concerns::UsageRecorder mirrors the OpenAI sibling but persists
cache_creation_input_tokens / cache_read_input_tokens to dedicated
columns instead of metadata.
- Migration adds cache_creation_tokens, cache_read_tokens (nullable
integers) to llm_usages. OpenAI rows leave them null.
- LlmUsage::PRICING gains Claude 4.x rows (opus-4-7 $15/$75, sonnet-4-6
$3/$15, haiku-4-5 $1/$5 per MTok). infer_provider returns "anthropic"
for claude-* via the existing exact/prefix lookup.
- Provider::Anthropic#chat_response now persists cache columns directly
rather than stashing them in metadata.
- 25-transaction batch cap mirrors the OpenAI provider so the cost
ledger sees the same shape regardless of which provider ran a batch.
Tests cover the forced-tool-call path, null/None normalization,
case-insensitive merchant matching, the missing-tool_use error path,
and Anthropic-specific pricing + provider inference on LlmUsage.
Stacked on #1983 (PR 1/5). 3/5 PDF + vision next.
* fix(ai): attribute Bedrock model IDs to anthropic + clean nil enum
- LlmUsage.infer_provider now returns "anthropic" for Bedrock /
Vertex shaped IDs (anthropic.* and anthropic/*), so cost-ledger
filtering by provider stays correct even when no per-MTok rate is
stored. Previously these IDs fell through to the "openai" default.
- AutoCategorizer drops the redundant nil sentinel from the
category_name enum — the union type [string, null] already permits
null, and some JSON Schema validators reject nil literals inside
enum arrays.
* test(ai): require "ostruct" in Anthropic batch op tests
Same rationale as the PR1 ostruct fix — explicit require so the tests
don't depend on ActiveSupport's transitive load when Ruby 3.5+ removes
OpenStruct from the default load path.
* feat(ai): Anthropic native PDF processing (3/5)
Implements process_pdf and extract_bank_statement on Provider::Anthropic
using the native `document` content block — no rasterization, no text
pre-extraction.
- Provider::Anthropic::PdfProcessor classifies the document, summarizes
it, and extracts statement metadata via a forced report_document_analysis
tool whose input_schema mirrors the existing Provider::Openai output
(document_type from Import::DOCUMENT_TYPES, summary, extracted_data).
- Provider::Anthropic::BankStatementExtractor returns the same
{ transactions, period, account_holder, account_number, bank_name,
opening_balance, closing_balance } shape via report_bank_statement so
downstream pdf_import code is provider-agnostic.
- Both attach the PDF as
{ type: "document", source: { type: "base64", media_type: "application/pdf", data: <b64> } }
— Claude 3.5+ / 4.x accept this natively (up to 32MB / 100 pages).
No pdf-reader, no pdftoppm, no chunking for typical statements.
- supports_pdf_processing? (introduced in PR 1) already returns true for
claude-* models, gating process_pdf with a clear error otherwise.
- Cost ledger rows are persisted via the shared UsageRecorder concern,
including cache_creation/cache_read tokens.
Tests verify the document block shape, tool_choice forcing, normalized
document_type for unknown classifications, transaction normalization
(date / amount / reference → notes), and the missing-tool_use error
path. Blank pdf_content raises before any client call.
Stacked on #1984 (PR 2/5). 4/5 pgvector RAG next.
* fix(ai): guard PDF size + surface bank-statement truncation
- PdfProcessor and BankStatementExtractor raise upfront when
pdf_content.bytesize exceeds MAX_PDF_BYTES (32 MB, matching
Anthropic's hard limit). Previously a 100 MB PDF would be
base64-encoded (~133 MB) and packed into the JSON body before
the API rejected it — peak heap ~270 MB per Sidekiq worker.
- BankStatementExtractor inspects response.stop_reason; when the
model hit max_tokens it logs a warning and flags result[:truncated]
so downstream callers know the transaction list may be incomplete.
- ISO date pattern added to statement_period_start/end schema in
PdfProcessor so the model can't return "March 2026" — Anthropic
enforces the regex via the tool's input_schema.
Tests cover the size guard (raises before any client.messages call),
truncated-result flagging, and the warning log path.
* test(ai): require "ostruct" in Anthropic PDF tests
Match the explicit ostruct require added in PR1/PR2 — same Ruby 3.5+
load-path reason.
* feat(ai): default Anthropic installs to pgvector RAG (4/5)
The provider-agnostic vector store stack (VectorStore::Pgvector + the
Embeddable concern) already shipped to main. This PR closes the
Anthropic loop:
- VectorStore::Registry.adapter_name now returns :pgvector when
Setting.llm_provider == "anthropic" and no explicit
VECTOR_STORE_PROVIDER override is set. Anthropic has no hosted vector
store, so falling back to the local pgvector adapter is the only
correct default. Explicit VECTOR_STORE_PROVIDER still wins.
- SearchFamilyFiles surfaces a longer message when no adapter is wired
up — calling out pgvector + EMBEDDING_URI_BASE as the supported
Anthropic-only path so the user is not stuck with an "OpenAI required"
hint that is no longer accurate.
The Embeddable concern already pulls embeddings from
EMBEDDING_URI_BASE / EMBEDDING_ACCESS_TOKEN (with OpenAI as fallback),
so Anthropic installs point this at Voyage AI, a local Ollama instance,
or OpenAI embeddings — independent of the chat provider.
Tests cover the new default routing, the existing OpenAI default
staying intact, and explicit VECTOR_STORE_PROVIDER overriding the
Anthropic default.
Stacked on #1985 (PR 3/5). 5/5 settings UI + retention disclosure next.
* fix(ai): provision pgvector table when it is the default store
#1986 makes pgvector the default vector store for Anthropic installs, but
CreateVectorStoreChunks only ran when VECTOR_STORE_PROVIDER=pgvector was set
explicitly — so a fresh Anthropic-only install migrated without the
vector_store_chunks table and failed on uploads/searches.
Add VectorStore::Registry.pgvector_effective? as the single source of truth
for "is pgvector active?" (explicit env OR the Anthropic default), and a new
idempotent migration that enables the extension + creates the table whenever
pgvector is effective and the table is missing — covering fresh and
already-migrated installs without drift. Addresses Codex P1.
* fix(ai): provision pgvector table for Anthropic-default installs
Migration gated on raw VECTOR_STORE_PROVIDER==pgvector, so an
Anthropic-default install (which selects pgvector implicitly via
Setting.llm_provider without setting VECTOR_STORE_PROVIDER) skipped
table creation and failed later on a missing vector_store_chunks
relation. Route through VectorStore::Registry.pgvector_effective? —
the single source of truth already shared by the adapter selection.
Addresses Codex P1 review finding.
* fix(ai): provision pgvector chunks table on schema-load installs
The ensure-migration only helps db:migrate upgraders. Fresh installs go
through bin/docker-entrypoint's db:prepare, which loads schema.rb (the
conditional table can't be dumped there — it needs the vector extension)
and marks every migration applied without running it. An Anthropic-only
fresh install therefore selected the pgvector adapter but had no table,
failing with raw PG errors on first upload or search.
Two layers close it:
- VectorStore::Pgvector#ensure_schema! provisions the table idempotently
on first use (mirrors CreateVectorStoreChunks; memoized; failures wrap
in VectorStore::Error, which with_response turns into a clean failed
response).
- VectorStore::Registry#build_pgvector now gates on
VectorStore::Pgvector.available? (table exists, or extension present),
so installs whose Postgres lacks pgvector entirely degrade to the
assistant's provider_not_configured message instead of raising
mid-chat.
Also resolves the schema.rb version conflict against main (keep the
branch's 2026_06_01_120000, on top of main's current tables).
* fix(ai): address review nitpicks on pgvector provisioning
- Registry: update the adapter doc comment to mention the
Anthropic-to-pgvector default alongside the openai fallback.
- ensure_schema!: guard the DDL with if_not_exists instead of a Mutex.
Adapter instances are built per call and never shared across threads,
so the realistic race is two processes (web + Sidekiq) provisioning
concurrently; IF NOT EXISTS makes the loser a no-op where a Mutex
would only serialize threads inside one process.
|
||
|
Shibu M
|
ecf19ed1f9 | Merge remote-tracking branch 'upstream/main' into Transfer-charges | ||
|
Blaž Dular
|
94422955f8 |
feat(merchants): add raw data import (csv) for merchants (#1992)
* feat(merchants): add csv import endpoint for merchants * docs: update endpoint docs * fix(merchant): recommended ai fixes |
||
|
Shibu M
|
69f3212cac | removed other changes | ||
|
Shibu M
|
b923889103 | removed other changes | ||
|
Shibu M
|
c07c0b2e61 | change labels | ||
|
Shibu M
|
d917143fd6 | resloved issues raised by ai chatbot | ||
|
Shibu M
|
e73006af54 | Add transfer fee support for bank charges on account-to-account transfers | ||
|
ghost
|
683f518780 |
fix(balance-sheet): preserve disabled-account net worth history (#1730)
* fix(balance-sheet): preserve disabled account history * fix(balance-sheet): tighten historical account scope * fix(balance-sheet): stop disabled account carry-forward --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
ghost
|
6e04c6927d |
feat(imports): add SureImport session batches (#1785)
* feat(imports): add SureImport session batches Add first-class SureImport sessions for ordered multi-file NDJSON imports. Persist source mappings across chunks, make session/chunk processing idempotent, expose progress readback, and keep existing single-file import behavior compatible. Includes the devcontainer libvips runtime dependency needed by ActiveStorage variant tests. Addresses #1610. Related to #1458. * fix(imports): avoid scanner-like API key test data * test(imports): assert skipped balances are not persisted * fix(imports): harden session publish retries Validate expected import chunk sequences exactly before publish, and restore session state with error details when enqueueing the publish job fails. * fix(imports): close session retry edge cases Backfill expected chunk counts after client-session insert races and enqueue import-session jobs after the status transition commits. Persist a safe enqueue failure body so API readback does not expose raw queue errors. * fix(imports): address session publish review gaps Remove dead transaction external-id assignment, harden session publish retry/sync behavior, align session chunk status docs, and add regression coverage for partial retries and safe enqueue error readback. * fix(imports): include sessions in family reset Clear import sessions through the family reset job so chunk imports and source mappings do not survive a reset. Expose import session and source mapping counts in the reset status response and regenerated OpenAPI schema so polling reflects the full reset surface. * test(imports): cover split import mapping invariants * test(imports): cover session verification invariants * fix(imports): scope SureImport session reimports * Tighten SureImport session batching * fix(imports): export rule source ids for sessions * test(imports): stabilize rule id export assertion * test(imports): restore reset status session fixture |