* fix Prism issue - assigned but unused variable plaid_item
* measurement to improve accounts_controller__index performance based on skylight
* fix FEEDBACK - Stop eager-loading full sync histories on accounts index
* fix FEEDBACK - Remove sticky memoization from SimplefinItem#accounts
* fix lint error in PR review
* fix FEEDBACK - Avoid returning map misses as authoritative results
* fix test error
* fix FEEDBACK - Avoid redundant query by using already-loaded @manual_accounts
* Address maintainer review on accounts index sync preloading
Memoize SimplefinItem#accounts, align syncing? with Sync#visible?,
and add fallback regression tests.
* Add composite index for sync DISTINCT ON queries
Support latest_by_syncable ordering with syncable_type, syncable_id,
created_at DESC, and id DESC.
* fix error in dockerfile.preview file
* Suppress Pipelock false positives on CI database fixtures.
Pipelock 2.7.0 full-repo audit flags ephemeral postgres:// URLs in
workflow env blocks. Re-apply inline suppressions lost in the main merge.
* Address review: guard partial Current sync maps and drop unrelated diff
Add key? checks so partially populated Current sync maps fall back to DB
queries. Revert Dockerfile.preview and pipelock.yml changes unrelated to
the accounts index N+1 optimization.
* fix(sync): fully populate Current sync maps for all preloaded syncables
* fix(ci): skip scheduled preview cleanup on forks
Only run the hourly Cloudflare preview cleanup on we-promise/sure,
where the required secrets exist.
* Remove schema.rb Postgres-version churn, keep only new syncs index
Reset db/schema.rb to upstream/main and re-add only the
index_syncs_on_syncable_and_created_at_and_id index. The prior diff
included ~30 lines of noise (check-constraint reformatting, virtual
column reformat, and column reorderings) caused by dumping under a
different PostgreSQL version, which obscured the single intended
schema change and invited merge conflicts.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat(sync): family-facing cancellation for syncs, imports, and exports
Users had no way to stop or recover any background operation: a
mistaken "Sync all" runs to completion, and an import or export whose
job died (hard worker kills lose in-flight Sidekiq jobs) wedges with a
spinner forever.
Sync cancellation (cooperative — nothing is ever killed):
- New syncs.cancel_requested_at column. Only the cancelled sync
carries the flag: pending descendants are marked stale immediately
(their queued jobs no-op via the existing may_start? guard), while
descendants whose jobs are already executing finish their work
honestly.
- Family::Syncer stops fanning out child syncs once the flag is set
(fresh read per iteration — the flag comes from the web process).
- Finalization resolves a cancel-requested sync to stale instead of
completed, which also skips post-sync (transfer matching, rules,
broadcasts) via the existing stale gate.
- The `visible` scope excludes cancel-requested syncs, so spinners
clear immediately and — fixing a latent bug this feature would have
amplified — sync_later no longer piggybacks a new sync request onto
a dying sync it would silently swallow.
- Cancel button appears next to "Sync all" on the accounts page while
a family sync is visible. SyncsController#cancel scopes through
Sync.for_family with resource_owner, so cross-family ids 404 and
account-level syncs respect per-user account access.
Stuck import/export self-service:
- Import#force_fail! / FamilyExport#force_fail!: allowed only once the
record has been idle past PRESUMED_LOST_AFTER (1 hour — dwarfs any
legitimate run), and applied inside with_lock with a status
re-check, so a job finishing between page render and button click
wins. Imports fail into the existing retry path (reverting ->
revert_failed keeps the revert retryable); PdfImports release their
processing claim back to pending; exports fail so a new one can be
created.
- "Mark as failed" buttons appear on the imports/exports index rows
only when a record is presumed lost, behind the pages' existing
permission gates (statement-import permission for imports, admin
for exports).
* fix(sync-cancel): cascade pending cancels, guard late finalizers, scope provider syncs
Review feedback on #2685 (CodeRabbit, Codex):
- request_cancel! now cascades finalization for pending syncs too: a
pending child resolved to stale never runs its job, so nothing else
would ever call finalize_if_all_children_finalized — its waiting
parent hung in syncing until the 24h sweep (CodeRabbit critical)
- SimplefinItem::Syncer#mark_completed re-reads the sync under a row
lock and skips finalization once cancellation was requested or the
row went terminal — its in-memory copy predates the cancel, and the
unguarded complete! (plus the raw status fallback) resurrected a
cancelled sync and re-ran post-sync (Codex)
- Cancelling provider-item syncs now requires admin: for_family's
resource_owner only scopes the Account branch, so a restricted member
could cancel admin-managed provider syncs spanning accounts they
cannot see. Family- and account-level syncs stay member-cancellable,
matching the buttons the UI shows (Codex)
- Lost-import error copy moved behind i18n
(imports.errors.presumed_lost), resolved at call time (CodeRabbit)
- Tests: pending-child cancel finalizes the parent; late provider
complete! cannot resurrect a cancelled sync; provider-sync
cancellation is admin-only
* fix(sync-cancel): capture the skipped-finalization case via DebugLogEntry
CodeRabbit round-2: the mark_completed skip (cancelled/terminal sync)
is support-relevant — record it in the super-admin debug UI with the
family and provider attached instead of a raw Rails.logger line.
category: provider_sync, matching the other provider syncers.
A sync marked stale by SyncCleanerJob while its job is still running
hits two lost-update paths when that job finishes:
- success path: finalize skipped the status transition but still ran
perform_post_sync, re-applying transfer matching, rules, and
broadcasts for a sync the system had already written off
- failure path: the rescue's unguarded fail! silently overwrote the
terminal stale status with failed (the in-memory record still read
syncing, so the AASM guard never fired)
Fix: re-check state under a row lock (with_lock reloads) before
failing, and skip post-sync for stale syncs in finalize. Post-sync
still runs for failed syncs — that behavior is intentional and
covered by existing tests.
* fix(jobs): delegate recurring-transaction sync gate to Sync.for_family
`IdentifyRecurringTransactionsJob#family_has_incomplete_syncs?` hand-rolled
the list of provider `*_items` associations it polled — plaid, simplefin,
lunchflow, enable_banking, sophtron — missing nine other `Syncable`
provider concerns on `Family`: coinbase, binance, kraken, coinstats,
snaptrade, mercury, brex, indexa_capital, ibkr. When a sync on any of those
nine was in flight, the debounce gate fell through and
`RecurringTransaction::Identifier` ran against a partial dataset; the
follow-up re-enqueue then hit the `find_or_initialize_by` upsert path and
inherited the stale `occurrence_count`. Same drift pattern that bolted
sophtron on as the 5th entry (#591) was already an iteration of.
The maintainers' own `Sync.for_family` (sync.rb:61) already enumerates every
`*_items` association via `Family.reflect_on_all_associations(:has_many)`
filtered by inclusion of `Syncable` — exactly the helper the gate should
delegate to so the list cannot drift again.
- Add `Sync.any_incomplete_for?(family)` class method that wraps
`for_family(family).incomplete.exists?`.
- Rewrite `family_has_incomplete_syncs?` to delegate. 14 lines → 1.
- New test file `test/jobs/identify_recurring_transactions_job_test.rb`
covers in-flight Coinbase + Mercury (gate fires), idle (identifier runs),
missing family, and superseded-by-newer-schedule.
- `test/models/sync_test.rb` gets 2 new tests pinning
`any_incomplete_for?` against a provider `_items` sync and a
family-itself sync.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(jobs): stub Rails.cache.read for supersession test (NullStore in test env)
`Rails.cache` is `ActiveSupport::Cache::NullStore` in the Rails test env, so
the previous test's `Rails.cache.write(cache_key, @scheduled_at + 10, ...)`
was a no-op and `Rails.cache.read(cache_key)` returned `nil`. The
supersession short-circuit `return if latest_scheduled && latest_scheduled
> scheduled_at` then fell through, the job proceeded to invoke
`RecurringTransaction::Identifier`, and the Mocha
`.expects(:identify_recurring_patterns).never` failed in CI.
Switch to `Rails.cache.stubs(:read).with(cache_key).returns(...)` — the
same idiom `test/models/provider/twelve_data_test.rb:186-197` already uses
for the cache layer. Add an `assert_nil` on the bare `perform` return so
Minitest's assertion counter sees an explicit assertion (silences the
"missing assertions" warning).
No production-code change. Behavior under test is unchanged; only the test
mechanism for simulating "newer scheduled run already in cache" is fixed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Basic plaid data model and linking
* Remove institutions, add plaid items
* Improve schema and Plaid provider
* Add webhook verification sketch
* Webhook verification
* Item accounts and balances sync setup
* Provide test encryption keys
* Fix test
* Only provide encryption keys in prod
* Try defining keys in test env
* Consolidate account sync logic
* Add back plaid account initialization
* Plaid transaction sync
* Sync UI overhaul for Plaid
* Add liability and investment syncing
* Handle investment webhooks and process current day holdings
* Remove logs
* Remove "all" period select for performance
* fix amount calc
* Remove todo comment
* Coming soon for investment historical data
* Document Plaid configuration
* Listen for holding updates
