* feat(jobs): reap imports and exports stuck by lost background jobs
A hard-killed worker (OOM, SIGKILL during deploy) loses its in-flight
Sidekiq job permanently. Sync is the only model with a stale sweep;
everything else wedges in a non-terminal status forever:
- Import stuck "importing"/"reverting" (e.g. #2274)
- ImportSession stuck "importing" — unrecoverable, publish_later
refuses to re-publish while importing
- FamilyExport stuck "pending"/"processing" — exports index polls
every 3s indefinitely
- PdfImport's AI-processing claim held forever (ProcessPdfJob's
reclaim only runs if the job is redelivered)
- Provider activities_fetch_pending flags stranded when a
self-rescheduling fetch chain loses a link
SyncCleanerJob (existing hourly cron) now sweeps all of them, each
isolated so one failure doesn't block the rest, and records every
reaped record as a DebugLogEntry with the family attached.
Idempotency guards so a stray or redelivered job cannot corrupt a
record the reaper (or a user retry) has since moved on:
- Import#publish skips complete/reverting/revert_failed imports —
a redelivered ImportJob after completion double-applies data for
import types without row dedup. Failed imports deliberately stay
publishable: their transaction rolled back, so a re-run is a retry.
- FamilyDataExportJob refuses terminal exports but still allows
processing ones through, since graceful-shutdown redelivery is what
completes them.
Reaper thresholds (6h imports, 2h exports) key off updated_at and
dwarf legitimate runtimes, so live jobs are not swept in practice.
* fix(jobs): make the reapers race-safe and commit-aware
Review feedback on #2681 (jjmata, Codex, CodeRabbit):
- Every sweep now mutates under record.with_lock with a staleness
re-check, mirroring the guard Sync#perform gained in #2680 — a job
finishing between the sweep query and the write can no longer be
clobbered mid-flight
- Import.clean distinguishes which side of import!'s single transaction
the worker died on: rows attached means the data committed and only
the status write was lost, so the record is finalized as complete
(marking it failed invited a re-publish that double-imports types
without row dedup, e.g. TradeImport); no rows means a clean rollback
and the failed/try-again path stays
- PdfImport.clean applies the same split: no rows → the AI-extraction
claim died, reclaim to pending; rows → the publish died post-commit,
finalize complete instead of letting the same extracted rows be
published twice. Stuck PdfImport reverts (previously unswept by
either clean) now go to revert_failed like every other import
- ImportSession.clean reconciles chunks whose import! committed but
never got the complete-status write before failing the session, so
re-publish skips them instead of duplicating their rows via
SureImport's split path
- Import#publish redelivery skip is captured via DebugLogEntry instead
of Rails.logger so support can see it in /settings/debug
- The interrupted-error copy is i18n-backed (imports.errors.interrupted)
* fix(jobs): round-2 review feedback on the reapers
- Import.clean excludes session-owned chunks (import_session_id: nil) —
SyncCleanerJob runs it before ImportSession.clean, so it could
finalize or fail a SureImport chunk outside the session flow that
owns its lifecycle. Regression test added (CodeRabbit major)
- family.sync_later moved outside the row-lock transaction in both
reap paths — Rails doesn't defer enqueues to after-commit by default,
so Sidekiq could pick the job up before the status write was visible
- Per-record rescue in Import.clean / PdfImport.clean so one bad record
doesn't abort the rest of the hourly sweep
- ImportSession.clean uses the importing? enum predicate;
reconcile_committed_chunks! iterates with each (default-ordered
association made find_each warn, and chunk counts are tiny)
* fix(jobs): address round-3 reaper review (isolation + commit signal)
Per-record rescue isolation for the three sweeps that lacked it —
FamilyExport.clean, ImportSession.clean, and SyncCleanerJob's
activity-flag loop — mirroring Import.clean/PdfImport.clean. One bad
record (validation error, DB blip) no longer aborts the rest of that
sweep for the hour; the activity-flag guard is per-record so a failure
also stops skipping the models that follow.
data_committed? now covers Category/Rule/Merchant imports, whose
records hang off the family rather than the import (no entries or
accounts), via the new Import#committed_by_named_records? helper. A
committed one is reaped to complete instead of a retryable failed;
nameless RuleImport rows carry no stable key, so a nameless-only file
has no commit signal and stays retryable.
* fix(schema): drop duplicate enable_banking_accounts columns from merge
The merge of main into this branch re-appended product, credit_limit,
and identification_hashes after updated_at, so schema.rb declared each
twice and db:schema:load raised "you can't define an already defined
column 'product'". Removed the duplicate declarations (kept the new
treat_balance_as_available_credit column); the test database loads
again.
* feat(sync): family-facing cancellation for syncs, imports, and exports
Users had no way to stop or recover any background operation: a
mistaken "Sync all" runs to completion, and an import or export whose
job died (hard worker kills lose in-flight Sidekiq jobs) wedges with a
spinner forever.
Sync cancellation (cooperative — nothing is ever killed):
- New syncs.cancel_requested_at column. Only the cancelled sync
carries the flag: pending descendants are marked stale immediately
(their queued jobs no-op via the existing may_start? guard), while
descendants whose jobs are already executing finish their work
honestly.
- Family::Syncer stops fanning out child syncs once the flag is set
(fresh read per iteration — the flag comes from the web process).
- Finalization resolves a cancel-requested sync to stale instead of
completed, which also skips post-sync (transfer matching, rules,
broadcasts) via the existing stale gate.
- The `visible` scope excludes cancel-requested syncs, so spinners
clear immediately and — fixing a latent bug this feature would have
amplified — sync_later no longer piggybacks a new sync request onto
a dying sync it would silently swallow.
- Cancel button appears next to "Sync all" on the accounts page while
a family sync is visible. SyncsController#cancel scopes through
Sync.for_family with resource_owner, so cross-family ids 404 and
account-level syncs respect per-user account access.
Stuck import/export self-service:
- Import#force_fail! / FamilyExport#force_fail!: allowed only once the
record has been idle past PRESUMED_LOST_AFTER (1 hour — dwarfs any
legitimate run), and applied inside with_lock with a status
re-check, so a job finishing between page render and button click
wins. Imports fail into the existing retry path (reverting ->
revert_failed keeps the revert retryable); PdfImports release their
processing claim back to pending; exports fail so a new one can be
created.
- "Mark as failed" buttons appear on the imports/exports index rows
only when a record is presumed lost, behind the pages' existing
permission gates (statement-import permission for imports, admin
for exports).
* fix(sync-cancel): cascade pending cancels, guard late finalizers, scope provider syncs
Review feedback on #2685 (CodeRabbit, Codex):
- request_cancel! now cascades finalization for pending syncs too: a
pending child resolved to stale never runs its job, so nothing else
would ever call finalize_if_all_children_finalized — its waiting
parent hung in syncing until the 24h sweep (CodeRabbit critical)
- SimplefinItem::Syncer#mark_completed re-reads the sync under a row
lock and skips finalization once cancellation was requested or the
row went terminal — its in-memory copy predates the cancel, and the
unguarded complete! (plus the raw status fallback) resurrected a
cancelled sync and re-ran post-sync (Codex)
- Cancelling provider-item syncs now requires admin: for_family's
resource_owner only scopes the Account branch, so a restricted member
could cancel admin-managed provider syncs spanning accounts they
cannot see. Family- and account-level syncs stay member-cancellable,
matching the buttons the UI shows (Codex)
- Lost-import error copy moved behind i18n
(imports.errors.presumed_lost), resolved at call time (CodeRabbit)
- Tests: pending-child cancel finalizes the parent; late provider
complete! cannot resurrect a cancelled sync; provider-sync
cancellation is admin-only
* fix(sync-cancel): capture the skipped-finalization case via DebugLogEntry
CodeRabbit round-2: the mark_completed skip (cancelled/terminal sync)
is support-relevant — record it in the super-admin debug UI with the
family and provider attached instead of a raw Rails.logger line.
category: provider_sync, matching the other provider syncers.
* Remove stale 1.0 import logic and model
* Fresh start
* Checkpoint before removing nav
* First working prototype
* Add trade, account, and mint import flows
* Basic working version with tests
* System tests for each import type
* Clean up mappings flow
* Clean up PR, refactor stale code, tests
* Add back row validations
* Row validations
* Fix import job test
* Fix import navigation
* Fix mint import configuration form
* Currency preset for new accounts
* add col_sep to import model
* add validation for col_sep column
* add col_sep option to csv import model
* make use of col_sep option in import model
* add column separator field to new/edit action of an import
* add col_sep parameter to create/update action
* fix spacing between fields
Co-authored-by: Zach Gollwitzer <zach.gollwitzer@gmail.com>
Signed-off-by: Alexander Schrot <alexander@axs-labs.com>
---------
Signed-off-by: Alexander Schrot <alexander@axs-labs.com>
Co-authored-by: Zach Gollwitzer <zach.gollwitzer@gmail.com>
* Initial entryable models
* Update transfer and tests
* Update transaction controllers and tests
* Update sync process to use new entries model
* Get dashboard working again
* Update transfers, imports, and accounts to use Account::Entry
* Update system tests
* Consolidate transaction management into entries controller
* Add permitted partial key helper
* Move account transactions list to entries controller
* Delegate transaction entries search
* Move transfer relation to entry
* Update bulk transaction management flows to use entries
* Remove test code
* Test fix attempt
* Update demo data script
* Consolidate remaining transaction partials to entries
* Consolidate valuations controller to entries controller
* Lint fix
* Remove unused files, additional cleanup
* Add back valuation creation
* Make migrations fully reversible
* Stale routes cleanup
* Migrations reversible fix
* Move types to entryable concern
* Fix search when no entries found
* Remove more unused code
Introduces a basic CSV import module for bulk-importing account transactions.
Changes include:
- User can load a CSV
- User can configure the column mappings for a CSV
- Imported CSV shows invalid cells
- User can clean up their data directly in the UI
- User can see a preview of the import rows and confirm import
- Layout refactor + Import nav stepper
- System test stability improvements
