* feat(jobs): reap imports and exports stuck by lost background jobs
A hard-killed worker (OOM, SIGKILL during deploy) loses its in-flight
Sidekiq job permanently. Sync is the only model with a stale sweep;
everything else wedges in a non-terminal status forever:
- Import stuck "importing"/"reverting" (e.g. #2274)
- ImportSession stuck "importing" — unrecoverable, publish_later
refuses to re-publish while importing
- FamilyExport stuck "pending"/"processing" — exports index polls
every 3s indefinitely
- PdfImport's AI-processing claim held forever (ProcessPdfJob's
reclaim only runs if the job is redelivered)
- Provider activities_fetch_pending flags stranded when a
self-rescheduling fetch chain loses a link
SyncCleanerJob (existing hourly cron) now sweeps all of them, each
isolated so one failure doesn't block the rest, and records every
reaped record as a DebugLogEntry with the family attached.
Idempotency guards so a stray or redelivered job cannot corrupt a
record the reaper (or a user retry) has since moved on:
- Import#publish skips complete/reverting/revert_failed imports —
a redelivered ImportJob after completion double-applies data for
import types without row dedup. Failed imports deliberately stay
publishable: their transaction rolled back, so a re-run is a retry.
- FamilyDataExportJob refuses terminal exports but still allows
processing ones through, since graceful-shutdown redelivery is what
completes them.
Reaper thresholds (6h imports, 2h exports) key off updated_at and
dwarf legitimate runtimes, so live jobs are not swept in practice.
* fix(jobs): make the reapers race-safe and commit-aware
Review feedback on #2681 (jjmata, Codex, CodeRabbit):
- Every sweep now mutates under record.with_lock with a staleness
re-check, mirroring the guard Sync#perform gained in #2680 — a job
finishing between the sweep query and the write can no longer be
clobbered mid-flight
- Import.clean distinguishes which side of import!'s single transaction
the worker died on: rows attached means the data committed and only
the status write was lost, so the record is finalized as complete
(marking it failed invited a re-publish that double-imports types
without row dedup, e.g. TradeImport); no rows means a clean rollback
and the failed/try-again path stays
- PdfImport.clean applies the same split: no rows → the AI-extraction
claim died, reclaim to pending; rows → the publish died post-commit,
finalize complete instead of letting the same extracted rows be
published twice. Stuck PdfImport reverts (previously unswept by
either clean) now go to revert_failed like every other import
- ImportSession.clean reconciles chunks whose import! committed but
never got the complete-status write before failing the session, so
re-publish skips them instead of duplicating their rows via
SureImport's split path
- Import#publish redelivery skip is captured via DebugLogEntry instead
of Rails.logger so support can see it in /settings/debug
- The interrupted-error copy is i18n-backed (imports.errors.interrupted)
* fix(jobs): round-2 review feedback on the reapers
- Import.clean excludes session-owned chunks (import_session_id: nil) —
SyncCleanerJob runs it before ImportSession.clean, so it could
finalize or fail a SureImport chunk outside the session flow that
owns its lifecycle. Regression test added (CodeRabbit major)
- family.sync_later moved outside the row-lock transaction in both
reap paths — Rails doesn't defer enqueues to after-commit by default,
so Sidekiq could pick the job up before the status write was visible
- Per-record rescue in Import.clean / PdfImport.clean so one bad record
doesn't abort the rest of the hourly sweep
- ImportSession.clean uses the importing? enum predicate;
reconcile_committed_chunks! iterates with each (default-ordered
association made find_each warn, and chunk counts are tiny)
* fix(jobs): address round-3 reaper review (isolation + commit signal)
Per-record rescue isolation for the three sweeps that lacked it —
FamilyExport.clean, ImportSession.clean, and SyncCleanerJob's
activity-flag loop — mirroring Import.clean/PdfImport.clean. One bad
record (validation error, DB blip) no longer aborts the rest of that
sweep for the hour; the activity-flag guard is per-record so a failure
also stops skipping the models that follow.
data_committed? now covers Category/Rule/Merchant imports, whose
records hang off the family rather than the import (no entries or
accounts), via the new Import#committed_by_named_records? helper. A
committed one is reaped to complete instead of a retryable failed;
nameless RuleImport rows carry no stable key, so a nameless-only file
has no commit signal and stays retryable.
* fix(schema): drop duplicate enable_banking_accounts columns from merge
The merge of main into this branch re-appended product, credit_limit,
and identification_hashes after updated_at, so schema.rb declared each
twice and db:schema:load raised "you can't define an already defined
column 'product'". Removed the duplicate declarations (kept the new
treat_balance_as_available_credit column); the test database loads
again.
* Extract hardcoded strings to i18n
Replace numerous hardcoded English strings with I18n lookups (t / I18n.t) across controllers, views, helpers, and components, and convert model validation error messages to symbol keys. Added multiple locale files under config/locales for models and views. This centralizes user-facing notices/alerts, UI text, import/validation messages, and prepares the app for localization and easier translation maintenance.
* Update en.yml
* Update preview-cleanup.yml
* Revert "Update preview-cleanup.yml"
This reverts commit 1ba6d3c34c.
* test: align i18n assertions with translated messages
* Standardize balance error key and tweak locales
Replace SophtronAccount's :requires_balance error key with :no_balance and update related locale strings for sophtron, plaid, and simplefin accounts to use the new key and clearer copy. Also switch the QIF upload redirect notice to use a relative translation key (t('.qif_uploaded')), remove an unused SSO providers help line, and fix a trailing-newline/whitespace issue in the subscriptions locale. These changes standardize validation keys and improve translation consistency and messaging.
---------
Co-authored-by: KiloClaw <kiloclaw@openclaw.ai>
* Make categories global
This solves us A LOT of cash flow and budgeting problems.
* Update schema.rb
* Update auto_categorizer.rb
* Update income_statement.rb
* FIX budget sub-categories
* FIX sub-categories and tests
* Add 2 step migration
* Implement API v1 Imports controller
- Add Api::V1::ImportsController with index, show, and create actions
- Add Jbuilder views for index and show
- Add integration tests
- Implement row generation logic in create action
- Update routes
* Validate import account belongs to family
- Add validation to Import model to ensure account belongs to the same family
- Add regression test case in Api::V1::ImportsControllerTest
* updating docs to be more detailed
* Rescue StandardError instead of bare rescue in ImportsController
* Optimize Imports API and fix documentation
- Implement rows_count counter cache for Imports
- Preload rows in Api::V1::ImportsController#show
- Update documentation to show correct OAuth scopes
* Fix formatting in ImportsControllerTest
* Permit all import parameters and fix unknown attribute error
* Restore API routes for auth, chats, and messages
* removing pr summary
* Fix trailing whitespace and configured? test failure
- Update Import#configured? to use rows_count for performance and consistency
- Mock rows_count in TransactionImportTest
- Fix trailing whitespace in migration
* Harden security and fix mass assignment in ImportsController
- Handle type and account_id explicitly in create action
- Rename import_params to import_config_params for clarity
- Validate type against Import::TYPES
* Fix MintImport rows_count update and migration whitespace
- Update MintImport#generate_rows_from_csv to update rows_count counter cache
- Fix trailing whitespace and final newline in AddRowsCountToImports migration
* Implement full-screen Drag and Drop CSV import on Transactions page
- Add DragAndDropImport Stimulus controller listening on document
- Add full-screen overlay with icon and text to Transactions index
- Update ImportsController to handle direct file uploads via create action
- Add system test for drag and drop functionality
* Implement Drag and Drop CSV upload on Import Upload page
- Add drag-and-drop-import controller to import/uploads/show
- Add full-screen overlay to import/uploads/show
- Annotate upload form and input with drag-and-drop targets
- Add PR_SUMMARY.md
* removing pr summary
* Add file validation to ImportsController
- Validate file size (max 10MB) and MIME type in create action
- Prevent memory exhaustion and invalid file processing
- Defined MAX_CSV_SIZE and ALLOWED_MIME_TYPES in Import model
* Refactor dragLeave logic with counter pattern to prevent flickering
* Extract shared drag-and-drop overlay partial
- Create app/views/imports/_drag_drop_overlay.html.erb
- Update transactions/index and import/uploads/show to use the partial
- Reduce code duplication in views
* Update Brakeman and harden ImportsController security
- Update brakeman to 7.1.2
- Explicitly handle type assignment in ImportsController#create to avoid mass assignment
- Remove :type from permitted import parameters
* Fix trailing whitespace in DragAndDropImportTest
* Don't commit LLM comments as file
* FIX add api validation
---------
Co-authored-by: Carlos Adames <cj@Carloss-MacBook-Air.local>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: sokie <sokysrm@gmail.com>
* Add full import/export support for rules with versioned JSON schema
This commit implements comprehensive import/export functionality for rules,
allowing users to back up and restore their rule definitions.
Key features:
- Export rules to both CSV and NDJSON formats with versioned schema (v1)
- Import rules from CSV with full support for nested conditions and actions
- UUID to name mapping for categories and merchants for portability
- Support for compound conditions with sub-conditions
- Comprehensive test coverage for export and import functionality
- UI integration for rules import in the imports interface
Technical details:
- Extended Family::DataExporter to generate rules.csv and include rules in all.ndjson
- Created RuleImport model following the existing Import STI pattern
- Added migration for rule-specific columns in import_rows table
- Implemented serialization helpers to map UUIDs to human-readable names
- Added i18n support for the new import option
- Included versioning in NDJSON export to support future schema evolution
The implementation ensures rules can be safely exported from one family
and imported into another, even when category/merchant IDs differ,
by mapping between names and IDs during export/import.
* Fix AR migration version
* Mention support for rules export
* Rabbit suggestion
* Fix tests
* Missed schema.rb
* Fix sample CSV download for rule import
* Fix parsing in Rules import
* Fix tests
* Rule import message i18n
* Export tag names, not UUIDs
* Make sure tags are created if needed at import
* Avoid test errors when running in parallel
---------
Co-authored-by: Claude <noreply@anthropic.com>
