mirror of
https://github.com/we-promise/sure.git
synced 2026-07-27 12:12:13 +00:00
9d4b4bc27c
* feat(jobs): reap imports and exports stuck by lost background jobs A hard-killed worker (OOM, SIGKILL during deploy) loses its in-flight Sidekiq job permanently. Sync is the only model with a stale sweep; everything else wedges in a non-terminal status forever: - Import stuck "importing"/"reverting" (e.g. #2274) - ImportSession stuck "importing" — unrecoverable, publish_later refuses to re-publish while importing - FamilyExport stuck "pending"/"processing" — exports index polls every 3s indefinitely - PdfImport's AI-processing claim held forever (ProcessPdfJob's reclaim only runs if the job is redelivered) - Provider activities_fetch_pending flags stranded when a self-rescheduling fetch chain loses a link SyncCleanerJob (existing hourly cron) now sweeps all of them, each isolated so one failure doesn't block the rest, and records every reaped record as a DebugLogEntry with the family attached. Idempotency guards so a stray or redelivered job cannot corrupt a record the reaper (or a user retry) has since moved on: - Import#publish skips complete/reverting/revert_failed imports — a redelivered ImportJob after completion double-applies data for import types without row dedup. Failed imports deliberately stay publishable: their transaction rolled back, so a re-run is a retry. - FamilyDataExportJob refuses terminal exports but still allows processing ones through, since graceful-shutdown redelivery is what completes them. Reaper thresholds (6h imports, 2h exports) key off updated_at and dwarf legitimate runtimes, so live jobs are not swept in practice. * fix(jobs): make the reapers race-safe and commit-aware Review feedback on #2681 (jjmata, Codex, CodeRabbit): - Every sweep now mutates under record.with_lock with a staleness re-check, mirroring the guard Sync#perform gained in #2680 — a job finishing between the sweep query and the write can no longer be clobbered mid-flight - Import.clean distinguishes which side of import!'s single transaction the worker died on: rows attached means the data committed and only the status write was lost, so the record is finalized as complete (marking it failed invited a re-publish that double-imports types without row dedup, e.g. TradeImport); no rows means a clean rollback and the failed/try-again path stays - PdfImport.clean applies the same split: no rows → the AI-extraction claim died, reclaim to pending; rows → the publish died post-commit, finalize complete instead of letting the same extracted rows be published twice. Stuck PdfImport reverts (previously unswept by either clean) now go to revert_failed like every other import - ImportSession.clean reconciles chunks whose import! committed but never got the complete-status write before failing the session, so re-publish skips them instead of duplicating their rows via SureImport's split path - Import#publish redelivery skip is captured via DebugLogEntry instead of Rails.logger so support can see it in /settings/debug - The interrupted-error copy is i18n-backed (imports.errors.interrupted) * fix(jobs): round-2 review feedback on the reapers - Import.clean excludes session-owned chunks (import_session_id: nil) — SyncCleanerJob runs it before ImportSession.clean, so it could finalize or fail a SureImport chunk outside the session flow that owns its lifecycle. Regression test added (CodeRabbit major) - family.sync_later moved outside the row-lock transaction in both reap paths — Rails doesn't defer enqueues to after-commit by default, so Sidekiq could pick the job up before the status write was visible - Per-record rescue in Import.clean / PdfImport.clean so one bad record doesn't abort the rest of the hourly sweep - ImportSession.clean uses the importing? enum predicate; reconcile_committed_chunks! iterates with each (default-ordered association made find_each warn, and chunk counts are tiny) * fix(jobs): address round-3 reaper review (isolation + commit signal) Per-record rescue isolation for the three sweeps that lacked it — FamilyExport.clean, ImportSession.clean, and SyncCleanerJob's activity-flag loop — mirroring Import.clean/PdfImport.clean. One bad record (validation error, DB blip) no longer aborts the rest of that sweep for the hour; the activity-flag guard is per-record so a failure also stops skipping the models that follow. data_committed? now covers Category/Rule/Merchant imports, whose records hang off the family rather than the import (no entries or accounts), via the new Import#committed_by_named_records? helper. A committed one is reaped to complete instead of a retryable failed; nameless RuleImport rows carry no stable key, so a nameless-only file has no commit signal and stays retryable. * fix(schema): drop duplicate enable_banking_accounts columns from merge The merge of main into this branch re-appended product, credit_limit, and identification_hashes after updated_at, so schema.rb declared each twice and db:schema:load raised "you can't define an already defined column 'product'". Removed the duplicate declarations (kept the new treat_balance_as_available_credit column); the test database loads again.
203 lines
6.7 KiB
Ruby
203 lines
6.7 KiB
Ruby
require "test_helper"
|
|
|
|
class ImportTest < ActiveSupport::TestCase
|
|
test "publish skips imports in terminal statuses" do
|
|
import = imports(:transaction)
|
|
import.update_columns(status: "complete")
|
|
|
|
import.expects(:import!).never
|
|
|
|
import.publish
|
|
|
|
assert_equal "complete", import.reload.status
|
|
end
|
|
|
|
test "publish skips imports that are reverting" do
|
|
import = imports(:transaction)
|
|
import.update_columns(status: "reverting")
|
|
|
|
import.expects(:import!).never
|
|
|
|
import.publish
|
|
|
|
assert_equal "reverting", import.reload.status
|
|
end
|
|
|
|
test "clean fails stuck imports but leaves PdfImports to their own reclaim" do
|
|
stuck_csv = imports(:transaction)
|
|
stuck_csv.update_columns(status: "importing", updated_at: 7.hours.ago)
|
|
|
|
stuck_pdf = imports(:pdf)
|
|
stuck_pdf.update_columns(status: "importing", updated_at: 7.hours.ago)
|
|
|
|
Import.clean
|
|
|
|
assert_equal "failed", stuck_csv.reload.status
|
|
assert_equal "importing", stuck_pdf.reload.status
|
|
end
|
|
|
|
test "clean leaves session-owned chunks to ImportSession.clean" do
|
|
family = families(:dylan_family)
|
|
session = family.import_sessions.create!(status: "importing")
|
|
chunk = TransactionImport.create!(
|
|
family: family,
|
|
import_session: session,
|
|
sequence: 1,
|
|
checksum: Digest::SHA256.hexdigest("chunk"),
|
|
status: "importing"
|
|
)
|
|
chunk.update_columns(updated_at: 7.hours.ago)
|
|
|
|
Import.clean
|
|
|
|
assert_equal "importing", chunk.reload.status
|
|
end
|
|
|
|
test "clean completes a stuck import whose data already committed" do
|
|
stuck = imports(:transaction)
|
|
stuck.update_columns(status: "importing", updated_at: 7.hours.ago)
|
|
entries(:transaction).update_columns(import_id: stuck.id)
|
|
|
|
Import.clean
|
|
|
|
stuck.reload
|
|
assert_equal "complete", stuck.status
|
|
assert_nil stuck.error
|
|
end
|
|
|
|
test "clean moves a stuck revert to revert_failed" do
|
|
stuck = imports(:transaction)
|
|
stuck.update_columns(status: "reverting", updated_at: 7.hours.ago)
|
|
|
|
Import.clean
|
|
|
|
assert_equal "revert_failed", stuck.reload.status
|
|
end
|
|
|
|
test "PdfImport clean reclaims the AI claim but completes a committed publish" do
|
|
ai_claim = imports(:pdf)
|
|
ai_claim.update_columns(status: "importing", updated_at: 7.hours.ago)
|
|
|
|
published = imports(:pdf_processed)
|
|
published.update_columns(status: "importing", updated_at: 7.hours.ago)
|
|
entries(:transaction).update_columns(import_id: published.id)
|
|
|
|
PdfImport.clean
|
|
|
|
assert_equal "pending", ai_claim.reload.status
|
|
assert_equal "complete", published.reload.status
|
|
end
|
|
|
|
test "PdfImport clean moves a stuck revert to revert_failed" do
|
|
stuck = imports(:pdf)
|
|
stuck.update_columns(status: "reverting", updated_at: 7.hours.ago)
|
|
|
|
PdfImport.clean
|
|
|
|
assert_equal "revert_failed", stuck.reload.status
|
|
end
|
|
|
|
test "force_fail! refuses records that have not been idle long enough" do
|
|
import = imports(:transaction)
|
|
import.update_columns(status: "importing", updated_at: 5.minutes.ago)
|
|
|
|
assert_not import.force_fail!
|
|
assert_equal "importing", import.reload.status
|
|
end
|
|
|
|
test "force_fail! fails a lost import and keeps reverts retryable" do
|
|
lost_import = imports(:transaction)
|
|
lost_import.update_columns(status: "importing", updated_at: 2.hours.ago)
|
|
|
|
assert lost_import.force_fail!
|
|
assert_equal "failed", lost_import.reload.status
|
|
assert_equal Import.lost_error_message, lost_import.error
|
|
|
|
lost_revert = imports(:trade)
|
|
lost_revert.update_columns(status: "reverting", updated_at: 2.hours.ago)
|
|
|
|
assert lost_revert.force_fail!
|
|
assert_equal "revert_failed", lost_revert.reload.status
|
|
end
|
|
|
|
test "force_fail! refuses terminal statuses" do
|
|
import = imports(:transaction)
|
|
import.update_columns(status: "complete", updated_at: 2.hours.ago)
|
|
|
|
assert_not import.force_fail!
|
|
assert_equal "complete", import.reload.status
|
|
end
|
|
|
|
test "force_fail! releases a lost PdfImport claim back to pending" do
|
|
pdf = imports(:pdf)
|
|
pdf.update_columns(status: "importing", updated_at: 2.hours.ago)
|
|
|
|
assert pdf.force_fail!
|
|
assert_equal "pending", pdf.reload.status
|
|
end
|
|
|
|
# Category/Merchant/Rule imports attach nothing to the import (their records
|
|
# hang off the family), so reap_stuck! leans on the type-specific
|
|
# data_committed? override to tell a committed run from a rolled-back one.
|
|
test "clean completes a stuck CategoryImport whose categories committed" do
|
|
family = families(:dylan_family)
|
|
stuck = CategoryImport.create!(family: family, status: "importing")
|
|
stuck.rows.create!(source_row_number: 1, name: categories(:income).name, currency: "USD")
|
|
stuck.update_columns(updated_at: 7.hours.ago)
|
|
|
|
Import.clean
|
|
|
|
stuck.reload
|
|
assert_equal "complete", stuck.status
|
|
assert_nil stuck.error
|
|
end
|
|
|
|
test "clean fails a stuck CategoryImport whose categories never committed" do
|
|
family = families(:dylan_family)
|
|
stuck = CategoryImport.create!(family: family, status: "importing")
|
|
stuck.rows.create!(source_row_number: 1, name: "Never Committed Category", currency: "USD")
|
|
stuck.update_columns(updated_at: 7.hours.ago)
|
|
|
|
Import.clean
|
|
|
|
assert_equal "failed", stuck.reload.status
|
|
end
|
|
|
|
test "clean completes a stuck MerchantImport whose merchants committed" do
|
|
family = families(:dylan_family)
|
|
stuck = MerchantImport.create!(family: family, status: "importing")
|
|
stuck.rows.create!(source_row_number: 1, name: merchants(:netflix).name, currency: "USD")
|
|
stuck.update_columns(updated_at: 7.hours.ago)
|
|
|
|
Import.clean
|
|
|
|
assert_equal "complete", stuck.reload.status
|
|
end
|
|
|
|
test "clean completes a stuck RuleImport whose named rules committed" do
|
|
family = families(:dylan_family)
|
|
# Skip Rule's own validations — the reaper only asks whether a rule with
|
|
# this name exists in the family, which is the state a committed import
|
|
# leaves behind.
|
|
family.rules.new(resource_type: "transaction", name: "Committed rule").save!(validate: false)
|
|
stuck = RuleImport.create!(family: family, status: "importing")
|
|
stuck.rows.create!(source_row_number: 1, name: "Committed rule", resource_type: "transaction", conditions: "[]", actions: "[]", currency: "USD")
|
|
stuck.update_columns(updated_at: 7.hours.ago)
|
|
|
|
Import.clean
|
|
|
|
assert_equal "complete", stuck.reload.status
|
|
end
|
|
|
|
test "clean fails a stuck RuleImport of only nameless rows (no commit signal)" do
|
|
family = families(:dylan_family)
|
|
stuck = RuleImport.create!(family: family, status: "importing")
|
|
stuck.rows.create!(source_row_number: 1, name: "", resource_type: "transaction", conditions: "[]", actions: "[]", currency: "USD")
|
|
stuck.update_columns(updated_at: 7.hours.ago)
|
|
|
|
Import.clean
|
|
|
|
assert_equal "failed", stuck.reload.status
|
|
end
|
|
end
|