mirror of
https://github.com/we-promise/sure.git
synced 2026-07-28 04:32:12 +00:00
* feat(jobs): reap imports and exports stuck by lost background jobs A hard-killed worker (OOM, SIGKILL during deploy) loses its in-flight Sidekiq job permanently. Sync is the only model with a stale sweep; everything else wedges in a non-terminal status forever: - Import stuck "importing"/"reverting" (e.g. #2274) - ImportSession stuck "importing" — unrecoverable, publish_later refuses to re-publish while importing - FamilyExport stuck "pending"/"processing" — exports index polls every 3s indefinitely - PdfImport's AI-processing claim held forever (ProcessPdfJob's reclaim only runs if the job is redelivered) - Provider activities_fetch_pending flags stranded when a self-rescheduling fetch chain loses a link SyncCleanerJob (existing hourly cron) now sweeps all of them, each isolated so one failure doesn't block the rest, and records every reaped record as a DebugLogEntry with the family attached. Idempotency guards so a stray or redelivered job cannot corrupt a record the reaper (or a user retry) has since moved on: - Import#publish skips complete/reverting/revert_failed imports — a redelivered ImportJob after completion double-applies data for import types without row dedup. Failed imports deliberately stay publishable: their transaction rolled back, so a re-run is a retry. - FamilyDataExportJob refuses terminal exports but still allows processing ones through, since graceful-shutdown redelivery is what completes them. Reaper thresholds (6h imports, 2h exports) key off updated_at and dwarf legitimate runtimes, so live jobs are not swept in practice. * fix(jobs): make the reapers race-safe and commit-aware Review feedback on #2681 (jjmata, Codex, CodeRabbit): - Every sweep now mutates under record.with_lock with a staleness re-check, mirroring the guard Sync#perform gained in #2680 — a job finishing between the sweep query and the write can no longer be clobbered mid-flight - Import.clean distinguishes which side of import!'s single transaction the worker died on: rows attached means the data committed and only the status write was lost, so the record is finalized as complete (marking it failed invited a re-publish that double-imports types without row dedup, e.g. TradeImport); no rows means a clean rollback and the failed/try-again path stays - PdfImport.clean applies the same split: no rows → the AI-extraction claim died, reclaim to pending; rows → the publish died post-commit, finalize complete instead of letting the same extracted rows be published twice. Stuck PdfImport reverts (previously unswept by either clean) now go to revert_failed like every other import - ImportSession.clean reconciles chunks whose import! committed but never got the complete-status write before failing the session, so re-publish skips them instead of duplicating their rows via SureImport's split path - Import#publish redelivery skip is captured via DebugLogEntry instead of Rails.logger so support can see it in /settings/debug - The interrupted-error copy is i18n-backed (imports.errors.interrupted) * fix(jobs): round-2 review feedback on the reapers - Import.clean excludes session-owned chunks (import_session_id: nil) — SyncCleanerJob runs it before ImportSession.clean, so it could finalize or fail a SureImport chunk outside the session flow that owns its lifecycle. Regression test added (CodeRabbit major) - family.sync_later moved outside the row-lock transaction in both reap paths — Rails doesn't defer enqueues to after-commit by default, so Sidekiq could pick the job up before the status write was visible - Per-record rescue in Import.clean / PdfImport.clean so one bad record doesn't abort the rest of the hourly sweep - ImportSession.clean uses the importing? enum predicate; reconcile_committed_chunks! iterates with each (default-ordered association made find_each warn, and chunk counts are tiny) * fix(jobs): address round-3 reaper review (isolation + commit signal) Per-record rescue isolation for the three sweeps that lacked it — FamilyExport.clean, ImportSession.clean, and SyncCleanerJob's activity-flag loop — mirroring Import.clean/PdfImport.clean. One bad record (validation error, DB blip) no longer aborts the rest of that sweep for the hour; the activity-flag guard is per-record so a failure also stops skipping the models that follow. data_committed? now covers Category/Rule/Merchant imports, whose records hang off the family rather than the import (no entries or accounts), via the new Import#committed_by_named_records? helper. A committed one is reaped to complete instead of a retryable failed; nameless RuleImport rows carry no stable key, so a nameless-only file has no commit signal and stays retryable. * fix(schema): drop duplicate enable_banking_accounts columns from merge The merge of main into this branch re-appended product, credit_limit, and identification_hashes after updated_at, so schema.rb declared each twice and db:schema:load raised "you can't define an already defined column 'product'". Removed the duplicate declarations (kept the new treat_balance_as_available_credit column); the test database loads again.
123 lines
3.3 KiB
Ruby
123 lines
3.3 KiB
Ruby
class CategoryImport < Import
|
|
def import!
|
|
transaction do
|
|
rows.each do |row|
|
|
category_name = row.name.to_s.strip
|
|
category = family.categories.find_or_initialize_by(name: category_name)
|
|
category.color = row.category_color.presence || category.color || Category::UNCATEGORIZED_COLOR
|
|
category.lucide_icon = row.category_icon.presence || category.lucide_icon || "shapes"
|
|
category.parent = nil
|
|
category.save!
|
|
|
|
ensure_placeholder_category(row.category_parent)
|
|
end
|
|
|
|
rows.each do |row|
|
|
category = family.categories.find_by!(name: row.name.to_s.strip)
|
|
parent = ensure_placeholder_category(row.category_parent)
|
|
|
|
if parent && parent == category
|
|
errors.add(:base, :own_parent, name: category.name)
|
|
raise ActiveRecord::RecordInvalid.new(self)
|
|
end
|
|
|
|
next if category.parent == parent
|
|
|
|
category.update!(parent: parent)
|
|
end
|
|
end
|
|
end
|
|
|
|
# Categories hang off the family, not the import — see
|
|
# Import#committed_by_named_records?.
|
|
def data_committed?
|
|
committed_by_named_records?(family.categories)
|
|
end
|
|
|
|
def column_keys
|
|
%i[name category_color category_parent category_icon]
|
|
end
|
|
|
|
def required_column_keys
|
|
%i[name]
|
|
end
|
|
|
|
def mapping_steps
|
|
[]
|
|
end
|
|
|
|
def dry_run
|
|
{ categories: rows_count }
|
|
end
|
|
|
|
def csv_template
|
|
template = <<-CSV
|
|
name*,color,parent_category,lucide_icon
|
|
Food & Drink,#f97316,,carrot
|
|
Groceries,#407706,Food & Drink,shopping-basket
|
|
Salary,#22c55e,,briefcase
|
|
CSV
|
|
|
|
CSV.parse(template, headers: true)
|
|
end
|
|
|
|
def generate_rows_from_csv
|
|
rows.destroy_all
|
|
|
|
validate_required_headers!
|
|
|
|
name_header = header_for("name")
|
|
color_header = header_for("color")
|
|
parent_header = header_for("parent_category", "parent category")
|
|
icon_header = header_for("lucide_icon", "lucide icon", "icon")
|
|
|
|
csv_rows.each.with_index(1) do |row, index|
|
|
rows.create!(
|
|
source_row_number: index,
|
|
name: row[name_header].to_s.strip,
|
|
category_color: row[color_header].to_s.strip,
|
|
category_parent: row[parent_header].to_s.strip,
|
|
category_icon: row[icon_header].to_s.strip,
|
|
currency: default_currency
|
|
)
|
|
end
|
|
end
|
|
|
|
private
|
|
def validate_required_headers!
|
|
missing_headers = required_column_keys.map(&:to_s).reject { |key| header_for(key).present? }
|
|
return if missing_headers.empty?
|
|
|
|
errors.add(:base, :missing_columns, columns: missing_headers.join(", "))
|
|
raise ActiveRecord::RecordInvalid.new(self)
|
|
end
|
|
|
|
def header_for(*candidates)
|
|
candidates.each do |candidate|
|
|
normalized = normalize_header(candidate)
|
|
header = normalized_headers[normalized]
|
|
return header if header.present?
|
|
end
|
|
|
|
nil
|
|
end
|
|
|
|
def normalized_headers
|
|
@normalized_headers ||= csv_headers.to_h { |header| [ normalize_header(header), header ] }
|
|
end
|
|
|
|
def normalize_header(header)
|
|
header.to_s.strip.downcase.gsub(/\*/, "").gsub(/[\s-]+/, "_")
|
|
end
|
|
|
|
def ensure_placeholder_category(name)
|
|
trimmed_name = name.to_s.strip
|
|
return if trimmed_name.blank?
|
|
|
|
family.categories.find_or_create_by!(name: trimmed_name) do |placeholder|
|
|
placeholder.color = Category::UNCATEGORIZED_COLOR
|
|
placeholder.lucide_icon = "shapes"
|
|
end
|
|
end
|
|
end
|