mirror of
https://github.com/we-promise/sure.git
synced 2026-07-27 20:22:16 +00:00
* feat(settings): super-admin background jobs console Neither managed nor self-hosted production deployments have any view into background job state (/sidekiq is only mounted outside production), so a stuck sync, import, or export is invisible until a user complains — and even then there is no way to act on it. Adds /settings/background_jobs, gated by the same super-admin Admin::BaseController as /settings/debug: - Worker status header: Sidekiq processes, busy count, queue depths and latency, retry/scheduled/dead set sizes. Read through a fail-closed PORO (BackgroundJobConsole) — when Redis is unreachable the console says so and disables actions instead of pretending health (deliberate contrast to BackgroundJobHealth's fail-open). - In-flight operations across all families: incomplete Syncs, Imports importing/reverting, ImportSessions importing, FamilyExports pending/processing, each with a liveness verdict derived from Sidekiq::Workers (job GlobalIDs in running payloads). - One mutation: mark a presumed-lost operation as such (Sync → stale, Import → failed/revert_failed, PdfImport → claim released back to pending, ImportSession/FamilyExport → failed). Guard rails on the mutation, server-side re-checked: - refused while Redis state is unknown (fail closed) - refused while the record's job is visibly executing - refused until the record has been idle past 30 minutes, so a merely queued job cannot be shot down and then still run - refused for parent Syncs with children still in flight (a parent legitimately has no live job of its own while children run) - applied inside with_lock with a status re-check, so a job finishing between render and click wins - audited as a DebugLogEntry (actor, prior status, family) The cancel endpoint gets its own Rack Attack throttle since the console deliberately lives under /settings rather than the throttled /admin prefix (its 10 req/min limit would fight the page's polling). * fix(jobs-console): count waiting jobs as live and harden cancel paths Review feedback on #2682 (Codex, CodeRabbit): - Liveness now covers jobs sitting in queues, the retry set, and the scheduled set, not just visibly-executing workers — a job waiting out a backlog or retry backoff WILL run later, and most affected job classes don't abort on a flipped status, so cancelling invited duplicate work. The backlog scan is bounded (5k entries); a truncated scan fails closed like redis_error?. The liveness column shows "Queued" for these - find_record! resolves STI subclass names (TransactionImport, PdfImport, …) against a base-class whitelist via safe_constantize instead of a fixed name map, so non-UI callers naming the subclass don't 404 - A PdfImport stuck in reverting goes to revert_failed like every other import instead of being released to pending — pending presented a possibly half-reverted import as publishable again; only the AI extraction claim (importing) is released to pending - Redis-unreachable warning renders via DS::Alert; operation id cast to_s before splitting; admin-cancel error copy moved to i18n * fix(jobs-console): re-check the stuck window inside the cancel row lock CodeRabbit round-2: cancellable? evaluates Sidekiq liveness outside the with_lock transaction (re-running Redis calls under a row lock would be worse), so a worker picking the job up between the liveness check and the lock acquisition was invisible. Repeating the updated_at staleness check inside the lock closes that window — a freshly-started job touches the record, and the re-check refuses the cancel. * fix(jobs-console): resolve record_type without reflection, i18n nits Brakeman flagged safe_constantize on params[:record_type] as a High-confidence UnsafeReflection (ci/scan_ruby). Replace the constant lookup with a reverse lookup: find the id in each cancellable base table and require the claimed type to match the found record's class or its base class. STI subclass names still resolve; unknown types still 404. Also from DS Drift Patrol: - "Sync · " label in _operation.html.erb now goes through t(".sync_label", type:) - drop the redundant default: on background_jobs_label now that the key exists in the locale file
124 lines
4.3 KiB
Ruby
124 lines
4.3 KiB
Ruby
# frozen_string_literal: true
|
|
|
|
class Settings::BackgroundJobsController < Admin::BaseController
|
|
CANCELLABLE_BASE_TYPES = [ Sync, Import, ImportSession, FamilyExport ].freeze
|
|
|
|
def show
|
|
@breadcrumbs = [
|
|
[ t("breadcrumbs.home"), root_path ],
|
|
[ t("settings.background_jobs.show.page_title"), nil ]
|
|
]
|
|
|
|
@console = BackgroundJobConsole.new
|
|
@operations = @console.operations
|
|
end
|
|
|
|
def cancel
|
|
record = find_record!
|
|
console = BackgroundJobConsole.new
|
|
|
|
# Server-side re-check — the button state in the UI is not trusted. The
|
|
# with_lock block re-reads the record, so a job finishing between render
|
|
# and click cannot be clobbered (it already moved the status on). The
|
|
# stuck-window check repeats inside the lock too: cancellable? evaluated
|
|
# Sidekiq liveness outside the transaction (re-running Redis calls under
|
|
# a row lock would be worse), so a worker that grabbed the job in between
|
|
# shows up here as a freshly-touched updated_at.
|
|
cancelled = console.cancellable?(record) && record.with_lock do
|
|
if cancellable_status?(record) && record.updated_at <= BackgroundJobConsole::STUCK_AFTER.ago
|
|
prior_status = record.status
|
|
apply_cancel!(record)
|
|
audit_cancel!(record, prior_status)
|
|
true
|
|
else
|
|
false
|
|
end
|
|
end
|
|
|
|
if cancelled
|
|
redirect_to settings_background_jobs_path, notice: t(".cancelled", type: record.class.name)
|
|
else
|
|
redirect_to settings_background_jobs_path, alert: t(".not_cancellable")
|
|
end
|
|
end
|
|
|
|
private
|
|
# Resolves record_type without reflecting on request input: rather than
|
|
# turning the param into a constant, look the id up in each cancellable
|
|
# base table and require the claimed type to match the found record's
|
|
# class (or its base class). STI subclass names are still accepted —
|
|
# the UI sends base_class names, but a direct request naming e.g.
|
|
# TransactionImport shouldn't 404.
|
|
def find_record!
|
|
claimed_type = params[:record_type].to_s
|
|
|
|
CANCELLABLE_BASE_TYPES.each do |base|
|
|
record = base.find_by(id: params[:id])
|
|
return record if record && [ record.class.name, base.name ].include?(claimed_type)
|
|
end
|
|
|
|
raise ActiveRecord::RecordNotFound, "Unknown record type"
|
|
end
|
|
|
|
# User-facing: surfaces as the failed operation's error in the family UI.
|
|
def cancelled_error_message
|
|
t("settings.background_jobs.cancel.cancelled_error")
|
|
end
|
|
|
|
def cancellable_status?(record)
|
|
case record
|
|
when Sync then record.in_progress?
|
|
when Import then record.importing? || record.reverting?
|
|
when ImportSession then record.importing?
|
|
when FamilyExport then record.pending? || record.processing?
|
|
end
|
|
end
|
|
|
|
def apply_cancel!(record)
|
|
case record
|
|
when Sync
|
|
record.mark_stale!
|
|
when PdfImport
|
|
if record.reverting?
|
|
# A stuck revert may have half-deleted entries — pending would
|
|
# present the import as publishable again. Route it through the
|
|
# same revert_failed retry path as every other import.
|
|
record.update!(status: :revert_failed, error: cancelled_error_message)
|
|
else
|
|
# importing is the AI-processing claim — release it so the user
|
|
# can re-trigger, mirroring ProcessPdfJob's own reclaim.
|
|
record.update!(status: :pending)
|
|
end
|
|
when Import
|
|
record.update!(
|
|
status: record.reverting? ? :revert_failed : :failed,
|
|
error: cancelled_error_message
|
|
)
|
|
when ImportSession
|
|
record.update!(
|
|
status: :failed,
|
|
error_details: { "code" => "cancelled_by_admin", "message" => cancelled_error_message }
|
|
)
|
|
when FamilyExport
|
|
record.update!(status: :failed)
|
|
end
|
|
end
|
|
|
|
def audit_cancel!(record, prior_status)
|
|
DebugLogEntry.capture(
|
|
category: "background_jobs",
|
|
level: "warn",
|
|
message: "#{record.class.name} #{record.id} marked as lost from the background jobs console (was #{prior_status})",
|
|
source: self.class.name,
|
|
family: BackgroundJobConsole.family_for(record),
|
|
metadata: {
|
|
record_type: record.class.name,
|
|
record_id: record.id,
|
|
previous_status: prior_status,
|
|
new_status: record.status,
|
|
actor_user_id: Current.user.id
|
|
}
|
|
)
|
|
end
|
|
end
|