mirror of
https://github.com/we-promise/sure.git
synced 2026-07-28 12:42:14 +00:00
5c0a7d8cbd
* feat(settings): super-admin background jobs console Neither managed nor self-hosted production deployments have any view into background job state (/sidekiq is only mounted outside production), so a stuck sync, import, or export is invisible until a user complains — and even then there is no way to act on it. Adds /settings/background_jobs, gated by the same super-admin Admin::BaseController as /settings/debug: - Worker status header: Sidekiq processes, busy count, queue depths and latency, retry/scheduled/dead set sizes. Read through a fail-closed PORO (BackgroundJobConsole) — when Redis is unreachable the console says so and disables actions instead of pretending health (deliberate contrast to BackgroundJobHealth's fail-open). - In-flight operations across all families: incomplete Syncs, Imports importing/reverting, ImportSessions importing, FamilyExports pending/processing, each with a liveness verdict derived from Sidekiq::Workers (job GlobalIDs in running payloads). - One mutation: mark a presumed-lost operation as such (Sync → stale, Import → failed/revert_failed, PdfImport → claim released back to pending, ImportSession/FamilyExport → failed). Guard rails on the mutation, server-side re-checked: - refused while Redis state is unknown (fail closed) - refused while the record's job is visibly executing - refused until the record has been idle past 30 minutes, so a merely queued job cannot be shot down and then still run - refused for parent Syncs with children still in flight (a parent legitimately has no live job of its own while children run) - applied inside with_lock with a status re-check, so a job finishing between render and click wins - audited as a DebugLogEntry (actor, prior status, family) The cancel endpoint gets its own Rack Attack throttle since the console deliberately lives under /settings rather than the throttled /admin prefix (its 10 req/min limit would fight the page's polling). * fix(jobs-console): count waiting jobs as live and harden cancel paths Review feedback on #2682 (Codex, CodeRabbit): - Liveness now covers jobs sitting in queues, the retry set, and the scheduled set, not just visibly-executing workers — a job waiting out a backlog or retry backoff WILL run later, and most affected job classes don't abort on a flipped status, so cancelling invited duplicate work. The backlog scan is bounded (5k entries); a truncated scan fails closed like redis_error?. The liveness column shows "Queued" for these - find_record! resolves STI subclass names (TransactionImport, PdfImport, …) against a base-class whitelist via safe_constantize instead of a fixed name map, so non-UI callers naming the subclass don't 404 - A PdfImport stuck in reverting goes to revert_failed like every other import instead of being released to pending — pending presented a possibly half-reverted import as publishable again; only the AI extraction claim (importing) is released to pending - Redis-unreachable warning renders via DS::Alert; operation id cast to_s before splitting; admin-cancel error copy moved to i18n * fix(jobs-console): re-check the stuck window inside the cancel row lock CodeRabbit round-2: cancellable? evaluates Sidekiq liveness outside the with_lock transaction (re-running Redis calls under a row lock would be worse), so a worker picking the job up between the liveness check and the lock acquisition was invisible. Repeating the updated_at staleness check inside the lock closes that window — a freshly-started job touches the record, and the re-check refuses the cancel. * fix(jobs-console): resolve record_type without reflection, i18n nits Brakeman flagged safe_constantize on params[:record_type] as a High-confidence UnsafeReflection (ci/scan_ruby). Replace the constant lookup with a reverse lookup: find the id in each cancellable base table and require the claimed type to match the found record's class or its base class. STI subclass names still resolve; unknown types still 404. Also from DS Drift Patrol: - "Sync · " label in _operation.html.erb now goes through t(".sync_label", type:) - drop the redundant default: on background_jobs_label now that the key exists in the locale file
206 lines
6.5 KiB
Ruby
206 lines
6.5 KiB
Ruby
require "test_helper"
|
|
|
|
class Settings::BackgroundJobsControllerTest < ActionDispatch::IntegrationTest
|
|
setup do
|
|
stub_sidekiq
|
|
end
|
|
|
|
test "super admin can view the console" do
|
|
sign_in users(:sure_support_staff)
|
|
|
|
get settings_background_jobs_path
|
|
|
|
assert_response :success
|
|
end
|
|
|
|
test "console renders in-flight operations with actions" do
|
|
sign_in users(:sure_support_staff)
|
|
|
|
stuck = imports(:transaction)
|
|
stuck.update_columns(status: "importing", updated_at: 1.hour.ago)
|
|
|
|
fresh_sync = Sync.create!(syncable: accounts(:depository), status: :syncing)
|
|
|
|
get settings_background_jobs_path
|
|
|
|
assert_response :success
|
|
assert_match stuck.id, response.body
|
|
assert_match fresh_sync.id, response.body
|
|
assert_match I18n.t("settings.background_jobs.operation.actions.mark_failed"), response.body
|
|
end
|
|
|
|
test "family admin is redirected away" do
|
|
sign_in users(:family_admin)
|
|
|
|
get settings_background_jobs_path
|
|
|
|
assert_redirected_to root_path
|
|
end
|
|
|
|
test "member is redirected away" do
|
|
sign_in users(:family_member)
|
|
|
|
get settings_background_jobs_path
|
|
|
|
assert_redirected_to root_path
|
|
end
|
|
|
|
test "cancel marks a stuck import as failed and writes an audit entry" do
|
|
sign_in users(:sure_support_staff)
|
|
|
|
import = imports(:transaction)
|
|
import.update_columns(status: "importing", updated_at: 1.hour.ago)
|
|
|
|
assert_difference "DebugLogEntry.count", 1 do
|
|
post cancel_settings_background_jobs_path(record_type: "Import", id: import.id)
|
|
end
|
|
|
|
assert_redirected_to settings_background_jobs_path
|
|
assert_equal "failed", import.reload.status
|
|
|
|
entry = DebugLogEntry.order(:created_at).last
|
|
assert_equal "background_jobs", entry.category
|
|
assert_equal users(:sure_support_staff).id, entry.metadata["actor_user_id"]
|
|
end
|
|
|
|
test "cancel releases a stuck PdfImport claim back to pending" do
|
|
sign_in users(:sure_support_staff)
|
|
|
|
pdf = imports(:pdf)
|
|
pdf.update_columns(status: "importing", updated_at: 1.hour.ago)
|
|
|
|
post cancel_settings_background_jobs_path(record_type: "Import", id: pdf.id)
|
|
|
|
assert_equal "pending", pdf.reload.status
|
|
end
|
|
|
|
test "cancel marks a stuck sync as stale" do
|
|
sign_in users(:sure_support_staff)
|
|
|
|
sync = Sync.create!(syncable: accounts(:depository), status: :syncing)
|
|
sync.update_columns(updated_at: 1.hour.ago)
|
|
|
|
post cancel_settings_background_jobs_path(record_type: "Sync", id: sync.id)
|
|
|
|
assert_equal "stale", sync.reload.status
|
|
end
|
|
|
|
test "cancel refuses a record inside the stuck window" do
|
|
sign_in users(:sure_support_staff)
|
|
|
|
import = imports(:transaction)
|
|
import.update_columns(status: "importing", updated_at: 1.minute.ago)
|
|
|
|
post cancel_settings_background_jobs_path(record_type: "Import", id: import.id)
|
|
|
|
assert_equal "importing", import.reload.status
|
|
assert_equal I18n.t("settings.background_jobs.cancel.not_cancellable"), flash[:alert]
|
|
end
|
|
|
|
test "cancel refuses when the record's job is visibly running" do
|
|
sign_in users(:sure_support_staff)
|
|
|
|
import = imports(:transaction)
|
|
import.update_columns(status: "importing", updated_at: 1.hour.ago)
|
|
|
|
stub_sidekiq(worker_payloads: [
|
|
{ "wrapped" => "ImportJob", "args" => [ { "arguments" => [ { "_aj_globalid" => import.to_global_id.to_s } ] } ] }
|
|
])
|
|
|
|
post cancel_settings_background_jobs_path(record_type: "Import", id: import.id)
|
|
|
|
assert_equal "importing", import.reload.status
|
|
end
|
|
|
|
test "cancel resolves STI subclass record types" do
|
|
sign_in users(:sure_support_staff)
|
|
|
|
import = imports(:transaction)
|
|
import.update_columns(status: "importing", updated_at: 1.hour.ago)
|
|
|
|
post cancel_settings_background_jobs_path(record_type: "TransactionImport", id: import.id)
|
|
|
|
assert_redirected_to settings_background_jobs_path
|
|
assert_equal "failed", import.reload.status
|
|
end
|
|
|
|
test "cancel routes a stuck PdfImport revert to revert_failed, not pending" do
|
|
sign_in users(:sure_support_staff)
|
|
|
|
pdf = imports(:pdf)
|
|
pdf.update_columns(status: "reverting", updated_at: 1.hour.ago)
|
|
|
|
post cancel_settings_background_jobs_path(record_type: "Import", id: pdf.id)
|
|
|
|
assert_equal "revert_failed", pdf.reload.status
|
|
end
|
|
|
|
test "cancel refuses when the record's job is waiting in a queue" do
|
|
sign_in users(:sure_support_staff)
|
|
|
|
import = imports(:transaction)
|
|
import.update_columns(status: "importing", updated_at: 1.hour.ago)
|
|
|
|
queue = mock("Queue")
|
|
queue.stubs(name: "default", size: 1, latency: 0.0)
|
|
queue.stubs(:each).multiple_yields([ stub(item: { "wrapped" => "ImportJob", "args" => [ { "arguments" => [ { "_aj_globalid" => import.to_global_id.to_s } ] } ] }) ])
|
|
Sidekiq::Queue.stubs(:all).returns([ queue ])
|
|
|
|
post cancel_settings_background_jobs_path(record_type: "Import", id: import.id)
|
|
|
|
assert_equal "importing", import.reload.status
|
|
assert_equal I18n.t("settings.background_jobs.cancel.not_cancellable"), flash[:alert]
|
|
end
|
|
|
|
test "cancel refuses unknown record types" do
|
|
sign_in users(:sure_support_staff)
|
|
|
|
post cancel_settings_background_jobs_path(record_type: "User", id: users(:family_admin).id)
|
|
|
|
assert_response :not_found
|
|
end
|
|
|
|
test "cancel requires super admin" do
|
|
sign_in users(:family_admin)
|
|
|
|
import = imports(:transaction)
|
|
import.update_columns(status: "importing", updated_at: 1.hour.ago)
|
|
|
|
post cancel_settings_background_jobs_path(record_type: "Import", id: import.id)
|
|
|
|
assert_redirected_to root_path
|
|
assert_equal "importing", import.reload.status
|
|
end
|
|
|
|
private
|
|
def stub_sidekiq(worker_payloads: [])
|
|
process_set = mock("ProcessSet")
|
|
process_set.stubs(:size).returns(1)
|
|
process_set.stubs(:sum).returns(worker_payloads.size)
|
|
Sidekiq::ProcessSet.stubs(:new).returns(process_set)
|
|
|
|
stats = mock("Stats")
|
|
stats.stubs(:enqueued).returns(0)
|
|
stats.stubs(:retry_size).returns(0)
|
|
stats.stubs(:dead_size).returns(0)
|
|
stats.stubs(:scheduled_size).returns(0)
|
|
Sidekiq::Stats.stubs(:new).returns(stats)
|
|
|
|
Sidekiq::Queue.stubs(:all).returns([])
|
|
|
|
empty_set = mock("JobSet")
|
|
empty_set.stubs(:each)
|
|
Sidekiq::RetrySet.stubs(:new).returns(empty_set)
|
|
Sidekiq::ScheduledSet.stubs(:new).returns(empty_set)
|
|
|
|
workers = mock("Workers")
|
|
yields = worker_payloads.map { |payload| [ "process", "thread", { "payload" => payload.to_json } ] }
|
|
if yields.any?
|
|
workers.stubs(:each).multiple_yields(*yields)
|
|
else
|
|
workers.stubs(:each)
|
|
end
|
|
Sidekiq::Workers.stubs(:new).returns(workers)
|
|
end
|
|
end
|