mirror of
https://github.com/we-promise/sure.git
synced 2026-08-06 00:52:16 +00:00
5c0a7d8cbd
* feat(settings): super-admin background jobs console Neither managed nor self-hosted production deployments have any view into background job state (/sidekiq is only mounted outside production), so a stuck sync, import, or export is invisible until a user complains — and even then there is no way to act on it. Adds /settings/background_jobs, gated by the same super-admin Admin::BaseController as /settings/debug: - Worker status header: Sidekiq processes, busy count, queue depths and latency, retry/scheduled/dead set sizes. Read through a fail-closed PORO (BackgroundJobConsole) — when Redis is unreachable the console says so and disables actions instead of pretending health (deliberate contrast to BackgroundJobHealth's fail-open). - In-flight operations across all families: incomplete Syncs, Imports importing/reverting, ImportSessions importing, FamilyExports pending/processing, each with a liveness verdict derived from Sidekiq::Workers (job GlobalIDs in running payloads). - One mutation: mark a presumed-lost operation as such (Sync → stale, Import → failed/revert_failed, PdfImport → claim released back to pending, ImportSession/FamilyExport → failed). Guard rails on the mutation, server-side re-checked: - refused while Redis state is unknown (fail closed) - refused while the record's job is visibly executing - refused until the record has been idle past 30 minutes, so a merely queued job cannot be shot down and then still run - refused for parent Syncs with children still in flight (a parent legitimately has no live job of its own while children run) - applied inside with_lock with a status re-check, so a job finishing between render and click wins - audited as a DebugLogEntry (actor, prior status, family) The cancel endpoint gets its own Rack Attack throttle since the console deliberately lives under /settings rather than the throttled /admin prefix (its 10 req/min limit would fight the page's polling). * fix(jobs-console): count waiting jobs as live and harden cancel paths Review feedback on #2682 (Codex, CodeRabbit): - Liveness now covers jobs sitting in queues, the retry set, and the scheduled set, not just visibly-executing workers — a job waiting out a backlog or retry backoff WILL run later, and most affected job classes don't abort on a flipped status, so cancelling invited duplicate work. The backlog scan is bounded (5k entries); a truncated scan fails closed like redis_error?. The liveness column shows "Queued" for these - find_record! resolves STI subclass names (TransactionImport, PdfImport, …) against a base-class whitelist via safe_constantize instead of a fixed name map, so non-UI callers naming the subclass don't 404 - A PdfImport stuck in reverting goes to revert_failed like every other import instead of being released to pending — pending presented a possibly half-reverted import as publishable again; only the AI extraction claim (importing) is released to pending - Redis-unreachable warning renders via DS::Alert; operation id cast to_s before splitting; admin-cancel error copy moved to i18n * fix(jobs-console): re-check the stuck window inside the cancel row lock CodeRabbit round-2: cancellable? evaluates Sidekiq liveness outside the with_lock transaction (re-running Redis calls under a row lock would be worse), so a worker picking the job up between the liveness check and the lock acquisition was invisible. Repeating the updated_at staleness check inside the lock closes that window — a freshly-started job touches the record, and the re-check refuses the cancel. * fix(jobs-console): resolve record_type without reflection, i18n nits Brakeman flagged safe_constantize on params[:record_type] as a High-confidence UnsafeReflection (ci/scan_ruby). Replace the constant lookup with a reverse lookup: find the id in each cancellable base table and require the claimed type to match the found record's class or its base class. STI subclass names still resolve; unknown types still 404. Also from DS Drift Patrol: - "Sync · " label in _operation.html.erb now goes through t(".sync_label", type:) - drop the redundant default: on background_jobs_label now that the key exists in the locale file
50 lines
2.3 KiB
Plaintext
50 lines
2.3 KiB
Plaintext
<% family = BackgroundJobConsole.family_for(operation) %>
|
|
<% running = console.running?(operation) %>
|
|
<tr>
|
|
<td class="px-4 py-3 text-sm text-primary whitespace-nowrap">
|
|
<%= operation.is_a?(Sync) ? t(".sync_label", type: operation.syncable_type) : operation.class.name %>
|
|
<p class="text-xs text-secondary font-mono" title="<%= operation.id %>"><%= operation.id.to_s.split("-").first %></p>
|
|
</td>
|
|
<td class="px-4 py-3 text-sm text-primary whitespace-nowrap"><%= family&.name || t(".missing_value") %></td>
|
|
<td class="px-4 py-3 text-sm text-primary whitespace-nowrap"><%= operation.status %></td>
|
|
<td class="px-4 py-3 text-sm text-secondary whitespace-nowrap">
|
|
<%= t(".ago", time: time_ago_in_words(operation.updated_at)) %>
|
|
</td>
|
|
<td class="px-4 py-3 text-sm whitespace-nowrap">
|
|
<% if running %>
|
|
<span class="flex items-center gap-2 text-primary">
|
|
<span class="animate-spin h-3 w-3 border-2 border-secondary border-t-transparent rounded-full"></span>
|
|
<%= t(".running") %>
|
|
</span>
|
|
<% elsif console.redis_error? %>
|
|
<span class="text-secondary"><%= t(".unknown") %></span>
|
|
<% elsif console.enqueued?(operation) %>
|
|
<span class="text-secondary"><%= t(".queued") %></span>
|
|
<% elsif operation.updated_at > BackgroundJobConsole::STUCK_AFTER.ago %>
|
|
<span class="text-secondary"><%= t(".recent") %></span>
|
|
<% else %>
|
|
<span class="flex items-center gap-2 text-primary">
|
|
<%= icon "alert-triangle", class: "w-4 h-4 text-warning" %>
|
|
<%= t(".stuck") %>
|
|
</span>
|
|
<% end %>
|
|
</td>
|
|
<td class="px-4 py-3 text-sm whitespace-nowrap text-right">
|
|
<% if console.cancellable?(operation) %>
|
|
<% action_key = case operation
|
|
when Sync then "mark_stale"
|
|
when PdfImport then "release_claim"
|
|
else "mark_failed"
|
|
end %>
|
|
<%= button_to cancel_settings_background_jobs_path(record_type: operation.class.base_class.name, id: operation.id),
|
|
method: :post,
|
|
class: "text-sm text-destructive hover:underline",
|
|
data: { turbo_confirm: t(".confirm"), turbo_frame: "_top" } do %>
|
|
<%= t(".actions.#{action_key}") %>
|
|
<% end %>
|
|
<% else %>
|
|
<span class="text-xs text-secondary"><%= t(".missing_value") %></span>
|
|
<% end %>
|
|
</td>
|
|
</tr>
|