mirror of
https://github.com/we-promise/sure.git
synced 2026-07-24 18:55:27 +00:00
* feat(sync): family-facing cancellation for syncs, imports, and exports Users had no way to stop or recover any background operation: a mistaken "Sync all" runs to completion, and an import or export whose job died (hard worker kills lose in-flight Sidekiq jobs) wedges with a spinner forever. Sync cancellation (cooperative — nothing is ever killed): - New syncs.cancel_requested_at column. Only the cancelled sync carries the flag: pending descendants are marked stale immediately (their queued jobs no-op via the existing may_start? guard), while descendants whose jobs are already executing finish their work honestly. - Family::Syncer stops fanning out child syncs once the flag is set (fresh read per iteration — the flag comes from the web process). - Finalization resolves a cancel-requested sync to stale instead of completed, which also skips post-sync (transfer matching, rules, broadcasts) via the existing stale gate. - The `visible` scope excludes cancel-requested syncs, so spinners clear immediately and — fixing a latent bug this feature would have amplified — sync_later no longer piggybacks a new sync request onto a dying sync it would silently swallow. - Cancel button appears next to "Sync all" on the accounts page while a family sync is visible. SyncsController#cancel scopes through Sync.for_family with resource_owner, so cross-family ids 404 and account-level syncs respect per-user account access. Stuck import/export self-service: - Import#force_fail! / FamilyExport#force_fail!: allowed only once the record has been idle past PRESUMED_LOST_AFTER (1 hour — dwarfs any legitimate run), and applied inside with_lock with a status re-check, so a job finishing between page render and button click wins. Imports fail into the existing retry path (reverting -> revert_failed keeps the revert retryable); PdfImports release their processing claim back to pending; exports fail so a new one can be created. - "Mark as failed" buttons appear on the imports/exports index rows only when a record is presumed lost, behind the pages' existing permission gates (statement-import permission for imports, admin for exports). * fix(sync-cancel): cascade pending cancels, guard late finalizers, scope provider syncs Review feedback on #2685 (CodeRabbit, Codex): - request_cancel! now cascades finalization for pending syncs too: a pending child resolved to stale never runs its job, so nothing else would ever call finalize_if_all_children_finalized — its waiting parent hung in syncing until the 24h sweep (CodeRabbit critical) - SimplefinItem::Syncer#mark_completed re-reads the sync under a row lock and skips finalization once cancellation was requested or the row went terminal — its in-memory copy predates the cancel, and the unguarded complete! (plus the raw status fallback) resurrected a cancelled sync and re-ran post-sync (Codex) - Cancelling provider-item syncs now requires admin: for_family's resource_owner only scopes the Account branch, so a restricted member could cancel admin-managed provider syncs spanning accounts they cannot see. Family- and account-level syncs stay member-cancellable, matching the buttons the UI shows (Codex) - Lost-import error copy moved behind i18n (imports.errors.presumed_lost), resolved at call time (CodeRabbit) - Tests: pending-child cancel finalizes the parent; late provider complete! cannot resurrect a cancelled sync; provider-sync cancellation is admin-only * fix(sync-cancel): capture the skipped-finalization case via DebugLogEntry CodeRabbit round-2: the mark_completed skip (cancelled/terminal sync) is support-relevant — record it in the super-admin debug UI with the family and provider attached instead of a raw Rails.logger line. category: provider_sync, matching the other provider syncers.
285 lines
7.6 KiB
Ruby
285 lines
7.6 KiB
Ruby
class PdfImport < Import
|
|
has_one_attached :pdf_file, dependent: :purge_later
|
|
|
|
validates :document_type, inclusion: { in: DOCUMENT_TYPES }, allow_nil: true
|
|
validate :account_statement_matches_import
|
|
|
|
class << self
|
|
def create_from_upload!(family:, file:, user:)
|
|
statement = AccountStatement.create_from_prepared_upload!(
|
|
family: family,
|
|
account: nil,
|
|
prepared_upload: AccountStatement.prepare_upload!(file)
|
|
)
|
|
|
|
create_from_statement!(statement: statement)
|
|
rescue AccountStatement::DuplicateUploadError => e
|
|
raise unless e.statement.manageable_by?(user)
|
|
|
|
create_from_statement!(statement: e.statement)
|
|
end
|
|
|
|
def create_from_statement!(statement:)
|
|
reusable_import = statement.latest_reusable_pdf_import
|
|
return reusable_import if reusable_import &&
|
|
reusable_import.account_id == statement.account_id &&
|
|
reusable_import.date_format == statement.family.date_format
|
|
|
|
create!(family: statement.family, account: statement.account, account_statement: statement, date_format: statement.family.date_format, status: :pending)
|
|
end
|
|
end
|
|
|
|
# A PdfImport's importing status is a processing claim (AI extraction or
|
|
# publish). Release a lost claim back to pending so the user can re-trigger
|
|
# processing, mirroring ProcessPdfJob's own reclaim; lost reverts keep the
|
|
# base revert_failed behavior.
|
|
def force_fail!(error_message = Import.lost_error_message)
|
|
return super if reverting?
|
|
|
|
with_lock do
|
|
return false unless presumed_lost?
|
|
|
|
update!(status: :pending)
|
|
end
|
|
|
|
true
|
|
end
|
|
|
|
def import!
|
|
raise "Account required for PDF import" unless account.present?
|
|
|
|
transaction do
|
|
mappings.each(&:create_mappable!)
|
|
|
|
new_transactions = rows.map do |row|
|
|
category = mappings.categories.mappable_for(row.category)
|
|
|
|
Transaction.new(
|
|
category: category,
|
|
entry: Entry.new(
|
|
account: account,
|
|
date: row.date_iso,
|
|
amount: row.signed_amount,
|
|
name: row.name,
|
|
currency: row.currency,
|
|
notes: row.notes,
|
|
import: self,
|
|
import_locked: true
|
|
)
|
|
)
|
|
end
|
|
|
|
Transaction.import!(new_transactions, recursive: true) if new_transactions.any?
|
|
end
|
|
end
|
|
|
|
def assign_account!(account)
|
|
transaction do
|
|
update!(account: account)
|
|
if (statement = account_statement)
|
|
statement.lock!
|
|
statement.link_to_account!(account) if statement.account_id != account.id
|
|
end
|
|
end
|
|
end
|
|
|
|
def pdf_uploaded?
|
|
statement_backed? || pdf_file.attached?
|
|
end
|
|
|
|
def ai_processed?
|
|
ai_summary.present?
|
|
end
|
|
|
|
def process_with_ai_later
|
|
return false unless with_lock { pending? && !ai_processed? && rows_count.zero? && pdf_uploaded? && update!(status: :importing) }
|
|
|
|
begin
|
|
ProcessPdfJob.perform_later(self)
|
|
true
|
|
rescue StandardError => e
|
|
Rails.logger.error("Failed to enqueue PDF processing for import #{id}: #{e.class.name} - #{e.message}")
|
|
reload.with_lock { update!(status: :pending) }
|
|
false
|
|
end
|
|
end
|
|
|
|
def process_with_ai
|
|
# Honors Setting.llm_provider (issue #2113) — Provider::Anthropic implements
|
|
# process_pdf (PR #1985).
|
|
provider = Provider::Registry.preferred_llm_provider
|
|
raise "AI provider not configured" unless provider
|
|
raise "AI provider does not support PDF processing" unless provider.supports_pdf_processing?
|
|
|
|
response = provider.process_pdf(
|
|
pdf_content: pdf_file_content,
|
|
family: family
|
|
)
|
|
|
|
unless response.success?
|
|
error_message = response.error&.message || "Unknown PDF processing error"
|
|
raise error_message
|
|
end
|
|
|
|
result = response.data
|
|
update!(
|
|
ai_summary: result.summary,
|
|
document_type: result.document_type
|
|
)
|
|
|
|
result
|
|
end
|
|
|
|
def extract_transactions
|
|
return unless statement_with_transactions?
|
|
|
|
# Honors Setting.llm_provider (issue #2113) — Provider::Anthropic implements
|
|
# extract_bank_statement (PR #1985).
|
|
provider = Provider::Registry.preferred_llm_provider
|
|
raise "AI provider not configured" unless provider
|
|
|
|
response = provider.extract_bank_statement(
|
|
pdf_content: pdf_file_content,
|
|
family: family
|
|
)
|
|
|
|
unless response.success?
|
|
error_message = response.error&.message || "Unknown extraction error"
|
|
raise error_message
|
|
end
|
|
|
|
update!(extracted_data: response.data)
|
|
response.data
|
|
end
|
|
|
|
def bank_statement?
|
|
document_type == "bank_statement"
|
|
end
|
|
|
|
def statement_with_transactions?
|
|
document_type.in?(%w[bank_statement credit_card_statement])
|
|
end
|
|
|
|
def has_extracted_transactions?
|
|
extracted_data.present? && extracted_data["transactions"].present?
|
|
end
|
|
|
|
def extracted_transactions
|
|
extracted_data&.dig("transactions") || []
|
|
end
|
|
|
|
def generate_rows_from_extracted_data
|
|
transaction do
|
|
rows.destroy_all
|
|
|
|
unless has_extracted_transactions?
|
|
update_column(:rows_count, 0)
|
|
return
|
|
end
|
|
|
|
currency = account&.currency || family.currency
|
|
|
|
mapped_rows = extracted_transactions.map.with_index(1) do |txn, index|
|
|
{
|
|
import_id: id,
|
|
source_row_number: index,
|
|
date: format_date_for_import(txn["date"]),
|
|
amount: txn["amount"].to_s,
|
|
name: txn["name"].to_s,
|
|
category: txn["category"].to_s,
|
|
notes: txn["notes"].to_s,
|
|
currency: currency
|
|
}
|
|
end
|
|
|
|
Import::Row.insert_all!(mapped_rows) if mapped_rows.any?
|
|
update_column(:rows_count, mapped_rows.size)
|
|
end
|
|
end
|
|
|
|
def send_next_steps_email(user)
|
|
PdfImportMailer.with(
|
|
user: user,
|
|
pdf_import: self
|
|
).next_steps.deliver_later
|
|
end
|
|
|
|
def uploaded?
|
|
pdf_uploaded?
|
|
end
|
|
|
|
def configured?
|
|
ai_processed? && rows_count > 0
|
|
end
|
|
|
|
def cleaned?
|
|
configured? && rows.all?(&:valid?)
|
|
end
|
|
|
|
def publishable?
|
|
account.present? && statement_with_transactions? && cleaned? && mappings.all?(&:valid?)
|
|
end
|
|
|
|
def cleaned_from_validation_stats?(invalid_rows_count:)
|
|
account.present? && statement_with_transactions? && super
|
|
end
|
|
|
|
def publishable_from_validation_stats?(invalid_rows_count:)
|
|
account.present? && statement_with_transactions? && super
|
|
end
|
|
|
|
def column_keys
|
|
%i[date amount name category notes]
|
|
end
|
|
|
|
def requires_csv_workflow?
|
|
false
|
|
end
|
|
|
|
def pdf_file_content
|
|
return @pdf_file_content if defined?(@pdf_file_content)
|
|
return @pdf_file_content = account_statement.original_file.download if statement_backed?
|
|
|
|
@pdf_file_content = pdf_file.download if pdf_file.attached?
|
|
end
|
|
|
|
def pdf_filename
|
|
return account_statement.filename if statement_backed?
|
|
|
|
pdf_file.filename.to_s if pdf_file.attached?
|
|
end
|
|
|
|
def statement_backed?
|
|
account_statement&.original_file&.attached?
|
|
end
|
|
|
|
def required_column_keys
|
|
%i[date amount]
|
|
end
|
|
|
|
def mapping_steps
|
|
base = []
|
|
# Only include CategoryMapping if rows have non-empty categories
|
|
base << Import::CategoryMapping if rows.where.not(category: [ nil, "" ]).exists?
|
|
# Note: PDF imports use direct account selection in the UI, not AccountMapping
|
|
# AccountMapping is designed for CSV imports where rows have different account values
|
|
base
|
|
end
|
|
|
|
private
|
|
|
|
def format_date_for_import(date_str)
|
|
return "" if date_str.blank?
|
|
|
|
Date.parse(date_str).strftime(date_format)
|
|
rescue ArgumentError
|
|
date_str.to_s
|
|
end
|
|
|
|
def account_statement_matches_import
|
|
return if account_statement.blank? || (account_statement.family_id == family_id && account_statement.pdf?)
|
|
|
|
errors.add(:account_statement, :invalid)
|
|
end
|
|
end
|