mirror of
https://github.com/we-promise/sure.git
synced 2026-07-27 20:22:16 +00:00
* feat(jobs): reap imports and exports stuck by lost background jobs A hard-killed worker (OOM, SIGKILL during deploy) loses its in-flight Sidekiq job permanently. Sync is the only model with a stale sweep; everything else wedges in a non-terminal status forever: - Import stuck "importing"/"reverting" (e.g. #2274) - ImportSession stuck "importing" — unrecoverable, publish_later refuses to re-publish while importing - FamilyExport stuck "pending"/"processing" — exports index polls every 3s indefinitely - PdfImport's AI-processing claim held forever (ProcessPdfJob's reclaim only runs if the job is redelivered) - Provider activities_fetch_pending flags stranded when a self-rescheduling fetch chain loses a link SyncCleanerJob (existing hourly cron) now sweeps all of them, each isolated so one failure doesn't block the rest, and records every reaped record as a DebugLogEntry with the family attached. Idempotency guards so a stray or redelivered job cannot corrupt a record the reaper (or a user retry) has since moved on: - Import#publish skips complete/reverting/revert_failed imports — a redelivered ImportJob after completion double-applies data for import types without row dedup. Failed imports deliberately stay publishable: their transaction rolled back, so a re-run is a retry. - FamilyDataExportJob refuses terminal exports but still allows processing ones through, since graceful-shutdown redelivery is what completes them. Reaper thresholds (6h imports, 2h exports) key off updated_at and dwarf legitimate runtimes, so live jobs are not swept in practice. * fix(jobs): make the reapers race-safe and commit-aware Review feedback on #2681 (jjmata, Codex, CodeRabbit): - Every sweep now mutates under record.with_lock with a staleness re-check, mirroring the guard Sync#perform gained in #2680 — a job finishing between the sweep query and the write can no longer be clobbered mid-flight - Import.clean distinguishes which side of import!'s single transaction the worker died on: rows attached means the data committed and only the status write was lost, so the record is finalized as complete (marking it failed invited a re-publish that double-imports types without row dedup, e.g. TradeImport); no rows means a clean rollback and the failed/try-again path stays - PdfImport.clean applies the same split: no rows → the AI-extraction claim died, reclaim to pending; rows → the publish died post-commit, finalize complete instead of letting the same extracted rows be published twice. Stuck PdfImport reverts (previously unswept by either clean) now go to revert_failed like every other import - ImportSession.clean reconciles chunks whose import! committed but never got the complete-status write before failing the session, so re-publish skips them instead of duplicating their rows via SureImport's split path - Import#publish redelivery skip is captured via DebugLogEntry instead of Rails.logger so support can see it in /settings/debug - The interrupted-error copy is i18n-backed (imports.errors.interrupted) * fix(jobs): round-2 review feedback on the reapers - Import.clean excludes session-owned chunks (import_session_id: nil) — SyncCleanerJob runs it before ImportSession.clean, so it could finalize or fail a SureImport chunk outside the session flow that owns its lifecycle. Regression test added (CodeRabbit major) - family.sync_later moved outside the row-lock transaction in both reap paths — Rails doesn't defer enqueues to after-commit by default, so Sidekiq could pick the job up before the status write was visible - Per-record rescue in Import.clean / PdfImport.clean so one bad record doesn't abort the rest of the hourly sweep - ImportSession.clean uses the importing? enum predicate; reconcile_committed_chunks! iterates with each (default-ordered association made find_each warn, and chunk counts are tiny) * fix(jobs): address round-3 reaper review (isolation + commit signal) Per-record rescue isolation for the three sweeps that lacked it — FamilyExport.clean, ImportSession.clean, and SyncCleanerJob's activity-flag loop — mirroring Import.clean/PdfImport.clean. One bad record (validation error, DB blip) no longer aborts the rest of that sweep for the hour; the activity-flag guard is per-record so a failure also stops skipping the models that follow. data_committed? now covers Category/Rule/Merchant imports, whose records hang off the family rather than the import (no entries or accounts), via the new Import#committed_by_named_records? helper. A committed one is reaped to complete instead of a retryable failed; nameless RuleImport rows carry no stable key, so a nameless-only file has no commit signal and stays retryable. * fix(schema): drop duplicate enable_banking_accounts columns from merge The merge of main into this branch re-appended product, credit_limit, and identification_hashes after updated_at, so schema.rb declared each twice and db:schema:load raised "you can't define an already defined column 'product'". Removed the duplicate declarations (kept the new treat_balance_as_available_credit column); the test database loads again.
380 lines
11 KiB
Ruby
380 lines
11 KiB
Ruby
class RuleImport < Import
|
|
def import!
|
|
transaction do
|
|
rows.each do |row|
|
|
create_or_update_rule_from_row(row)
|
|
end
|
|
end
|
|
end
|
|
|
|
# Rules hang off the family, not the import — see
|
|
# Import#committed_by_named_records?. Nameless rows build unconditionally and
|
|
# carry no stable key, so a file of only nameless rows has no commit signal.
|
|
def data_committed?
|
|
committed_by_named_records?(family.rules)
|
|
end
|
|
|
|
def column_keys
|
|
%i[name resource_type active effective_date conditions actions]
|
|
end
|
|
|
|
def required_column_keys
|
|
%i[resource_type conditions actions]
|
|
end
|
|
|
|
def mapping_steps
|
|
[]
|
|
end
|
|
|
|
def dry_run
|
|
{ rules: rows_count }
|
|
end
|
|
|
|
def csv_template
|
|
csv_string = CSV.generate do |csv|
|
|
csv << %w[name resource_type* active effective_date conditions* actions*]
|
|
|
|
csv << [
|
|
"Categorize groceries",
|
|
"transaction",
|
|
"true",
|
|
"2024-01-01",
|
|
'[{"condition_type":"transaction_name","operator":"like","value":"grocery"}]',
|
|
'[{"action_type":"set_transaction_category","value":"Groceries"}]'
|
|
]
|
|
|
|
csv << [
|
|
"Auto-categorize transactions",
|
|
"transaction",
|
|
"true",
|
|
"",
|
|
'[{"condition_type":"transaction_name","operator":"like","value":"amazon"}]',
|
|
'[{"action_type":"auto_categorize"}]'
|
|
]
|
|
end
|
|
|
|
CSV.parse(csv_string, headers: true)
|
|
end
|
|
|
|
def generate_rows_from_csv
|
|
rows.destroy_all
|
|
|
|
csv_rows.each.with_index(1) do |row, index|
|
|
normalized_row = normalize_rule_row(row)
|
|
|
|
rows.create!(
|
|
source_row_number: index,
|
|
name: normalized_row[:name].to_s.strip,
|
|
resource_type: normalized_row[:resource_type].to_s.strip,
|
|
active: parse_boolean(normalized_row[:active]),
|
|
effective_date: normalized_row[:effective_date].to_s.strip,
|
|
conditions: normalized_row[:conditions].to_s.strip,
|
|
actions: normalized_row[:actions].to_s.strip,
|
|
currency: default_currency
|
|
)
|
|
end
|
|
end
|
|
|
|
def parsed_csv
|
|
@parsed_csv ||= Import.parse_csv_str(raw_file_str, col_sep: col_sep)
|
|
end
|
|
|
|
private
|
|
|
|
def normalize_rule_row(row)
|
|
fields = row.fields
|
|
name, resource_type, active, effective_date = fields[0..3]
|
|
conditions, actions = extract_conditions_and_actions(fields[4..])
|
|
|
|
{
|
|
name: row["name"].presence || name,
|
|
resource_type: row["resource_type"].presence || resource_type,
|
|
active: row["active"].presence || active,
|
|
effective_date: row["effective_date"].presence || effective_date,
|
|
conditions: conditions,
|
|
actions: actions
|
|
}
|
|
end
|
|
|
|
def extract_conditions_and_actions(fragments)
|
|
pieces = Array(fragments).compact
|
|
return [ "", "" ] if pieces.empty?
|
|
|
|
combined = pieces.join(col_sep)
|
|
|
|
# If the CSV was split incorrectly because of unescaped quotes in the JSON
|
|
# payload, re-assemble the last two logical columns by splitting on the
|
|
# boundary between the two JSON arrays: ...]","[...
|
|
parts = combined.split(/(?<=\])"\s*,\s*"(?=\[)/, 2)
|
|
parts = [ pieces[0], pieces[1] ] if parts.length < 2
|
|
|
|
parts.map do |part|
|
|
next "" unless part
|
|
|
|
# Remove any stray leading/trailing quotes left from CSV parsing
|
|
part.to_s.strip.gsub(/\A"+|"+\z/, "")
|
|
end
|
|
end
|
|
|
|
def create_or_update_rule_from_row(row)
|
|
rule_name = row.name.to_s.strip.presence
|
|
resource_type = row.resource_type.to_s.strip
|
|
|
|
# Validate resource type
|
|
unless resource_type == "transaction"
|
|
errors.add(:base, :unsupported_resource_type, resource_type: resource_type)
|
|
raise ActiveRecord::RecordInvalid.new(self)
|
|
end
|
|
|
|
# Parse conditions and actions from JSON
|
|
begin
|
|
conditions_data = parse_json_safely(row.conditions, "conditions")
|
|
actions_data = parse_json_safely(row.actions, "actions")
|
|
rescue JSON::ParserError => e
|
|
errors.add(:base, :invalid_json, message: e.message)
|
|
raise ActiveRecord::RecordInvalid.new(self)
|
|
end
|
|
|
|
# Validate we have at least one action
|
|
if actions_data.empty?
|
|
errors.add(:base, :min_actions)
|
|
raise ActiveRecord::RecordInvalid.new(self)
|
|
end
|
|
|
|
# Find or create rule
|
|
rule = if rule_name.present?
|
|
family.rules.find_or_initialize_by(name: rule_name, resource_type: resource_type)
|
|
else
|
|
family.rules.build(resource_type: resource_type)
|
|
end
|
|
|
|
rule.active = row.active || false
|
|
rule.effective_date = parse_date(row.effective_date)
|
|
|
|
# Clear existing conditions and actions
|
|
rule.conditions.destroy_all
|
|
rule.actions.destroy_all
|
|
|
|
# Create conditions
|
|
conditions_data.each do |condition_data|
|
|
build_condition(rule, condition_data)
|
|
end
|
|
|
|
# Create actions
|
|
actions_data.each do |action_data|
|
|
build_action(rule, action_data)
|
|
end
|
|
|
|
rule.save!
|
|
end
|
|
|
|
def build_condition(rule, condition_data, parent: nil)
|
|
condition_type = condition_data["condition_type"]
|
|
operator = condition_data["operator"]
|
|
value = resolve_import_condition_value(condition_data)
|
|
|
|
condition = if parent
|
|
parent.sub_conditions.build(
|
|
condition_type: condition_type,
|
|
operator: operator,
|
|
value: value
|
|
)
|
|
else
|
|
rule.conditions.build(
|
|
condition_type: condition_type,
|
|
operator: operator,
|
|
value: value
|
|
)
|
|
end
|
|
|
|
# Handle compound conditions with sub_conditions
|
|
if condition_data["sub_conditions"].present?
|
|
condition_data["sub_conditions"].each do |sub_condition_data|
|
|
build_condition(rule, sub_condition_data, parent: condition)
|
|
end
|
|
end
|
|
|
|
condition
|
|
end
|
|
|
|
def build_action(rule, action_data)
|
|
action_type = action_data["action_type"]
|
|
value = resolve_import_action_value(action_data)
|
|
|
|
rule.actions.build(
|
|
action_type: action_type,
|
|
value: value
|
|
)
|
|
end
|
|
|
|
def resolve_import_condition_value(condition_data)
|
|
condition_type = condition_data["condition_type"]
|
|
value = condition_data["value"]
|
|
|
|
return value unless value.present?
|
|
|
|
# Map category names to UUIDs
|
|
if condition_type == "transaction_category"
|
|
category = family.categories.find_by(name: value)
|
|
unless category
|
|
category = family.categories.create!(
|
|
name: value,
|
|
color: Category::UNCATEGORIZED_COLOR,
|
|
lucide_icon: "shapes"
|
|
)
|
|
end
|
|
return category.id
|
|
end
|
|
|
|
# Map merchant names to UUIDs
|
|
if condition_type == "transaction_merchant"
|
|
merchant = family.merchants.find_by(name: value)
|
|
unless merchant
|
|
merchant = family.merchants.create!(name: value)
|
|
end
|
|
return merchant.id
|
|
end
|
|
|
|
value
|
|
end
|
|
|
|
def resolve_import_action_value(action_data)
|
|
action_type = action_data["action_type"]
|
|
value = action_data["value"]
|
|
|
|
return value unless value.present?
|
|
|
|
# Map category names to UUIDs
|
|
if action_type == "set_transaction_category"
|
|
category = family.categories.find_by(name: value)
|
|
# Create category if it doesn't exist
|
|
unless category
|
|
category = family.categories.create!(
|
|
name: value,
|
|
color: Category::UNCATEGORIZED_COLOR,
|
|
lucide_icon: "shapes"
|
|
)
|
|
end
|
|
return category.id
|
|
end
|
|
|
|
# Map merchant names to UUIDs
|
|
if action_type == "set_transaction_merchant"
|
|
merchant = family.merchants.find_by(name: value)
|
|
# Create merchant if it doesn't exist
|
|
unless merchant
|
|
merchant = family.merchants.create!(name: value)
|
|
end
|
|
return merchant.id
|
|
end
|
|
|
|
# Map tag names to UUIDs
|
|
if action_type == "set_transaction_tags"
|
|
tag = family.tags.find_by(name: value)
|
|
# Create tag if it doesn't exist
|
|
unless tag
|
|
tag = family.tags.create!(name: value)
|
|
end
|
|
return tag.id
|
|
end
|
|
|
|
value
|
|
end
|
|
|
|
def parse_boolean(value)
|
|
return true if value.to_s.downcase.in?(%w[true 1 yes y])
|
|
return false if value.to_s.downcase.in?(%w[false 0 no n])
|
|
false
|
|
end
|
|
|
|
def parse_date(value)
|
|
return nil if value.blank?
|
|
Date.parse(value.to_s)
|
|
rescue ArgumentError
|
|
nil
|
|
end
|
|
|
|
def parse_json_safely(json_string, field_name)
|
|
return [] if json_string.blank?
|
|
|
|
cleaned = json_string.to_s.strip
|
|
|
|
# Most API-created rows already store valid JSON. Parse them as-is before
|
|
# falling back to the legacy cleanup path for older malformed payloads.
|
|
parse_json_payload(cleaned, normalize_legacy_strings: false)
|
|
rescue JSON::ParserError
|
|
# Clean up the JSON string - remove extra escaping that might come from CSV parsing
|
|
|
|
# Remove surrounding quotes if present (both single and double)
|
|
cleaned = cleaned.gsub(/\A["']+|["']+\z/, "")
|
|
|
|
# Handle multiple levels of escaping iteratively
|
|
# Keep unescaping until no more changes occur
|
|
loop do
|
|
previous = cleaned.dup
|
|
|
|
# Unescape quotes - handle patterns like \" or \\\" or \\\\\" etc.
|
|
# Replace any number of backslashes followed by a quote with just a quote
|
|
cleaned = cleaned.gsub(/\\+"/, '"')
|
|
cleaned = cleaned.gsub(/\\+'/, "'")
|
|
|
|
# Unescape backslashes (\\\\ becomes \)
|
|
cleaned = cleaned.gsub(/\\\\/, "\\")
|
|
|
|
break if cleaned == previous
|
|
end
|
|
|
|
# Handle unicode escapes like \u003e (but only if not over-escaped)
|
|
# Try to find and decode unicode escapes
|
|
cleaned = cleaned.gsub(/\\u([0-9a-fA-F]{4})/i) do |match|
|
|
code_point = $1.to_i(16)
|
|
[ code_point ].pack("U")
|
|
rescue
|
|
match # If decoding fails, keep the original
|
|
end
|
|
|
|
# Try parsing
|
|
parse_json_payload(cleaned, normalize_legacy_strings: true)
|
|
rescue JSON::ParserError => e
|
|
raise JSON::ParserError.new("Invalid JSON in #{field_name}: #{e.message}. Raw value: #{json_string.inspect}")
|
|
end
|
|
|
|
def parse_json_payload(payload, normalize_legacy_strings:)
|
|
parsed = JSON.parse(payload)
|
|
parsed = JSON.parse(parsed) if wrapped_json_payload?(parsed)
|
|
|
|
normalize_json_values(parsed, normalize_legacy_strings:)
|
|
end
|
|
|
|
def wrapped_json_payload?(value)
|
|
return false unless value.is_a?(String)
|
|
|
|
stripped_value = value.strip
|
|
stripped_value.start_with?("[", "{")
|
|
end
|
|
|
|
def normalize_json_values(value, normalize_legacy_strings:)
|
|
case value
|
|
when Array
|
|
value.map { |item| normalize_json_values(item, normalize_legacy_strings:) }
|
|
when Hash
|
|
value.transform_values { |item| normalize_json_values(item, normalize_legacy_strings:) }
|
|
when String
|
|
normalized = value
|
|
.gsub(/\\u([0-9a-fA-F]{4})/i) { [ $1.to_i(16) ].pack("U") }
|
|
.gsub('\\"', '"')
|
|
|
|
if normalize_legacy_strings
|
|
normalized = normalized
|
|
.gsub("\\n", "\n")
|
|
.gsub("\\r", "\r")
|
|
.gsub("\\t", "\t")
|
|
end
|
|
|
|
normalized
|
|
else
|
|
value
|
|
end
|
|
end
|
|
end
|