Harden SimpleFIN sync: protect user data, fix stuck syncs, optimize API calls (#671)

* Implement entry protection flags for sync overwrites

- Added `user_modified` and `import_locked` flags to `entries` table to prevent provider sync from overwriting user-edited and imported data.
- Introduced backfill migration to mark existing entries based on conditions.
- Enhanced sync and processing logic to respect protection flags, track skipped entries, and log detailed stats.
- Updated UI to display skipped/protected entries and reasons in sync summaries.

* Localize error details summary text and adjust `sync_account_later` method placement

* Restored schema.rb

---------

Co-authored-by: luckyPipewrench <luckypipewrench@proton.me>
This commit is contained in:
LPW
2026-01-16 06:34:06 -05:00
committed by GitHub
parent 9b1188eab4
commit c391ba2b23
20 changed files with 344 additions and 37 deletions

View File

@@ -0,0 +1,17 @@
class AddEntryProtectionFlags < ActiveRecord::Migration[7.2]
def change
# user_modified: Set when user manually edits any field on an entry.
# Prevents provider sync from overwriting user's intentional changes.
# Does NOT prevent user from editing - only protects from automated overwrites.
add_column :entries, :user_modified, :boolean, default: false, null: false
# import_locked: Set when entry is created via CSV/manual import.
# Prevents provider sync from overwriting imported data.
# Does NOT prevent user from editing - only protects from automated overwrites.
add_column :entries, :import_locked, :boolean, default: false, null: false
# Partial indexes for efficient queries when filtering protected entries
add_index :entries, :user_modified, where: "user_modified = true", name: "index_entries_on_user_modified_true"
add_index :entries, :import_locked, where: "import_locked = true", name: "index_entries_on_import_locked_true"
end
end

View File

@@ -0,0 +1,34 @@
class BackfillEntryProtectionFlags < ActiveRecord::Migration[7.2]
disable_ddl_transaction!
def up
# Backfill import_locked for entries that came from CSV/manual imports
# These entries have import_id set but typically no external_id or source
say_with_time "Marking CSV-imported entries as import_locked" do
execute <<-SQL.squish
UPDATE entries
SET import_locked = true
WHERE import_id IS NOT NULL
AND import_locked = false
SQL
end
# Backfill user_modified for entries where user has manually edited fields
# These entries have non-empty locked_attributes (set when user edits)
say_with_time "Marking user-edited entries as user_modified" do
execute <<-SQL.squish
UPDATE entries
SET user_modified = true
WHERE locked_attributes != '{}'::jsonb
AND locked_attributes IS NOT NULL
AND user_modified = false
SQL
end
end
def down
# Reversible but generally not needed
execute "UPDATE entries SET import_locked = false WHERE import_locked = true"
execute "UPDATE entries SET user_modified = false WHERE user_modified = true"
end
end