mirror of
https://github.com/we-promise/sure.git
synced 2026-09-09 08:34:26 +00:00
3c14646d20ea76d85bc2d564dfa2a3ed0ea28e8e
66
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 GFRandGerald
|
fa8769f792 |
fix: support DD/MM/YY date format for CSV imports (#3110)
Adds "DD/MM/YY" as a CSV-only date format for transaction and account balance imports, addressing #1530. Kept out of Family::DATE_FORMATS (the global date preference) since a prior PR (#531) adding it there was rejected by a maintainer: 2-digit years are ambiguous (Ruby's %y assumes 1969-2068) and could silently misparse historical or future-dated transactions. Restricting it to Import::CSV_ONLY_DATE_FORMATS keeps it available where the user can see and verify a parsed preview against their own CSV data, per the maintainer's suggested approach in that PR's discussion. Co-authored-by: Gerald <248542187+gfr-free@users.noreply.github.com> |
||
|
Juan José MataandClaude Opus 5
|
8ca65ffd27 |
Reconcile PDF statement imports against transactions that already exist (#3105)
* Reset to main, keeping only the account matcher improvements Backs out the LLM-driven reconciliation work (PR #1382's approach and the two commits hardening it). That approach compares whole-statement aggregates, which is all-or-nothing: when 18 of 20 transactions have already synced the totals disagree, the import proceeds, and 18 duplicates are created. Reconciliation is a row-level problem and belongs in the import path, where TransactionImport already solves it via Account::ProviderImportAdapter. Kept from that work, because it stands on its own: - AccountMatcher gains a hint-based class-level entry point so callers without an AccountStatement row can score against the same rules. The instance path used by AccountStatement#assign_account_match is unchanged. - It also refuses to guess between equally-confident candidates rather than letting max_by take whichever the scan reached first. Account names are not unique within a family, so that tie was reachable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvWSJotDiqP6qSR7qEiC8Q * Reconcile PDF statement imports against transactions that already exist Reconciliation is a row-level problem. Comparing whole-statement totals is all-or-nothing: when 18 of 20 transactions have already synced the totals disagree, the import proceeds, and 18 duplicates are created. This matches each extracted transaction against what the account already holds, so only genuinely new transactions are ever offered for import and the rest are marked reconciled. TransactionImport#import! already does this for CSV via Account::ProviderImportAdapter. PdfImport#import! called it zero times and built one Transaction per row unconditionally -- the only import path in the repo with no duplicate protection. Reconciliation state follows Quicken's uncleared / cleared / reconciled, but only the last state is stored: - "Cleared" means the institution acknowledged the transaction, which is exactly what entries.source and entries.external_id already record. It stays accurate on its own, because the adapter stamps both onto a manual entry when a provider transaction claims it, so a hand-entered transaction that later appears in a download becomes cleared with no extra bookkeeping. Deriving it also keeps it non-editable, which is right: it is a fact about provenance, not an opinion. - "Reconciled" means a statement was matched against the transaction. Nothing can derive that, so entries gains reconciled_at and reconciled_by_statement_id. It is a judgement, so it can be set and unset, and it survives the statement being deleted (the FK nullifies, the timestamp stays). Matching: - find_duplicate_transaction grows include_provider_entries, which is what makes this work for Provider-backed accounts -- the existing where(external_id: nil) filter hid synced transactions from every import path, so this gap affected CSV imports equally. Default stays false: provider sync must not claim another provider's entry. - It also grows date_window, because a statement's posting date routinely differs by a day or two from the date a provider recorded. Nearest date wins. - Name is deliberately not matched on: statement descriptions and provider names for the same transaction rarely agree. The adapter makes the same choice for sync. - Candidates are built as real Import::Row objects so matching uses the same signed_amount and date_iso the import itself would write, rather than a second interpretation of signage that could drift. - Matching is per-account, so with no account assigned every row is offered and re-judged on assignment; reassigning also releases the previous account's reconciliations. - A row whose date or amount will not parse is offered for import rather than dropped, so nothing disappears silently. - import! re-checks at publish, since a sync can land between review and publish, and new transactions are born reconciled: the statement is their evidence. Provider-backed accounts are now offered in the import target picker. The manual-only restriction existed because importing into a synced account would duplicate what sync brought in, which is precisely what this removes. Also fixes a bug this uncovered on main: extract_transactions stored the extractor's symbol-keyed hash, while every reader digs with strings. jsonb keeps the hash as assigned until reload and ProcessPdfJob never reloads, so has_extracted_transactions? was false and PDF imports generated zero rows. The existing tests missed it -- one uses a YAML fixture, the other stubs the extractor with string keys. Supersedes #1382. Refs #1379. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvWSJotDiqP6qSR7qEiC8Q * Fix three review findings in statement reconciliation All three confirmed against the code before fixing. Publish-time recheck consumed the same entry twice. import! started its exclusion list empty, so a statement carrying two same-amount transactions against an account holding only one would re-match the surviving row against the entry row generation had already consumed -- silently dropping a genuinely new transaction instead of creating it. Seed the exclusions with the entries this statement already reconciled; newly synced entries are still caught, since only already-reconciled ones are excluded. A regeneration that emptied the row set left the import stranded. In the normal upload flow the account is assigned after extraction, so assign_account! regenerates -- and if everything then reconciled, rows_count went to zero while status stayed pending. _pdf_import.html.erb renders pending-with-no-rows as the processing screen, and process_with_ai_later cannot restart because ai_processed? is already true, so the import was stuck with no way forward. Status now follows the same rule ProcessPdfJob applies after initial processing, in both directions: no rows completes it, rows returning sends it back to pending. Guarded by data_committed? so a published import is never reopened. Unevaluatable rows went only to the Rails log. AGENTS.md asks for DebugLogEntry.capture on recoverable import failures so they surface in /settings/debug with structured context. Capture family, account, import, statement, row number and the raw date/amount that would not parse. Adds regression coverage for each. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvWSJotDiqP6qSR7qEiC8Q * Fix row regeneration collision and over-broad reconciliation release test_unit caught one error in 6462 tests, and it was real. Row regeneration collided on the second call. insert_all! bypasses ActiveRecord, so the rows association is never populated with what it wrote. Calling generate_rows_from_extracted_data twice on the same in-memory record -- which assign_account! now does after ProcessPdfJob has already generated once -- made rows.destroy_all clear a stale empty collection, delete nothing, and then collide on (import_id, source_row_number). Reload before destroying, and reset the association after inserting so sync_mappings and the view read what was actually written. Releasing reconciliations was scoped to the statement, not the account. A statement is evidence for exactly one account at a time but can back more than one import, so reassigning an account cleared reconciliations another account still relied on. Scoped to the account being moved away from; a blank scope releases nothing, which is correct because nothing is reconciled while no account is assigned. The second finding was raised by CodeRabbit. Its other flagged risk -- entries being marked reconciled before the import is published -- is deliberate and stays: the statement is the evidence, reconciling is reversible via unmark_reconciled!, and deferring it to publish would leave a fully reconciled statement with nothing to publish and therefore nothing ever marked. Adds regression coverage for both. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvWSJotDiqP6qSR7qEiC8Q * Address review nitpicks: model validation, scope placement, mock style, lock-safe DDL All four checked against the repo's own conventions before applying. Mirror the check constraint as a model validation. Assigning reconciled_by_statement without reconciled_at raised StatementInvalid rather than a validation error. CLAUDE.md Convention 5 asks for exactly this pairing -- the constraint in the database, an ActiveRecord validation for form-friendly errors. Stop hijacking the pending-scope comment. The reconciliation scopes were inserted directly under "Pending transaction scopes", so that header read as documentation for them and the provider note below read as a continuation of reconciled_by. Given the reconciliation scopes their own header. Use OpenStruct for the provider response double, per "Always prefer OpenStruct when creating mock instances". Verified OpenStruct.new(success?: true) responds to success?, and ostruct is already a dependency used elsewhere in test/. Make the migration lock-safe. entries is the largest table in the app: both indexes now build concurrently, and the check constraint is added unvalidated then validated separately so VALIDATE takes only SHARE UPDATE EXCLUSIVE instead of holding ACCESS EXCLUSIVE for a full scan. This follows existing practice -- 13 migrations already use disable_ddl_transaction! and 11 use algorithm: :concurrently, with add_offline_reason_to_securities combining add_column and a concurrent index in one migration exactly like this. The suggested follow-up migration for validation was not needed: validating in the same non- transactional migration gets the same lock behavior without a second file, and the repo has no validate: false precedent in 400 migrations. schema.rb is unchanged: a validated constraint and a concurrently-built index dump identically. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KvWSJotDiqP6qSR7qEiC8Q * Add the two review-requested regression tests Covers the invalid-amount half of "malformed rows are offered, not dropped" -- the existing test only exercised an unparseable date. Asserts the raw value is stored verbatim rather than coerced to 0, so the review step shows the user what the statement actually said. Also covers the Entry validation added in |
||
|
Guillem Arias Fauste
|
9d4b4bc27c |
feat(jobs): reap imports and exports stuck by lost background jobs (#2681)
* feat(jobs): reap imports and exports stuck by lost background jobs A hard-killed worker (OOM, SIGKILL during deploy) loses its in-flight Sidekiq job permanently. Sync is the only model with a stale sweep; everything else wedges in a non-terminal status forever: - Import stuck "importing"/"reverting" (e.g. #2274) - ImportSession stuck "importing" — unrecoverable, publish_later refuses to re-publish while importing - FamilyExport stuck "pending"/"processing" — exports index polls every 3s indefinitely - PdfImport's AI-processing claim held forever (ProcessPdfJob's reclaim only runs if the job is redelivered) - Provider activities_fetch_pending flags stranded when a self-rescheduling fetch chain loses a link SyncCleanerJob (existing hourly cron) now sweeps all of them, each isolated so one failure doesn't block the rest, and records every reaped record as a DebugLogEntry with the family attached. Idempotency guards so a stray or redelivered job cannot corrupt a record the reaper (or a user retry) has since moved on: - Import#publish skips complete/reverting/revert_failed imports — a redelivered ImportJob after completion double-applies data for import types without row dedup. Failed imports deliberately stay publishable: their transaction rolled back, so a re-run is a retry. - FamilyDataExportJob refuses terminal exports but still allows processing ones through, since graceful-shutdown redelivery is what completes them. Reaper thresholds (6h imports, 2h exports) key off updated_at and dwarf legitimate runtimes, so live jobs are not swept in practice. * fix(jobs): make the reapers race-safe and commit-aware Review feedback on #2681 (jjmata, Codex, CodeRabbit): - Every sweep now mutates under record.with_lock with a staleness re-check, mirroring the guard Sync#perform gained in #2680 — a job finishing between the sweep query and the write can no longer be clobbered mid-flight - Import.clean distinguishes which side of import!'s single transaction the worker died on: rows attached means the data committed and only the status write was lost, so the record is finalized as complete (marking it failed invited a re-publish that double-imports types without row dedup, e.g. TradeImport); no rows means a clean rollback and the failed/try-again path stays - PdfImport.clean applies the same split: no rows → the AI-extraction claim died, reclaim to pending; rows → the publish died post-commit, finalize complete instead of letting the same extracted rows be published twice. Stuck PdfImport reverts (previously unswept by either clean) now go to revert_failed like every other import - ImportSession.clean reconciles chunks whose import! committed but never got the complete-status write before failing the session, so re-publish skips them instead of duplicating their rows via SureImport's split path - Import#publish redelivery skip is captured via DebugLogEntry instead of Rails.logger so support can see it in /settings/debug - The interrupted-error copy is i18n-backed (imports.errors.interrupted) * fix(jobs): round-2 review feedback on the reapers - Import.clean excludes session-owned chunks (import_session_id: nil) — SyncCleanerJob runs it before ImportSession.clean, so it could finalize or fail a SureImport chunk outside the session flow that owns its lifecycle. Regression test added (CodeRabbit major) - family.sync_later moved outside the row-lock transaction in both reap paths — Rails doesn't defer enqueues to after-commit by default, so Sidekiq could pick the job up before the status write was visible - Per-record rescue in Import.clean / PdfImport.clean so one bad record doesn't abort the rest of the hourly sweep - ImportSession.clean uses the importing? enum predicate; reconcile_committed_chunks! iterates with each (default-ordered association made find_each warn, and chunk counts are tiny) * fix(jobs): address round-3 reaper review (isolation + commit signal) Per-record rescue isolation for the three sweeps that lacked it — FamilyExport.clean, ImportSession.clean, and SyncCleanerJob's activity-flag loop — mirroring Import.clean/PdfImport.clean. One bad record (validation error, DB blip) no longer aborts the rest of that sweep for the hour; the activity-flag guard is per-record so a failure also stops skipping the models that follow. data_committed? now covers Category/Rule/Merchant imports, whose records hang off the family rather than the import (no entries or accounts), via the new Import#committed_by_named_records? helper. A committed one is reaped to complete instead of a retryable failed; nameless RuleImport rows carry no stable key, so a nameless-only file has no commit signal and stays retryable. * fix(schema): drop duplicate enable_banking_accounts columns from merge The merge of main into this branch re-appended product, credit_limit, and identification_hashes after updated_at, so schema.rb declared each twice and db:schema:load raised "you can't define an already defined column 'product'". Removed the duplicate declarations (kept the new treat_balance_as_available_credit column); the test database loads again. |
||
|
super
|
335d7da162 |
fix(import): strip non-breaking-space thousands separators in sanitize_number (#2538)
* fix(import): strip non-breaking-space thousands separators in sanitize_number
The French/Scandinavian number format ("1 234,56") only removed ASCII
whitespace via \s, which does not match a non-breaking space (U+00A0) or
a narrow no-break space (U+202F). Real-world European CSV exports use
those Unicode spaces as the thousands separator, so such amounts failed
the final numeric guard and were silently coerced to "", losing the
value on import. The branch also skipped the non-numeric junk stripping
that the other formats apply.
Strip everything that isn't a digit, the decimal separator, or a minus
sign in this branch, matching the else branch's behavior.
Fixes #2537
* fix(import): strip only whitespace for space-delimited number format
Addresses review feedback: the previous filter removed any non-numeric
character, so a misconfigured/mixed row using US separators ("1,234.56")
under the "1 234,56" format had its period dropped and its comma turned
into a decimal point, silently importing 1.23456 (off by ~1000x) instead
of being rejected.
Strip only whitespace via \p{Space}, which matches ASCII, non-breaking
(U+00A0), and narrow no-break (U+202F) spaces. Unexpected punctuation is
left in place so the existing numeric guard still rejects malformed
values. Add a regression test for the mixed-punctuation case and a
docstring for sanitize_number.
* fix(import): strip leading/trailing currency junk for space-delimited numbers
Follow-up to review on #2538: the whitespace-only strip did not cover
issue #2537's currency-suffix row ("1 234,56 kr") or a leading currency
symbol ("€1 234,56"), which still failed the numeric guard and imported
blank.
Strip non-numeric junk only at the leading/trailing edges (currency
symbols/codes), leaving interior characters untouched. This fully closes
the #2537 repro table while preserving the earlier fix for the mixed
US-format hazard: "1,234.56" keeps its interior period and is still
rejected rather than parsed as 1.23456. Digits, the decimal separator,
and a leading/trailing minus are preserved so signed values and the
guard still behave correctly.
Add regression tests for the currency prefix/suffix cases.
* test(import): add U+202F narrow no-break-space regression case
Complements the existing U+00A0 case so both Unicode thousands-separator
variants named in the fix are covered.
|
||
|
Guillem Arias Fauste
|
5105752bbd |
feat(sync): family-facing cancellation for syncs, imports, and exports (#2685)
* feat(sync): family-facing cancellation for syncs, imports, and exports Users had no way to stop or recover any background operation: a mistaken "Sync all" runs to completion, and an import or export whose job died (hard worker kills lose in-flight Sidekiq jobs) wedges with a spinner forever. Sync cancellation (cooperative — nothing is ever killed): - New syncs.cancel_requested_at column. Only the cancelled sync carries the flag: pending descendants are marked stale immediately (their queued jobs no-op via the existing may_start? guard), while descendants whose jobs are already executing finish their work honestly. - Family::Syncer stops fanning out child syncs once the flag is set (fresh read per iteration — the flag comes from the web process). - Finalization resolves a cancel-requested sync to stale instead of completed, which also skips post-sync (transfer matching, rules, broadcasts) via the existing stale gate. - The `visible` scope excludes cancel-requested syncs, so spinners clear immediately and — fixing a latent bug this feature would have amplified — sync_later no longer piggybacks a new sync request onto a dying sync it would silently swallow. - Cancel button appears next to "Sync all" on the accounts page while a family sync is visible. SyncsController#cancel scopes through Sync.for_family with resource_owner, so cross-family ids 404 and account-level syncs respect per-user account access. Stuck import/export self-service: - Import#force_fail! / FamilyExport#force_fail!: allowed only once the record has been idle past PRESUMED_LOST_AFTER (1 hour — dwarfs any legitimate run), and applied inside with_lock with a status re-check, so a job finishing between page render and button click wins. Imports fail into the existing retry path (reverting -> revert_failed keeps the revert retryable); PdfImports release their processing claim back to pending; exports fail so a new one can be created. - "Mark as failed" buttons appear on the imports/exports index rows only when a record is presumed lost, behind the pages' existing permission gates (statement-import permission for imports, admin for exports). * fix(sync-cancel): cascade pending cancels, guard late finalizers, scope provider syncs Review feedback on #2685 (CodeRabbit, Codex): - request_cancel! now cascades finalization for pending syncs too: a pending child resolved to stale never runs its job, so nothing else would ever call finalize_if_all_children_finalized — its waiting parent hung in syncing until the 24h sweep (CodeRabbit critical) - SimplefinItem::Syncer#mark_completed re-reads the sync under a row lock and skips finalization once cancellation was requested or the row went terminal — its in-memory copy predates the cancel, and the unguarded complete! (plus the raw status fallback) resurrected a cancelled sync and re-ran post-sync (Codex) - Cancelling provider-item syncs now requires admin: for_family's resource_owner only scopes the Account branch, so a restricted member could cancel admin-managed provider syncs spanning accounts they cannot see. Family- and account-level syncs stay member-cancellable, matching the buttons the UI shows (Codex) - Lost-import error copy moved behind i18n (imports.errors.presumed_lost), resolved at call time (CodeRabbit) - Tests: pending-child cancel finalizes the parent; late provider complete! cannot resurrect a cancelled sync; provider-sync cancellation is admin-only * fix(sync-cancel): capture the skipped-finalization case via DebugLogEntry CodeRabbit round-2: the mark_completed skip (cancelled/terminal sync) is support-relevant — record it in the super-admin debug UI with the family and provider attached instead of a raw Rails.logger line. category: provider_sync, matching the other provider syncers. |
||
|
Augusto Xavier
|
844cba2fd3 |
fix(imports): normalize CSV upload encoding when validations are skipped (#2299)
Import::UploadsController#update persists CSV uploads with save!(validate: false), which skips the before_validation :ensure_utf8_encoding callback. Non-UTF-8 files (e.g. ISO-8859-1 / Windows-1252 exports from Brazilian banks) were written to the text column as-is and rejected by Postgres with PG::CharacterNotInRepertoire, surfacing as a 500 during upload. Also register ensure_utf8_encoding on before_save so the existing normalization (rchardet detection + Latin-1/Windows fallback) still runs when validations are skipped. The callback is idempotent and no-ops on valid UTF-8, so the validated path is unchanged. Fixes #2294 |
||
|
ghost
|
6ea931f3fe |
feat(imports): add YNAB CSV import (#2361)
Adds YnabImport (mirroring ActualImport) for YNAB "Export budget" register CSVs: - Amount — combines the split Outflow/Inflow columns into a single signed amount (inflow - |outflow|), stripping currency symbols and thousands separators. A single signed Amount column takes precedence when present. - Category — resolves across export shapes: the combined "Category Group/Category" column, the split "Category Group" + "Category", or legacy YNAB 4 "Master Category" + "Sub Category". - Names — falls back from a blank Payee to the Memo, then the default row name. - Validation — requires at least one amount source (Outflow/Inflow or Amount); a file exposing none leaves rows un-clean instead of importing zero-dollar entries. Enables the previously-disabled YNAB option on the imports screen (using the YNAB logo, like Mint) with its configuration partial, and removes the now-dead imports.new.coming_soon locale key. Documents the type in the API import-type enums (rswag request spec + swagger_helper + generated openapi.yaml). Closes #1255. |
||
|
Blaž Dular
|
94422955f8 |
feat(merchants): add raw data import (csv) for merchants (#1992)
* feat(merchants): add csv import endpoint for merchants * docs: update endpoint docs * fix(merchant): recommended ai fixes |
||
|
ghost
|
6e04c6927d |
feat(imports): add SureImport session batches (#1785)
* feat(imports): add SureImport session batches Add first-class SureImport sessions for ordered multi-file NDJSON imports. Persist source mappings across chunks, make session/chunk processing idempotent, expose progress readback, and keep existing single-file import behavior compatible. Includes the devcontainer libvips runtime dependency needed by ActiveStorage variant tests. Addresses #1610. Related to #1458. * fix(imports): avoid scanner-like API key test data * test(imports): assert skipped balances are not persisted * fix(imports): harden session publish retries Validate expected import chunk sequences exactly before publish, and restore session state with error details when enqueueing the publish job fails. * fix(imports): close session retry edge cases Backfill expected chunk counts after client-session insert races and enqueue import-session jobs after the status transition commits. Persist a safe enqueue failure body so API readback does not expose raw queue errors. * fix(imports): address session publish review gaps Remove dead transaction external-id assignment, harden session publish retry/sync behavior, align session chunk status docs, and add regression coverage for partial retries and safe enqueue error readback. * fix(imports): include sessions in family reset Clear import sessions through the family reset job so chunk imports and source mappings do not survive a reset. Expose import session and source mapping counts in the reset status response and regenerated OpenAPI schema so polling reflects the full reset surface. * test(imports): cover split import mapping invariants * test(imports): cover session verification invariants * fix(imports): scope SureImport session reimports * Tighten SureImport session batching * fix(imports): export rule source ids for sessions * test(imports): stabilize rule id export assertion * test(imports): restore reset status session fixture |
||
|
ghost
|
3bcc86f4a8 | refactor(imports): back PDF imports with statements (#1786) | ||
|
Guillem Arias Fauste
|
1ddd8bd040 |
feat(i18n): complete Catalan translations + extract residual hardcoded strings (#1836)
* feat(i18n): complete Catalan translations + extract residual hardcoded strings
CA coverage
- All view/model/breadcrumb/doorkeeper/mailer locale files for ca: 0 missing
keys (was ~3,400). Translations follow informal "tu" register, sentence case,
domain glossary (Compte/Saldo/Transacció/Posició/Operació/Pressupost/...).
- Catalan pluralization test: ca uses one/other; mirrors
test/lib/polish_pluralization_test.rb.
- 8 LanguageTool-flagged grammar fixes applied (Connexió òrfena, Secret de
l'API, comma-pero, apostrophe elisions, etc).
Hardcoded string extraction (also fixes EN parity)
- UI::Account::Chart#title + chart.html.erb view tabs -> UI.account.chart.*
- UI::Account::BalanceReconciliation labels + tooltips ->
UI.account.balance_reconciliation.{labels,tooltips}.*
- transactions/_transfer_match.html.erb (Auto-matched, A/M, Confirm/Reject
match, Payment/Transfer is confirmed) -> transactions.transfer_match.*
- AccountOrder labels (Name/Balance asc/desc) -> account_order.* keys with
fallback to existing hardcoded labels.
- Depository::SUBTYPES surface in account list -> depositories.subtypes.*.*
- User role badge -> users.roles.* (admin / member / super_admin).
- 110+ country names -> countries.* (config/locales/countries.ca.yml).
Breadcrumb locale fix
- Breadcrumbable was a before_action that ran before Localize's around_action
switched I18n.locale, so default crumbs rendered in EN even when locale=ca.
- Convert to helper_method that defers translation to render-time (when
I18n.locale is already correct). Add all missing breadcrumb keys to ca + en.
- Layouts switched from @breadcrumbs to breadcrumbs helper.
Locale-aware helpers / formatters
- ApplicationHelper#localized_ordinal: ordinalize that respects ca
(1r/2n/3r/4t/Nè). Wired into preferences month_start_day select.
- Family#moniker_label / moniker_label_plural: translate the default "Family"/
"Group" monikers via shared.family_moniker.* with fallback to the family's
custom override.
- Budget#name: use I18n.l for month_year/short/long instead of strftime("%B %Y")
so the budget header date follows the active locale.
Tooling
- script/lt_check_ca.rb: batched LanguageTool checker (premium endpoint when
LT_USERNAME/LT_API_KEY are set, free fallback otherwise), picky mode,
motherTongue=en for false-friend detection.
- lib/tasks/i18n_screenshot.rake: dev-only rake to set user.locale=ca and
role=super_admin on the demo user so the i18n surfaces can be walked.
Out of scope (pre-existing, not introduced here)
- Native browser file input "Choose Files / No file chosen" (browser locale).
- D3.js client-side chart x-axis dates (JS-side Intl.DateTimeFormat needed).
- Sankey/donut labels = seed category names (data, not i18n).
- 2 rails-i18n datetime/errors interpolation warnings inherited from
config/locales/defaults/ca.yml.
* fix(i18n): apply idiomatic Catalan review (3-agent + native review)
Three parallel review agents flagged 203 findings (31 high / 73 medium / 99 low)
across all 111 ca.yml files. This commit applies the high-severity bugs plus a
curated subset of medium-impact fixes.
Grammar / agreement
- provider_sync_summary.health.stale_pending: `(exclòs)` -> `(exclosa/excloses)`
to agree with feminine `transacció(s)`.
- accounts.confirm_unlink.warning_no_sync: added reflexive `es` -
`el compte ja no es sincronitzarà`.
- sophtron_setup_required.heading: `no configurats` -> `sense configurar`
(avoids broken agreement across "ID" masc. + "clau" fem.).
- admin.sso_providers.form.errors_title: split into one/other pluralization
keys (en + ca); singular `ha impedit` was wrong for count > 1.
Brand consistency
- IndexaCapital -> Indexa Capital (37 occurrences across one file).
- Lunchflow -> Lunch Flow in two remaining places.
Anglicisms / domain mistranslations
- kraken_items setup_accounts.instructions: `ompliments d'operacions`
(lit. dental/food fillings) -> `execucions d'operacions`.
- settings kraken_panel.read_only_title: `Sincronització d'intercanvi`
(swap/trade) -> `Sincronització només de lectura amb l'exchange`.
- transactions convert_to_trade.security_custom + security_not_listed_hint:
`cotització` (price quote) -> `ticker` (the EN field IS a ticker symbol).
- loans.form.rate_type: `Tipus d'interès` collided with sibling
interest_rate -> `Modalitat del tipus`.
- brex_items.provider_panel.sandbox_note_html: `L'staging` (broken
contraction) -> `el staging`.
Idiom traps
- coinbase/binance/kraken wait_for_sync: `acabi de sincronitzar` is
ambiguous in CA (`acabar de + inf` reads as "has just done X") ->
`acabi la sincronització`.
- chats.ai_greeting.there: `a tothom` -> `''` (the EN fallback "Hey there"
is singular; literal CA `tothom` is plural and wrong for 1:1 chat).
- transactions.split_parent_row.split_label: `Divideix` (imperative) is
wrong as a status badge -> `Divisió` (noun).
- transactions.keep_both (2 occurrences): infinitive `mantenir ambdues` ->
imperative `mantén-les totes dues` to match the sibling Yes/No buttons.
- rules.clear_ai_cache: `Reinicia` (restart) -> `Buida` (empty/clear),
which matches the success notice (`s'està netejant`).
Moniker gender breakage (cross-file)
%{moniker} is interpolated downcased from family.moniker_label and may
resolve to feminine `família`/`llar` or masculine `grup`. Strings that
hard-code a gendered article ('al teu %{moniker}', 'aquesta %{moniker}',
'aquest/a %{moniker}') broke on at least one branch. Restructured the
affected sentences to drop the gendered determiner:
- account_sharings.show.no_members
- merchants.family_empty / family_title / provider_empty
- registrations.new.join_family_title
- settings.preferences.show.currencies_subtitle / sharing_subtitle
- simplefin_items.select_existing_account.no_accounts_found
- invitations.new.subtitle
- invitation_mailer.invite_email.subject (mailers/) + body (views/)
- snaptrade_items.providers.snaptrade.free_tier_warning
Terminology consistency
- models/account_statement/ca.yml attributes aligned with view-side
forms: `Saldo d'obertura`/`Saldo de tancament` ->
`Saldo inicial`/`Saldo final`; `Suggeriment de...` -> `Pista de...`.
- account_statements.coverage.status.not_expected:
`No s'esperava` -> `No previst` (status label, not past action).
- account_statements.index.empty_unmatched: aligned with the section's
own label `Safata sense aparellar`.
- imports.create.document_provider_not_configured + document_upload_failed:
`arxiu vectorial` -> `magatzem vectorial` (correct TermCat term).
- coinstats_items blockchain gender: `els blockchains` / `un blockchain` ->
`les blockchains` / `una blockchain` (feminine per TermCat).
- accounts.account.remove_default: `Treu el predeterminat` ->
`Treu com a predeterminat` (pairs with sibling `Estableix com a
predeterminat`).
- accounts.tax_treatments.tax_deferred: `Diferit fiscalment` (lit. calque)
-> `Tributació diferida` (standard CA tax-accounting term).
- settings.payments.show.currently_on_plan: `Actualment al` ->
`Actualment al pla:` (was a fragment).
Out of scope (review flagged, not applied here)
- LOW-severity stylistic preferences (Veure vs Mostra, etc).
- `models/category/ca.yml` default category names — seeded at family
creation, not via I18n at runtime, so changes wouldn't affect existing
families.
- `models/period/ca.yml` short labels mixing EN (MTD/YTD) and CA (STD/MA)
— needs a one-convention decision separately.
* fix(i18n,ca): drop gendered article in period_activity + tighten cash-flow terms
- pages.dashboard.investment_summary.period_activity: 'Activitat del
%{period}' contracted 'del' = 'de el' (masc.sg.). %{period} resolves
to mixed forms ('Setmana en curs' fem, 'Últims 30 dies' pl., 'Any en
curs' apostrophe), so hard-coded 'del' was wrong on most labels.
Replaced with 'Activitat — %{period}' (em-dash) to skip the
contraction entirely.
- pages.dashboard.outflows_donut.title / total_outflows: switched from
bare 'Sortides' / 'Total de sortides' to 'Sortides de caixa' /
'Total de sortides de caixa' to match TermCat's precise term
('sortida de caixa' = cash outflow).
* fix(i18n,ca): rephrase transfer source/destination amount labels
'Import d'origen' / 'Import de destinació' were literal calques of
'Source amount' / 'Destination amount'. In a multi-currency transfer
form (sender/receiver in different currencies) the natural CA pair is
'Import enviat' / 'Import rebut'.
* fix(i18n,ca): 'Dades en brut' -> 'Dades sense processar'
The literal calque of 'Raw data' read as too technical for personal-
finance UI. 'Dades sense processar' is the more natural Catalan
equivalent for raw/unprocessed data files.
* fix(i18n): localize Import col_sep label + separator options
The CSV upload form rendered 'Col sep' (the auto-humanized attribute
name) plus hardcoded English 'Comma (,)' / 'Semicolon (;)' options
from Import::SEPARATORS.
- activerecord.attributes.import.col_sep added (en + ca: 'Column
separator' / 'Separador de columnes').
- Import.separator_options class method returns translated tuples;
view switched from Import::SEPARATORS to Import.separator_options.
- activerecord.attributes.import.col_seps.{comma,semicolon} added so
the option labels follow the active locale.
* fix(i18n,ca): drop moniker apposition in sharing/currencies section titles
- sharing_title 'Compartició de %{moniker}' rendered as 'Compartició
de Família' (a noun-noun apposition that's odd in CA) -> 'Compartició
de comptes'.
- sharing_subtitle replaced '%{moniker}' with 'entre els membres' so
the sentence reads naturally and doesn't depend on moniker gender.
- currencies_title 'Divises de %{moniker}' had the same apposition
-> 'Divises'. Subtitle no longer references moniker either.
* fix(i18n,ca): keep 'Self Hosting' untranslated
Reverted 'Autoallotjament' / 'autoallotjada' / 'autoallotjats' usages
to the original English 'Self Hosting' (sidebar label, breadcrumbs,
hostings page title, chat assistant settings hint, redis configuration
subheading, LLM usages cost-estimates description).
The brand-style term reads more naturally in EN for technical users
configuring their own deployment.
* fix(i18n,ca): lowercase 'self hosting' (sentence case in labels)
* fix(i18n): extract budget_categories stepper + allocation_progress strings
Hardcoded English strings on the budget category editor:
- 'Setup' / 'Categories' stepper labels in budgets/_budget_nav.html.erb
- 'X% set' / '> 100% set' / 'left to allocate' / 'Budget exceeded by ...'
in budget_categories/_allocation_progress.erb
- '/m avg' caption + 'Shared' placeholder + 'Leave empty to share
parent's budget' tooltip in budget_categories/_budget_category_form
and _uncategorized_budget_category_form
Extracted to:
- budgets.budget_nav.{setup,categories}
- budget_categories.allocation_progress.{percent_set,over_set,left_to_allocate,budget_exceeded_html}
- budget_categories.budget_category_form.{monthly_average,shared_placeholder,shared_title}
CA translations added; EN keys mirror the prior literals.
* chore(i18n): drop translation tooling from PR
These were dev-only helpers used during the Catalan translation pass:
- script/lt_check_ca.rb: LanguageTool API checker (premium/free
endpoint, picky mode, batching). Useful for ongoing locale QA but
shouldn't ship in this feature PR.
- lib/tasks/i18n_screenshot.rake: rake task that flips user.locale and
role on the demo user for walking the i18n surfaces locally.
Both stay available locally; pulled out of the PR scope.
* fix(i18n): apply PR review feedback (CodeRabbit + Codex)
- balance_reconciliation crypto_items: use :end_balance_crypto tooltip
(was :end_balance_investment). Added new UI.account.balance_reconciliation.tooltips.end_balance_crypto key in en + ca.
- doorkeeper.ca.yml confidentiality.no: was YAML boolean false, now string 'No'.
- views/categories: 'Poor contrast, choose darker color or' continued with hardcoded 'auto-adjust.' button text; extracted to categories.form.auto_adjust key (en + ca).
- imports.create.document_upload_failed: 'a l'magatzem' was broken
contraction -> 'al magatzem'.
- invitation_mailer body + mailer subject: 'unir-se' -> 'unir-te' (was
3rd person, should be 2nd to match the rest of the copy).
- 7 strings across mercury_items / sophtron_items / simplefin_items /
lunchflow_items / brex_items / indexa_capital_items / other_assets:
'se sincronitzaran' -> 'es sincronitzaran', 'se segueixen' ->
'es segueixen' (correct reflexive pronoun before consonants).
- settings.providers.status: key was 'false' (YAML-coerced), now 'off'
to match settings/en.yml status.off used in view lookups.
- sophtron_items.sophtron_setup_required.message: stripped trailing
blank line from the quoted scalar.
- settings/profiles/show.html.erb: switched 'family_moniker ==
"Group"' branch checks to 'Current.family&.moniker == "Group"'.
After Family#moniker_label started returning translated values,
callers using the display label for branching would render the
household copy for group families in ca. Compare the stored sentinel
instead.
- Did not apply CodeRabbit's webauthn 'eliminada' -> 'desada' suggestion:
the key is wired to the destroy action (verified at
settings/webauthn_credentials_controller.rb:55), so 'eliminada' is
correct.
|
||
|
Sure Admin (bot)
|
4fd460d551 |
Add Actual Budget CSV import flow (#1830)
* Add Actual Budget CSV import flow * Address Actual import review feedback |
||
|
Brendon ScheiberandKiloClaw
|
0c126b1674 |
feat(i18n): extract hardcoded English strings to locale files (#1806)
* Extract hardcoded strings to i18n
Replace numerous hardcoded English strings with I18n lookups (t / I18n.t) across controllers, views, helpers, and components, and convert model validation error messages to symbol keys. Added multiple locale files under config/locales for models and views. This centralizes user-facing notices/alerts, UI text, import/validation messages, and prepares the app for localization and easier translation maintenance.
* Update en.yml
* Update preview-cleanup.yml
* Revert "Update preview-cleanup.yml"
This reverts commit
|
||
|
ghostandJuan José Mata
|
42e7ae677a |
fix(exports): align CSV roundtrip contracts (#1725)
* fix(exports): align CSV roundtrip contracts * fix(exports): version CSV export contract * fix(exports): stabilize CSV export values * fix(imports): preserve legacy CSV roundtrip contracts * fix(imports): escape pipe characters in CSV tags --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
ghost
|
1fedc43f68 |
feat(api): add import preflight validation (#1755)
* feat(api): add import preflight validation * fix(api): harden import preflight validation |
||
|
ghost
|
1ec8bd90b7 |
feat(api): expose import row diagnostics (#1644)
* feat(api): expose import row diagnostics * fix(api): stabilize import row diagnostics * fix(api): harden import row diagnostics * fix(api): number Mint import diagnostics rows * fix(api): enforce unique import row diagnostics * fix(api): address import row diagnostics review |
||
|
ghost
|
c4414c4fbb |
feat(api): expose import status details (#1599)
* feat(api): expose import status details * fix(api): reuse import status validation counts * fix(api): cache Sure import status reads * fix(imports): invalidate cached Sure import blobs * docs(api): split import status schemas * fix(api): refine import status detail contract |
||
|
Serge LandClaude Opus 4.6
|
78b334277c |
QIF imports: Add date format auto-detection and manual override (#1368)
* feat: improve QIF import date format selection - Added a reusable date format auto-detection method. - Show a live preview of the first parsed date that updates client-side as the user changes the dropdown selection, via a new qif-date-format Stimulus controller. - Show an error alert and disable the submit button when no supported date format can parse the file's dates. * A few polishing fixes: - Missing return on redirects Stale REASONABLE_DATE_RANGE constant. - Replaced the frozen constant with a class method Bare inline rescue — Replaced Date.strptime(s, fmt) rescue nil with an explicit begin/rescue catching. - save!(validate: false) in controller — Changed to update_column(:column_mappings, ...) in qif_category_selections_controller.rb:22, matching the pattern used in detect_and_set_qif_date_format!. - Unescaped JSON in HTML attribute — Replaced the raw <div> with tag.div ... do block in show.html.erb:16, letting Rails properly escape the data attribute value. * fix: address review feedback for QIF date format feature - Add missing `return` after redirect for non-QIF imports - Pass date_format to parse_opening_balance in will_adjust_opening_anchor? - Return empty array when no usable date sample exists for format preview - Add sr-only label to date format select for accessibility - Consolidate duplicate try_parse_date/parse_qif_date into single method - Remove misleading ambiguity scoring comment from detect_date_format - Skip redundant sync_mappings when date format already triggered a sync Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Use %{product_name} interpolation in locale strings --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Juan José MataandClaude
|
2595885eb7 |
Full .ndjson import / reorganize UI with Financial Tools / Raw Data tabs (#1208)
* Reorganize import UI with Financial Tools / Raw Data tabs Split the flat list of import sources into two tabbed sections using DS::Tabs: "Financial Tools" (Mint, Quicken/QIF, YNAB coming soon) and "Raw Data" (transactions, investments, accounts, categories, rules, documents). This prepares for adding more tool-specific importers without cluttering the list. https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3 * Fix import controller test to account for YNAB coming soon entry The new YNAB "coming soon" disabled entry adds a 5th aria-disabled element to the import dialog. https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3 * Fix system tests to click Raw Data tab before selecting import type Transaction, trade, and account imports are now under the Raw Data tab and need an explicit tab click before the buttons are visible. https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3 * feat: Add bulk import for NDJSON export files Implements an import flow that accepts the full all.ndjson file from data exports, allowing users to restore their complete data including: - Accounts with accountable types - Categories with parent relationships - Tags and merchants - Transactions with category, merchant, and tag references - Trades with securities - Valuations - Budgets and budget categories - Rules with conditions and actions (including compound conditions) Key changes: - Add BulkImport model extending Import base class - Add Family::DataImporter to handle NDJSON parsing and import logic - Update imports controller and views to support NDJSON workflow - Skip configuration/mapping steps for structured NDJSON imports - Add i18n translations for bulk import UI - Add tests for BulkImport and DataImporter * fix: Fix category import and test query issues - Add default lucide_icon ("shapes") for categories when not provided - Fix valuation test to use proper ActiveRecord joins syntax * Linter errors * fix: Add default color for tags when not provided in import * fix: Add default kind for transactions when not provided in import * Fix test * Fix tests * Fix remaining merge conflicts from PR 766 cherry-pick Resolve conflict markers in test fixtures and clean up BulkImport entry in new.html.erb to use the _import_option partial consistently. https://claude.ai/code/session_01BM4SBWNhATqoKTEvy3qTS3 * Import Sure `.ndjson` * Remove `.ndjson` import from raw data * Fix support for Sure "bulk" import from old branch * Linter * Fix CI test * Fix more CI tests * Fix tests * Fix tests / move PDF import to first tab * Remove redundant title --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Serge LandClaude Sonnet 4.6
|
57199d6eb9 |
Feat: Add QIF (Quicken Interchange Format) import functionality (#1074)
* Feat: Add QIF (Quicken Interchange Format) import functionality - Add the ability to import QIF files for users coming from Quicken - Includes categories and tags - Comprehensive tests for QifImport, including parsing, row generation, and import functionality. - Ensure handling of hierarchical categories (ex "Home:Home Improvement" is imported as Parent:Child) * Fix QIF import issues raised in code review - Fix two-digit year windowing in QIF date parser (e.g. '99 → 1999, not 2099) - Fix ArgumentError from invalid `undef: :raise` encoding option - Nil-safe `leaf_category_name` with blank guard and `.to_s` coercion - Memoize `qif_account_type` to avoid re-parsing the full QIF file - Add strong parameters (`selection_params`) to QifCategorySelectionsController - Wrap all mutations in DB transactions in uploads and category-selections controllers - Skip unchanged tag rows (only write rows where tags actually differ) - Replace hardcoded strings with i18n keys across QIF views and nav - Fix potentially colliding checkbox/label IDs in category selection view - Improve keyboard accessibility: use semantic `<label>` for file picker area Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix QIF import test count and Brakeman mass assignment warning - Update ImportsControllerTest to expect 4 disabled import options (was 3), accounting for the new QIF import type added in this branch - Remove :account_id from upload_params permit list; it was never accessed through strong params (always via params.dig with Current.family scope), so this resolves the Brakeman high-confidence mass assignment warning Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Fix: QIF import security, safety, and i18n issues raised in code review - Added french, spanish and german translations for newly added i18n keys - Replace params.dig(:import, :account_id) with a proper strong-params accessor (import_account_id) in UploadsController to satisfy Rails parameter filtering requirements - Guard ImportsController#show against QIF imports reaching the publish screen before a file has been uploaded, preventing an unrescued error on publish - Gate the QIF "Clean" nav step link on import.uploaded? to prevent routing to CleansController with an unconfigured import (which would raise "Unknown import type: QifImport" via ImportsHelper) - Replace hard-coded "txn" pluralize calls in the category/tag selection view with t(".txn_count") and add pluralization keys to the locale file - Localize all hard-coded strings in the QIF upload section of uploads/show.html.erb and add corresponding en.yml keys - Convert the CSV upload drop zone from a clickable <div> (JS-only) to a semantic <label> element, making it keyboard-accessible without JavaScript * Fix: missing translations keys * Add icon mapping and random color assignment to new categories * fix a lint issue * Add a warning about splits and some plumbing for future support. Updated locales. --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> |
||
|
|
6f8858b1a6 |
feat/Add AI-Powered Bank Statement Import (step 1, PDF import & analysis) (#808)
* feat: Add PDF import with AI-powered document analysis This enhances the import functionality to support PDF files with AI-powered document analysis. When a PDF is uploaded, it is processed by AI to: - Identify the document type (bank statement, credit card statement, etc.) - Generate a summary of the document contents - Extract key metadata (institution, dates, balances, transaction count) After processing, an email is sent to the user asking for next steps. Key changes: - Add PdfImport model for handling PDF document imports - Add Provider::Openai::PdfProcessor for AI document analysis - Add ProcessPdfJob for async PDF processing - Add PdfImportMailer for user notification emails - Update imports controller to detect and handle PDF uploads - Add PDF import option to the new import page - Add i18n translations for all new strings - Add comprehensive tests for the new functionality * Add bank statement import with AI extraction - Create ImportBankStatement assistant function for MCP - Add BankStatementExtractor with chunked processing for small context windows - Register function in assistant configurable - Make PdfImport#pdf_file_content public for extractor access - Increase OpenAI request timeout to 600s for slow local models - Increase DB connection pool to 20 for concurrent operations Tested with M-Pesa bank statement via remote Ollama (qwen3:8b): - Successfully extracted 18 transactions - Generated CSV and created TransactionImport - Works with 3000 char chunks for small context windows * Add pdf-reader gem dependency The BankStatementExtractor uses PDF::Reader to parse bank statement PDFs, but the gem was not properly declared in the Gemfile. This would cause NameError in production when processing bank statements. Added pdf-reader ~> 2.12 to Gemfile dependencies. * Fix transaction deduplication to preserve legitimate duplicates The previous deduplication logic removed ALL duplicate transactions based on [date, amount, name], which would drop legitimate same-day duplicates like multiple ATM withdrawals or card authorizations. Changed to only deduplicate transactions that appear in consecutive chunks (chunking artifacts) while preserving all legitimate duplicates within the same chunk or non-adjacent chunks. * Refactor bank statement extraction to use public provider method Address code review feedback: - Add public extract_bank_statement method to Provider::Openai - Remove direct access to private client via send(:client) - Update ImportBankStatement to use new public method - Add require 'set' to BankStatementExtractor - Remove PII-sensitive content from error logs - Add defensive check for nil response.error - Handle oversized PDF pages in chunking logic - Remove unused process_native and process_generic methods - Update email copy to reflect feature availability - Add guard for nil document_type in email template - Document pdf-reader gem rationale in Gemfile Tested with both OpenAI (gpt-4o) and Ollama (qwen3:8b): - OpenAI: 49 transactions extracted in 30s - Ollama: 40 transactions extracted in 368s - All encapsulation and error handling working correctly * Update schema.rb with ai_summary and document_type columns * Address PR #808 review comments - Rename :csv_file to :import_file across controllers/views/tests - Add PDF test fixture (sample_bank_statement.pdf) - Add supports_pdf_processing? method for graceful degradation - Revert unrelated database.yml pool change (600->3) - Remove month_start_day schema bleed from other PR - Fix PdfProcessor: use .strip instead of .strip_heredoc - Add server-side PDF magic byte validation - Conditionally show PDF import option when AI provider available - Fix ProcessPdfJob: sanitize errors, handle update failure - Move pdf_file attachment from Import to PdfImport - Document deduplication logic limitations - Fix ImportBankStatement: catch specific exceptions only - Remove unnecessary require 'set' - Remove dead json_schema method from PdfProcessor - Reduce default OpenAI timeout from 600s to 60s - Fix nil guard in text mailer template - Add require 'csv' to ImportBankStatement - Remove Gemfile pdf-reader comment * Fix RuboCop indentation in ProcessPdfJob * Refactor PDF import check to use model predicate method Replace is_a?(PdfImport) type check with requires_csv_workflow? predicate that leverages STI inheritance for cleaner controller logic. * Fix missing 'unknown' locale key and schema version mismatch - Add 'unknown: Unknown Document' to document_types locale - Fix schema version to match latest migration (2026_01_24_180211) * Document OPENAI_REQUEST_TIMEOUT env variable Added to .env.local.example and docs/hosting/ai.md * Rename ALLOWED_MIME_TYPES to ALLOWED_CSV_MIME_TYPES for clarity * Add comment explaining requires_csv_workflow? predicate * Remove redundant required_column_keys from PdfImport Base class already returns [] by default * Add ENV toggle to disable PDF processing for non-vision endpoints OPENAI_SUPPORTS_PDF_PROCESSING=false can be used for OpenAI-compatible endpoints (e.g., Ollama) that don't support vision/PDF processing. * Wire up transaction extraction for PDF bank statements - Add extracted_data JSONB column to imports - Add extract_transactions method to PdfImport - Call extraction in ProcessPdfJob for bank statements - Store transactions in extracted_data for later review * Fix ProcessPdfJob retry logic, sanitize and localize errors - Allow retries after partial success (classification ok, extraction failed) - Log sanitized error message instead of raw message to avoid data leakage - Use i18n for user-facing error messages * Add vision-capable model validation for PDF processing * Fix drag-and-drop test to use correct field name csv_file * Schema bleedover from another branch * Fix drag-drop import form field name to match controller * Add vision capability guard to process_pdf method --------- Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com> Co-authored-by: Juan José Mata <jjmata@jjmata.com> |
||
|
Mark HendriksenandJuan José Mata
|
9b84c5bdbc |
Enhanced Import Amount Type Selection (#506)
* Enhanced Import Amount Type Selection updated version of https://github.com/we-promise/sure/pull/179 * copilot sugestions * ai sugestions * Update import.rb * Update schema.rb * Update schema.rb * Update schema.rb --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
sokie
|
11a011db15 | Update import.rb | ||
|
sokie
|
c9b334ac4f | Add rows validation | ||
|
sokie
|
7297554a55 |
Fix issues
Issue 1 Fixed - Template now carries rows_to_skip. Issue 2 Fixed - Column headers refresh when rows_to_skip changes. |
||
|
Juan José Mata
|
accdbb799b |
Merge branch 'main' into add-config-import-csv-skip-first-x-rows
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
Copilotandjjmata
|
5b736bf691 |
Fix CSV import for non-UTF-8 encodings (Windows-1250, ISO-8859-2, etc.) (#617)
* Initial plan * Add encoding detection for CSV imports to handle Windows-1250 and other non-UTF-8 encodings Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com> * Improve encoding detection: prioritize Windows-1250 and increase confidence threshold Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com> * Update Gemfile.lock with rchardet dependency Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com> * Refactor: Extract common encodings to constant and deduplicate code Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com> * Fix Rubocop style violations Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com> * Fix linter violations and encoding detection logic - Remove trailing whitespace from test file (9 lines) - Fix ensure_utf8_encoding to handle binary strings properly by checking bytesize instead of blank? - Add error handling for ArgumentError and Encoding::CompatibilityError - Add invalid/undef replacement options to encode calls for robustness Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com> * Fix encoding error in ensure_utf8_encoding method - Use will_save_change_to_raw_file_str? instead of raw_file_str_changed? to avoid encoding errors when checking if attribute changed - Wrap UTF-8 validation check in begin/rescue to handle ArgumentError from invalid encodings - This fixes the test failure: "ArgumentError: invalid byte sequence in UTF-8" Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com> * Fix test: add missing column labels and reload import before checking rows Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com> * Fix test: ensure import is reloaded before checking rows_count and accessing rows Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com> |
||
|
Juan José Mata
|
664a00678e |
Merge branch 'main' into add-config-import-csv-skip-first-x-rows
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
|
b56dbdb9eb |
Feat: /import endpoint & drag-n-drop imports (#501)
* Implement API v1 Imports controller - Add Api::V1::ImportsController with index, show, and create actions - Add Jbuilder views for index and show - Add integration tests - Implement row generation logic in create action - Update routes * Validate import account belongs to family - Add validation to Import model to ensure account belongs to the same family - Add regression test case in Api::V1::ImportsControllerTest * updating docs to be more detailed * Rescue StandardError instead of bare rescue in ImportsController * Optimize Imports API and fix documentation - Implement rows_count counter cache for Imports - Preload rows in Api::V1::ImportsController#show - Update documentation to show correct OAuth scopes * Fix formatting in ImportsControllerTest * Permit all import parameters and fix unknown attribute error * Restore API routes for auth, chats, and messages * removing pr summary * Fix trailing whitespace and configured? test failure - Update Import#configured? to use rows_count for performance and consistency - Mock rows_count in TransactionImportTest - Fix trailing whitespace in migration * Harden security and fix mass assignment in ImportsController - Handle type and account_id explicitly in create action - Rename import_params to import_config_params for clarity - Validate type against Import::TYPES * Fix MintImport rows_count update and migration whitespace - Update MintImport#generate_rows_from_csv to update rows_count counter cache - Fix trailing whitespace and final newline in AddRowsCountToImports migration * Implement full-screen Drag and Drop CSV import on Transactions page - Add DragAndDropImport Stimulus controller listening on document - Add full-screen overlay with icon and text to Transactions index - Update ImportsController to handle direct file uploads via create action - Add system test for drag and drop functionality * Implement Drag and Drop CSV upload on Import Upload page - Add drag-and-drop-import controller to import/uploads/show - Add full-screen overlay to import/uploads/show - Annotate upload form and input with drag-and-drop targets - Add PR_SUMMARY.md * removing pr summary * Add file validation to ImportsController - Validate file size (max 10MB) and MIME type in create action - Prevent memory exhaustion and invalid file processing - Defined MAX_CSV_SIZE and ALLOWED_MIME_TYPES in Import model * Refactor dragLeave logic with counter pattern to prevent flickering * Extract shared drag-and-drop overlay partial - Create app/views/imports/_drag_drop_overlay.html.erb - Update transactions/index and import/uploads/show to use the partial - Reduce code duplication in views * Update Brakeman and harden ImportsController security - Update brakeman to 7.1.2 - Explicitly handle type assignment in ImportsController#create to avoid mass assignment - Remove :type from permitted import parameters * Fix trailing whitespace in DragAndDropImportTest * Don't commit LLM comments as file * FIX add api validation --------- Co-authored-by: Carlos Adames <cj@Carloss-MacBook-Air.local> Co-authored-by: Juan José Mata <jjmata@jjmata.com> Co-authored-by: sokie <sokysrm@gmail.com> |
||
|
kasra
|
e8216984a7 | Add rows_to_skip config for transaction import | ||
|
soky srm
|
64c25725c9 |
Fix CSV import with no currency (#462)
* FIX use the accounts we are importing currency as default, not family default * FIX add family fallback for multi account import |
||
|
Juan José MataandClaude
|
e5ed946959 |
Add rules import/export support (#424)
* Add full import/export support for rules with versioned JSON schema This commit implements comprehensive import/export functionality for rules, allowing users to back up and restore their rule definitions. Key features: - Export rules to both CSV and NDJSON formats with versioned schema (v1) - Import rules from CSV with full support for nested conditions and actions - UUID to name mapping for categories and merchants for portability - Support for compound conditions with sub-conditions - Comprehensive test coverage for export and import functionality - UI integration for rules import in the imports interface Technical details: - Extended Family::DataExporter to generate rules.csv and include rules in all.ndjson - Created RuleImport model following the existing Import STI pattern - Added migration for rule-specific columns in import_rows table - Implemented serialization helpers to map UUIDs to human-readable names - Added i18n support for the new import option - Included versioning in NDJSON export to support future schema evolution The implementation ensures rules can be safely exported from one family and imported into another, even when category/merchant IDs differ, by mapping between names and IDs during export/import. * Fix AR migration version * Mention support for rules export * Rabbit suggestion * Fix tests * Missed schema.rb * Fix sample CSV download for rule import * Fix parsing in Rules import * Fix tests * Rule import message i18n * Export tag names, not UUIDs * Make sure tags are created if needed at import * Avoid test errors when running in parallel --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Juan José Mata
|
65bd9e813a |
Support category .csv import (#326)
* Allow category imports to set icons * Linter * Update category import description in English locale Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> * Rename icon in CSV header to `lucide-icon` * Make sure we export the icon while we're at this --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> |
||
|
soky srm
|
f882484bba |
Add transaction dedup support for CSV imports (#304)
* Support dedup for transaction also for CSV * Fix to exclude CSV importing duplicates * Guard nil account |
||
|
Zach Gollwitzer
|
bacab94a1b | Fix import reverts | ||
|
Zach Gollwitzer
|
e26e5c5aec |
Auto sync preference, max limit on account CSV imports (#2259)
* Auto sync preference, max limit on account CSV imports * MaxRowCountExceededError |
||
|
Zach Gollwitzer
|
10dd9e061a |
Improve account sync performance, handle concurrent market data syncing (#2236)
* PlaidConnectable concern * Remove bad abstraction * Put sync implementations in own concerns * Sync strategies * Move sync orchestration to Sync class * Clean up sync class, add state machine * Basic market data sync cron * Fix price sync * Improve sync window column names, add timestamps * 30 day syncs by default * Clean up market data methods * Report high duplicate sync counts to Sentry * Add sync states throughout app * account tab session * Persistent account tab selections * Remove manual sleep * Add migration to clear stale syncs on self hosted apps * Tweak sync states * Sync completion event broadcasts * Fix timezones in tests * Cleanup * More cleanup * Plaid item UI broadcasts for sync * Fix account ID namespace conflict * Sync broadcasters * Smoother account sync refreshes * Remove test sync delay |
||
|
Zach Gollwitzer
|
c88fe2e3b2 |
Feature: Add "amount type" configuration column for CSV imports (#1947)
* Rough draft * Schema conflict update * Implement signage * Update system tests * Lint fixes |
||
|
Zach Gollwitzer
|
e657c40d19 |
Account:: namespace simplifications and cleanup (#2110)
* Flatten Holding model * Flatten balance model * Entries domain renames * Fix valuations reference * Fix trades stream * Fix brakeman warnings * Fix tests * Replace existing entryable type references in DB |
||
|
Zach Gollwitzer
|
0544089710 |
Account-level import configuration templates (#1946)
* Account-level import configuration templates * Default import to family's preferred date format |
||
|
Zach Gollwitzer
|
c5da8ea550 |
Allow CSV imports to be configured with single or multi-account mode (#1943)
* Allow CSV imports to be configured to a single account or multiple accounts * Initialize import directly from accounts page * Fix brakeman warnings * Fix schema * Fix Synth check |
||
|
Zach Gollwitzer
|
4d0df9b950 |
Escape quotations in CSV imports properly (#1929)
* Parse quotes in imports * Update invalid CSV for test |
||
|
David Anyatonwu
|
32ef6ca154 |
Add exchange and currency fields to trade imports (#1822)
* Add exchange and currency fields to trade imports * Add exchange_operating_mic support for trade imports - Added required columns and updated models * refactor: remove exchange and currency columns * fix: consolidate import schema and remove redundant columns * feat: Enhance trade import with exchange_operating_mic support * Revert changes to existing migration * Simplify migration to use change method * Restore previously deleted migration * Remove unused import_col_labels method * Update schema.rb after running migrations * Update trade_import.rb and fix schema.rb with db:migrate:reset * fix: improve trade import security creation --------- Signed-off-by: David Anyatonwu <51977119+onyedikachi-david@users.noreply.github.com> |
||
|
Zach Gollwitzer
|
8539ac7dec | Fix import configuration failures (#1876) | ||
|
Daniel Estevesanddanestves
|
077694bbde |
feat(import): add currency and number format support for CSV imports (#1819)
* feat(import): add currency and number format support for CSV imports * feat(import): add currency and format for mint and trade * fix(imports): remove currency field in favor of currency csv col * fix(imports): remove validate column from import model * test(import): add some tests for imports * test(import): add some tests for generate_rows_from_csv * fix: change method name for import model * fix: change before validation --------- Co-authored-by: danestves <danestves@users.noreply.github.com> |
||
|
Zach Gollwitzer
|
536c82f2aa |
Feature: Add the ability to "revert" a CSV import (#1814)
* Allow reverting imports * Fix tests * Add currency column to all imports * Don't auto-enrich demo account |
||
|
Zach Gollwitzer
|
79ca7e2039 | Preserve negative sign on raw CSV values | ||
|
Zach Gollwitzer
|
a20809eee3 | When unassigned accounts in CSV import, always allow new account creation | ||
|
Zach Gollwitzer
|
cd9f20747c |
Allow inline account creation when importing CSV (#1291)
* Allow inline account creation when importing CSV * Sanitize numeric inputs for CSV * CSV import date validation * Lint fix |
||
|
Zach Gollwitzer
|
398b246965 |
CSV Imports Overhaul (Transactions, Trades, Accounts, and Mint import support) (#1209)
* Remove stale 1.0 import logic and model * Fresh start * Checkpoint before removing nav * First working prototype * Add trade, account, and mint import flows * Basic working version with tests * System tests for each import type * Clean up mappings flow * Clean up PR, refactor stale code, tests * Add back row validations * Row validations * Fix import job test * Fix import navigation * Fix mint import configuration form * Currency preset for new accounts |