Commit Graph

37 Commits

Author SHA1 Message Date
Carlos Adames
b56dbdb9eb Feat: /import endpoint & drag-n-drop imports (#501)
* Implement API v1 Imports controller

- Add Api::V1::ImportsController with index, show, and create actions
- Add Jbuilder views for index and show
- Add integration tests
- Implement row generation logic in create action
- Update routes

* Validate import account belongs to family

- Add validation to Import model to ensure account belongs to the same family
- Add regression test case in Api::V1::ImportsControllerTest

* updating docs to be more detailed

* Rescue StandardError instead of bare rescue in ImportsController

* Optimize Imports API and fix documentation

- Implement rows_count counter cache for Imports
- Preload rows in Api::V1::ImportsController#show
- Update documentation to show correct OAuth scopes

* Fix formatting in ImportsControllerTest

* Permit all import parameters and fix unknown attribute error

* Restore API routes for auth, chats, and messages

* removing pr summary

* Fix trailing whitespace and configured? test failure

- Update Import#configured? to use rows_count for performance and consistency
- Mock rows_count in TransactionImportTest
- Fix trailing whitespace in migration

* Harden security and fix mass assignment in ImportsController

- Handle type and account_id explicitly in create action
- Rename import_params to import_config_params for clarity
- Validate type against Import::TYPES

* Fix MintImport rows_count update and migration whitespace

- Update MintImport#generate_rows_from_csv to update rows_count counter cache
- Fix trailing whitespace and final newline in AddRowsCountToImports migration

* Implement full-screen Drag and Drop CSV import on Transactions page

- Add DragAndDropImport Stimulus controller listening on document
- Add full-screen overlay with icon and text to Transactions index
- Update ImportsController to handle direct file uploads via create action
- Add system test for drag and drop functionality

* Implement Drag and Drop CSV upload on Import Upload page

- Add drag-and-drop-import controller to import/uploads/show
- Add full-screen overlay to import/uploads/show
- Annotate upload form and input with drag-and-drop targets
- Add PR_SUMMARY.md

* removing pr summary

* Add file validation to ImportsController

- Validate file size (max 10MB) and MIME type in create action
- Prevent memory exhaustion and invalid file processing
- Defined MAX_CSV_SIZE and ALLOWED_MIME_TYPES in Import model

* Refactor dragLeave logic with counter pattern to prevent flickering

* Extract shared drag-and-drop overlay partial

- Create app/views/imports/_drag_drop_overlay.html.erb
- Update transactions/index and import/uploads/show to use the partial
- Reduce code duplication in views

* Update Brakeman and harden ImportsController security

- Update brakeman to 7.1.2
- Explicitly handle type assignment in ImportsController#create to avoid mass assignment
- Remove :type from permitted import parameters

* Fix trailing whitespace in DragAndDropImportTest

* Don't commit LLM comments as file

* FIX add api validation

---------

Co-authored-by: Carlos Adames <cj@Carloss-MacBook-Air.local>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: sokie <sokysrm@gmail.com>
2026-01-10 16:39:18 +01:00
soky srm
64c25725c9 Fix CSV import with no currency (#462)
* FIX use the accounts we are importing currency as default, not family default

* FIX add family fallback for multi account import
2025-12-17 18:37:35 +01:00
Juan José Mata
e5ed946959 Add rules import/export support (#424)
* Add full import/export support for rules with versioned JSON schema

This commit implements comprehensive import/export functionality for rules,
allowing users to back up and restore their rule definitions.

Key features:
- Export rules to both CSV and NDJSON formats with versioned schema (v1)
- Import rules from CSV with full support for nested conditions and actions
- UUID to name mapping for categories and merchants for portability
- Support for compound conditions with sub-conditions
- Comprehensive test coverage for export and import functionality
- UI integration for rules import in the imports interface

Technical details:
- Extended Family::DataExporter to generate rules.csv and include rules in all.ndjson
- Created RuleImport model following the existing Import STI pattern
- Added migration for rule-specific columns in import_rows table
- Implemented serialization helpers to map UUIDs to human-readable names
- Added i18n support for the new import option
- Included versioning in NDJSON export to support future schema evolution

The implementation ensures rules can be safely exported from one family
and imported into another, even when category/merchant IDs differ,
by mapping between names and IDs during export/import.

* Fix AR migration version

* Mention support for rules export

* Rabbit suggestion

* Fix tests

* Missed schema.rb

* Fix sample CSV download for rule import

* Fix parsing in Rules import

* Fix tests

* Rule import message i18n

* Export tag names, not UUIDs

* Make sure tags are created if needed at import

* Avoid test errors when running in parallel

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-12-07 13:20:54 +01:00
Juan José Mata
65bd9e813a Support category .csv import (#326)
* Allow category imports to set icons

* Linter

* Update category import description in English locale

Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>

* Rename icon in CSV header to `lucide-icon`

* Make sure we export the icon while we're at this

---------

Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
2025-11-15 19:00:52 +01:00
soky srm
f882484bba Add transaction dedup support for CSV imports (#304)
* Support dedup for transaction also for CSV

* Fix to exclude CSV importing duplicates

* Guard nil account
2025-11-10 12:09:22 +01:00
Zach Gollwitzer
bacab94a1b Fix import reverts 2025-07-24 11:41:42 -04:00
Zach Gollwitzer
e26e5c5aec Auto sync preference, max limit on account CSV imports (#2259)
* Auto sync preference, max limit on account CSV imports

* MaxRowCountExceededError
2025-05-18 15:02:51 -04:00
Zach Gollwitzer
10dd9e061a Improve account sync performance, handle concurrent market data syncing (#2236)
* PlaidConnectable concern

* Remove bad abstraction

* Put sync implementations in own concerns

* Sync strategies

* Move sync orchestration to Sync class

* Clean up sync class, add state machine

* Basic market data sync cron

* Fix price sync

* Improve sync window column names, add timestamps

* 30 day syncs by default

* Clean up market data methods

* Report high duplicate sync counts to Sentry

* Add sync states throughout app

* account tab session

* Persistent account tab selections

* Remove manual sleep

* Add migration to clear stale syncs on self hosted apps

* Tweak sync states

* Sync completion event broadcasts

* Fix timezones in tests

* Cleanup

* More cleanup

* Plaid item UI broadcasts for sync

* Fix account ID namespace conflict

* Sync broadcasters

* Smoother account sync refreshes

* Remove test sync delay
2025-05-15 10:19:56 -04:00
Zach Gollwitzer
c88fe2e3b2 Feature: Add "amount type" configuration column for CSV imports (#1947)
* Rough draft

* Schema conflict update

* Implement signage

* Update system tests

* Lint fixes
2025-04-18 10:48:10 -04:00
Zach Gollwitzer
e657c40d19 Account:: namespace simplifications and cleanup (#2110)
* Flatten Holding model

* Flatten balance model

* Entries domain renames

* Fix valuations reference

* Fix trades stream

* Fix brakeman warnings

* Fix tests

* Replace existing entryable type references in DB
2025-04-14 11:40:34 -04:00
Zach Gollwitzer
0544089710 Account-level import configuration templates (#1946)
* Account-level import configuration templates

* Default import to family's preferred date format
2025-03-04 13:10:01 -05:00
Zach Gollwitzer
c5da8ea550 Allow CSV imports to be configured with single or multi-account mode (#1943)
* Allow CSV imports to be configured to a single account or multiple accounts

* Initialize import directly from accounts page

* Fix brakeman warnings

* Fix schema

* Fix Synth check
2025-03-03 12:47:30 -05:00
Zach Gollwitzer
4d0df9b950 Escape quotations in CSV imports properly (#1929)
* Parse quotes in imports

* Update invalid CSV for test
2025-02-28 12:21:07 -05:00
David Anyatonwu
32ef6ca154 Add exchange and currency fields to trade imports (#1822)
* Add exchange and currency fields to trade imports

* Add exchange_operating_mic support for trade imports - Added required columns and updated models

* refactor: remove exchange and currency columns

* fix: consolidate import schema and remove redundant columns

* feat: Enhance trade import with exchange_operating_mic support

* Revert changes to existing migration

* Simplify migration to use change method

* Restore previously deleted migration

* Remove unused import_col_labels method

* Update schema.rb after running migrations

* Update trade_import.rb and fix schema.rb with db:migrate:reset

* fix: improve trade import security creation

---------

Signed-off-by: David Anyatonwu <51977119+onyedikachi-david@users.noreply.github.com>
2025-02-24 10:00:24 -05:00
Zach Gollwitzer
8539ac7dec Fix import configuration failures (#1876) 2025-02-21 08:58:06 -05:00
Daniel Esteves
077694bbde feat(import): add currency and number format support for CSV imports (#1819)
* feat(import): add currency and number format support for CSV imports

* feat(import): add currency and format for mint and trade

* fix(imports): remove currency field in favor of currency csv col

* fix(imports): remove validate column from import model

* test(import): add some tests for imports

* test(import): add some tests for generate_rows_from_csv

* fix: change method name for import model

* fix: change before validation

---------

Co-authored-by: danestves <danestves@users.noreply.github.com>
2025-02-10 15:31:28 -05:00
Zach Gollwitzer
536c82f2aa Feature: Add the ability to "revert" a CSV import (#1814)
* Allow reverting imports

* Fix tests

* Add currency column to all imports

* Don't auto-enrich demo account
2025-02-07 15:36:05 -05:00
Zach Gollwitzer
79ca7e2039 Preserve negative sign on raw CSV values 2024-10-10 18:57:00 -04:00
Zach Gollwitzer
a20809eee3 When unassigned accounts in CSV import, always allow new account creation 2024-10-10 15:51:36 -04:00
Zach Gollwitzer
cd9f20747c Allow inline account creation when importing CSV (#1291)
* Allow inline account creation when importing CSV

* Sanitize numeric inputs for CSV

* CSV import date validation

* Lint fix
2024-10-10 15:14:38 -04:00
Zach Gollwitzer
398b246965 CSV Imports Overhaul (Transactions, Trades, Accounts, and Mint import support) (#1209)
* Remove stale 1.0 import logic and model

* Fresh start

* Checkpoint before removing nav

* First working prototype

* Add trade, account, and mint import flows

* Basic working version with tests

* System tests for each import type

* Clean up mappings flow

* Clean up PR, refactor stale code, tests

* Add back row validations

* Row validations

* Fix import job test

* Fix import navigation

* Fix mint import configuration form

* Currency preset for new accounts
2024-10-01 10:47:59 -04:00
Zach Gollwitzer
949d3d80fa Support multi-currency transfers (#1175) 2024-09-13 11:45:19 -04:00
Pedro Carmona
0c1ff00c1e Refactor: Allow other import files (#1099)
* Rename stimulus controller

* feature: rename raw_csv_str to raw_file_str
2024-08-19 09:25:07 -04:00
Alexander Schrot
4527482aa2 Add support for different column separator in csv import logic (#1096)
* add col_sep to import model

* add validation for col_sep column

* add col_sep option to csv import model

* make use of col_sep option in import model

* add column separator field to new/edit action of an import

* add col_sep parameter to create/update action

* fix spacing between fields

Co-authored-by: Zach Gollwitzer <zach.gollwitzer@gmail.com>
Signed-off-by: Alexander Schrot <alexander@axs-labs.com>

---------

Signed-off-by: Alexander Schrot <alexander@axs-labs.com>
Co-authored-by: Zach Gollwitzer <zach.gollwitzer@gmail.com>
2024-08-16 14:00:16 -04:00
Pedro Carmona
14c4b9e93c Refactor: Use native error i18n lookup (#1076) 2024-08-12 20:38:58 -04:00
Zach Gollwitzer
c3314e62d1 Account::Entry Delegated Type (namespace updates part 7) (#923)
* Initial entryable models

* Update transfer and tests

* Update transaction controllers and tests

* Update sync process to use new entries model

* Get dashboard working again

* Update transfers, imports, and accounts to use Account::Entry

* Update system tests

* Consolidate transaction management into entries controller

* Add permitted partial key helper

* Move account transactions list to entries controller

* Delegate transaction entries search

* Move transfer relation to entry

* Update bulk transaction management flows to use entries

* Remove test code

* Test fix attempt

* Update demo data script

* Consolidate remaining transaction partials to entries

* Consolidate valuations controller to entries controller

* Lint fix

* Remove unused files, additional cleanup

* Add back valuation creation

* Make migrations fully reversible

* Stale routes cleanup

* Migrations reversible fix

* Move types to entryable concern

* Fix search when no entries found

* Remove more unused code
2024-07-01 10:49:43 -04:00
Igor Carvalho
094128fef1 Fix issue #861: Correct header selection logic in get_selected_header_for_field method (#918)
The get_selected_header_for_field method was incorrectly using the entire field object instead of the field key to dig into the column_mappings hash. This caused an error when trying to retrieve the selected header for a field.
2024-06-24 10:31:21 -04:00
Zach Gollwitzer
2681dd96b1 Move categories to top-level namespace (#894) 2024-06-20 08:15:09 -04:00
Zach Gollwitzer
8372e26864 Allow optional import fields (#865) 2024-06-11 18:46:44 -04:00
Zach Gollwitzer
de53a50e45 Sync account after transaction import (#820) 2024-05-30 22:06:32 -04:00
Jakub Kottnauer
6e59fdb369 Add tag preview when importing (#800) 2024-05-24 10:39:24 -04:00
Zach Gollwitzer
457247da8e Create tagging system (#792)
* Repro

* Fix

* Update signage

* Create tagging system

* Add tags to transaction imports

* Build tagging UI

* Cleanup

* More cleanup
2024-05-23 08:09:33 -04:00
Zach Gollwitzer
41c991384a Fix duplicate category creation on import (#791)
* Repro

* Fix

* Update signage
2024-05-22 10:02:03 -04:00
Jakub Kottnauer
77f166a5f8 Ignore empty categories while importing (#789)
* Ignore empty categories while importing

* Review fixes
2024-05-22 08:12:56 -04:00
Jakub Kottnauer
32748b0632 Fix import crash with empty transaction name (#783) 2024-05-20 17:21:40 -04:00
Jakub Kottnauer
34811d8fd8 Fix currency when importing to foreign accounts (#762) 2024-05-20 09:55:45 -04:00
Zach Gollwitzer
45ae4a9737 CSV Transaction Imports (#708)
Introduces a basic CSV import module for bulk-importing account transactions.

Changes include:

- User can load a CSV
- User can configure the column mappings for a CSV
- Imported CSV shows invalid cells
- User can clean up their data directly in the UI
- User can see a preview of the import rows and confirm import
- Layout refactor + Import nav stepper
- System test stability improvements
2024-05-17 09:09:32 -04:00