Commit Graph

53 Commits

Author SHA1 Message Date
soky srm
76dc91377c Merchants improvements (#594)
* FIX logos

* Implement merchant mods

* FIX confirm issue

* FIX linter

* Add recently seen merchants to re-add if needed

* Update merge.html.erb

* FIX do security check

* Add error handling for update failures.
2026-01-09 19:38:04 +01:00
soky srm
e37c03d1d4 Implement Run all rules (#582) 2026-01-08 15:20:14 +01:00
Ethan
3b4ab735b0 Add (beta) CoinStats Crypto Wallet Integration with Balance and Transaction Syncing (#512)
* Feat(CoinStats): Scaffold implementation, not yet functional

* Feat(CoinStats): Implement crypto wallet balance and transactions

* Feat(CoinStats): Add tests, Minor improvements

* Feat(CoinStats): Utilize bulk fetch API endpoints

* Feat(CoinStats): Migrate strings to i8n

* Feat(CoinStats): Fix error handling in wallet link modal

* Feat(CoinStats): Implement hourly provider sync job

* Feat(CoinStats): Generate docstrings

* Fix(CoinStats): Validate API Key on provider update

* Fix(Providers): Safely handle race condition in merchance creation

* Fix(CoinStats): Don't catch system signals in account processor

* Fix(CoinStats): Preload before iterating accounts

* Fix(CoinStats): Add no opener / referrer to API dashboard link

* Fix(CoinStats): Use strict matching for symbols

* Fix(CoinStats): Remove dead code in transactions importer

* Fix(CoinStats): Avoid transaction fallback ID collisions

* Fix(CoinStats): Improve Blockchains fetch error handling

* Fix(CoinStats): Enforce NOT NULL constraint for API Key schema

* Fix(CoinStats): Migrate sync status strings to i8n

* Fix(CoinStats): Use class name rather than hardcoded string

* Fix(CoinStats): Use account currency rather than hardcoded USD

* Fix(CoinStats): Migrate from standalone to Provider class

* Fix(CoinStats): Fix test failures due to string changes
2026-01-07 15:59:04 +01:00
LPW
c12c585a0e Harden SimpleFin sync: retries, safer imports, manual relinking, and data-quality reconciliation (#544)
* Add tests and enhance logic for SimpleFin account synchronization and reconciliation

- Added retry logic with exponential backoff for network errors in `Provider::Simplefin`.
- Introduced tests to verify retry functionality and error handling for rate-limit, server errors, and stale data.
- Updated `SimplefinItem` to detect stale sync status and reconciliation issues.
- Enhanced UI to display stale sync warnings and data integrity notices.
- Improved SimpleFin account matching during updates with multi-tier strategy (ID, fingerprint, fuzzy match).
- Added transaction reconciliation logic to detect data gaps, transaction count drops, and duplicate transaction IDs.

* Introduce `SimplefinConnectionUpdateJob` for asynchronous SimpleFin connection updates

- Moved SimpleFin connection update logic to `SimplefinConnectionUpdateJob` to improve response times by offloading network retries, data fetching, and reconciliation tasks.
- Enhanced SimpleFin account matching with a multi-tier strategy (ID, fingerprint, fuzzy name match).
- Added retry logic and bounded latency for token claim requests in `Provider::Simplefin`.
- Updated tests to cover the new job flow and ensure correct account reconciliation during updates.

* Remove unused SimpleFin account matching logic and improve error handling in `SimplefinConnectionUpdateJob`

- Deleted the multi-tier account matching logic from `SimplefinItemsController` as it is no longer used.
- Enhanced error handling in `SimplefinConnectionUpdateJob` to gracefully handle import failures, ensuring orphaned items can be manually resolved.
- Updated job flow to conditionally set item status based on the success of import operations.

* Fix SimpleFin sync: check both legacy FK and AccountProvider for linked accounts

* Add crypto, checking, savings, and cash account detection; refine subtype selection and linking

- Enhanced `Simplefin::AccountTypeMapper` to include detection for crypto, checking, savings, and standalone cash accounts.
- Improved subtype selection UI with validation and warning indicators for missing selections.
- Updated SimpleFin account linking to handle both legacy FK and `AccountProvider` associations consistently.
- Refined job flow and importer logic for better handling of linked accounts and subtype inference.

* Improve `SimplefinConnectionUpdateJob` and holdings processing logic

- Fixed race condition in `SimplefinConnectionUpdateJob` by moving `destroy_later` calls outside of transactions.
- Updated fuzzy name match logic to use Levenshtein distance for better accuracy.
- Enhanced synthetic ticker generation in holdings processor with hash suffix for uniqueness.

* Refine SimpleFin entry processing logic and ensure `extra` data persistence

- Simplified pending flag determination to rely solely on provider-supplied values.
- Fixed potential stale values in `extra` by ensuring deep merge overwrite with `entry.transaction.save!`.

* Replace hardcoded fallback transaction description with localized string

* Refine pending flag logic in SimpleFin processor tests

- Adjust test to prevent falsely inferring pending status from missing posted dates.
- Ensure provider explicitly sets pending flag for transactions.

* Add `has_many :holdings` association to `AccountProvider` with `dependent: :nullify`

---------

Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
2026-01-05 22:11:47 +01:00
Juan José Mata
1af0f91779 Rescue me from errors
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
2025-12-19 23:18:05 +01:00
soky srm
0300bf9c24 Recurring fixes (#454)
* Fix record violation

and add toggle for recurring feature

* Run only once per sync cycle ( 30 sec )

* FIX params passing

* Add collapsible to recurring section

* FIX preferences error catch
2025-12-17 16:03:05 +01:00
LPW
4d3d9d10df Address remaining CodeRabbit comments from PR #267 #351 (#451)
* Address remaining CodeRabbit comments from PR #267

This commit addresses the remaining unresolved code review comments:

1. Fix down migration in drop_was_merged_from_transactions.rb
   - Add null: false, default: false constraints to match original column
   - Ensures proper rollback compatibility

2. Fix bare rescue in maps_helper.rb compute_duplicate_only_flag
   - Replace bare rescue with rescue StandardError => e
   - Add proper logging for debugging
   - Follows Ruby best practices by being explicit about exception handling

These changes improve code quality and follow Rails/Ruby best practices.

* Refactor `SimplefinItemsController` and add tests for balances sync and account relinking behavior

- Replaced direct sync execution with `SyncJob` for asynchronous handling of balances sync.
- Updated account relinking logic to prevent disabling accounts with other active provider links.
- Removed unused `compute_relink_candidates` method.
- Added tests to verify `balances` action enqueues `SyncJob` and relinking respects account-provider relationships.

* Refactor balances sync to use runtime-only `balances_only` flag

- Replaced persistent `sync_stats` usage with runtime `balances_only?` predicate via `define_singleton_method`.
- Updated `SimplefinItemsController` `balances` action to pass `balances_only` flag to `SyncJob`.
- Enhanced `SyncJob` to attach transient `balances_only?` flag for execution.
- Adjusted `SimplefinItem::Syncer` logic to rely on the runtime `balances_only?` method.
- Updated controller tests to validate runtime flag usage in `SyncJob`.

---------

Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
2025-12-15 09:47:16 +01:00
Juan José Mata
bf90cad9a0 Add Recent Runs visibility for rule executions (#376)
* Add Recent Runs visibility for rule executions

Adds a comprehensive tracking system for rule execution history with the following features:

- Creates RuleRun model to track execution metadata:
  * Date/time of execution
  * Execution type (manual/scheduled)
  * Success/failure status
  * Rule reference
  * Transaction counts (processed and modified)
  * Error messages for failed runs

- Updates RuleJob to automatically record execution results:
  * Captures transaction processing statistics
  * Handles success/failure states
  * Stores error details for debugging

- Adds "Recent Runs" section to rules index page:
  * Paginated display (20 runs per page)
  * Columnar layout similar to LLM usage page
  * Visual status indicators (success/failed badges)
  * Error tooltips for failed runs
  * Responsive design with design system tokens

- Includes i18n translations for all user-facing strings

This provides users with visibility into rule execution history, making it easier to debug issues and monitor rule performance.

* Update schema.rb with rule_runs table definition

* Linter noise

* Separate transaction counts into Queued, Processed, and Modified

Previously, the code eagerly reported transactions as "processed" when they
were only queued for processing. This commit separates the counts into three
distinct metrics:

- Transactions Queued: Count of transactions matching the rule's filter
  conditions before any processing begins
- Transactions Processed: Count of transactions that were actually processed
  and modified by the rule actions
- Transactions Modified: Count of transactions that had their values changed
  (currently same as Processed, but allows for future differentiation)

Changes:
- Add transactions_queued column to rule_runs table
- Update RuleJob to track all three counts separately
- Update action executors to return count of modified transactions
- Update Rule#apply to aggregate modification counts from actions
- Add transactions_queued label to locales
- Update Recent Runs view to display new column
- Add validation for transactions_queued in RuleRun model

The tracking now correctly reports:
1. How many transactions matched the filter (queued)
2. How many were actually modified (processed/modified)
3. Distinguishes between matching and modifying transactions

* Add Pending status to track async rule execution progress

Introduced a new "pending" status for rule runs to properly track async
AI operations. The system now:

- Tracks pending async jobs with a counter that decrements as jobs complete
- Updates transactions_modified incrementally as each job finishes
- Only counts transactions that were actually modified (not just queued)
- Displays pending status with yellow badge in the UI
- Automatically transitions from pending to success when all jobs complete

This provides better visibility into long-running AI categorization and
merchant detection operations, showing real-time progress as Sidekiq
processes the batches.

* Fix migration version to 7.2 as per project standards

* Consolidate rule_runs migrations into single migration file

Merged three separate migrations (create, add_transactions_queued,
add_pending_jobs_count) into a single CreateRuleRuns migration.
This provides better clarity and maintains a clean migration history.

Changes:
- Updated CreateRuleRuns migration to include all columns upfront
- Removed redundant add_column migrations
- Updated schema version to 2025_11_24_000000

* Linter and test fixes

* Space optimization

* LLM l10n is better than no l10n

* Fix implementation for tags/AI rules

* Fix tests

* Use batch_size

* Consider jobs "unknown" status sometimes

* Rabbit suggestion

* Rescue block for RuleRun.create!

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-12-07 16:30:02 +01:00
LPW
a91a4397e9 Simplefin liabilities recording fix (#410)
* Add tests and logic for Simplefin account balance normalization

- Introduced `SimplefinAccountProcessorTest` to verify balance normalization logic.
- Updated `SimplefinAccount::Processor` to invert negative balances for liability accounts (credit cards and loans) while keeping asset balances unchanged.
- Added comments to clarify balance conventions and sign normalization rules.

* Refactor balances-only sync logic and improve tests for edge cases

- Updated `SimplefinItem::Importer` and `SimplefinItem::Syncer` to ensure `last_synced_at` remains nil during balances-only runs, preserving chunked-history behavior for full syncs.
- Introduced additional comments to clarify balances-only implications and syncing logic.
- Added test case in `SimplefinAccountProcessorTest` to verify correct handling of overpayment for credit card liabilities.
- Refined balance normalization in `SimplefinAccount::Processor` to always invert liability balances for consistency.

---------

Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
2025-12-03 18:40:37 +01:00
Josh Waldrep
195dd18a52 Add tests and refactor for Simplefin holdings import and processing
- Introduced tests for importer post-import logic and `SimplefinHoldingsApplyJob`.
- Refactored `ProviderImportAdapter` to improve holding resolution strategy.
- Added handling of investment and crypto holdings in importer with debounce logic for job enqueuing.
- Updated rake task to use `SimplefinHoldingsApplyJob` for holding materialization.
2025-11-22 14:41:21 -05:00
Juan José Mata
d6cbf300c7 Add scheduled job to sync all accounts every 24 hours (#330)
* Add scheduled job to sync all accounts every 30 minutes

Signed-off-by: Nikhil Badyal <nikhill773384@gmail.com>

* Change job queue from default to scheduled

Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>

* Flatten job into single directory

* Every 30 minutes is a bit much and will trigger Sentry warnings

* Locking and logging improvements

* Add support for extra Sidekiq goodies

---------

Signed-off-by: Nikhil Badyal <nikhill773384@gmail.com>
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Nikhil Badyal <nikhill773384@gmail.com>
2025-11-20 20:58:14 +01:00
LPW
61eb611529 Simplefin enhancements v2 (#267)
* SimpleFin: metadata + merge fixes; holdings (incl. crypto) + Day Change; Sync Summary; ops rakes; lint

# Conflicts:
#	db/schema.rb

# Conflicts:
#	app/controllers/simplefin_items_controller.rb

* fix testing

* fix linting

* xfix linting x2

* Review PR #267 on we-promise/sure (SimpleFin enhancements v2). Address all 15 actionable CodeRabbit comments: Add UUID validations in rakes (e.g., simplefin_unlink), swap Ruby pattern matching/loops for efficient DB queries (e.g., where LOWER(name) LIKE ?), generate docstrings for low-coverage areas (31%), consolidate routes for simplefin_items, move view logic to helpers (e.g., format_transaction_extra), strengthen tests with exact assertions/fixtures for dedup/relink failures. Also, check for overlaps with merged #262 (merchants fix): Ensure merchant creation in simplefin_entry/processor.rb aligns with new payee-based flow and MD5 IDs; add tests for edge cases like empty payees or over-merging pendings. Prioritize security (PII redaction in logs, no hardcoded secrets).

* SimpleFin: address CodeRabbit comments (batch 1)

- Consolidate simplefin_items routes under a single resources block; keep URLs stable
- Replace inline JS with Stimulus auto-relink controller; auto-load relink modal via global modal frame
- Improve a11y in relink modal by wrapping rows in labels
- Harden unlink rake: default dry_run=true, UUID validation, redact PII in outputs, clearer errors
- Backfill rake: default dry_run=true, UUID validation; groundwork for per-SFA counters
- Fix-was-merged rake: default dry_run=true, UUID validation; clearer outputs
- Idempotent transfer auto-match (find_or_create_by! + RecordNotUnique rescue)
- Extract SimpleFin error tooltip assembly into helper and use it in view

RuboCop: maintain 2-space indentation, spaces inside array brackets, spaces after commas, and no redundant returns

* Linter noise

* removed filed commited by mistake.

* manual relink flow and tighten composite matching

* enforce manual relink UI; fix adapter keywords; guarantee extra.simplefin hash

* refactor(simplefin): extract relink service; enforce manual relink UI; tighten composite match; migration 7.2

* add provider date parser; refactor rake; move view queries; partial resilience

* run balances-only import in background job. make update flow enqueue balances-only job

* persists across all update redirects and initialize
used_manual_ids to prevent NameError in relink candidate computation.

* SimpleFin: metadata + merge fixes; holdings (incl. crypto) + Day Change; Sync Summary; ops rakes; lint

* Fixed failed test after rebase.

* scan_ruby fix

* Calming the rabbit:
Fix AccountProvider linking when accounts change
Drop the legacy unique index instead of duplicating it
Fix dynamic constant assignment
Use fixtures consistently; avoid rescue for control flow.
Replace bare rescue with explicit exception class.
Move business logic out of the view.
Critical: Transaction boundary excludes recompute phase, risking data loss.
Inconsistency between documentation and implementation for zero-error case.
Refactor to use the compute_unlinked_count helper for consistency.
Fix cleanup task default: it deletes by default.
Move sync stats computation to controller to avoid N+1 queries.
Consolidate duplicate sync query.
Clarify the intent of setting flash notice on the error path.
Fix Date/Time comparison in should_be_inactive?.
Move stats retrieval logic to controller.
Remove duplicate Sync summary section.
Remove the unnecessary sleep statement; use Capybara's built-in waiting.
Add label wrappers for accessibility and consistency.

* FIX SimpleFIN new account modal

Now new account properly loads as a Modal, instead of new page.
Fixes also form showing dashboard instead of settings page.

* Remove SimpleFin legacy UI components, migrate schema, and refine linking behavior.

# Conflicts:
#	app/helpers/settings_helper.rb

* Extract SimpleFin-related logic to `prepare_show_context` helper and refactor for consistency. Adjust conditional checks and ensure controller variables are properly initialized.

* Remove unused SimpleFin maps from prepare_show_context; select IDs to avoid N+1
Replace Tailwind bg-green-500 with semantic bg-success in _simplefin_panel/_provider_form
Add f.label :setup_token in simplefin_items/new for a11y
Remove duplicate require in AccountsControllerSimplefinCtaTest

* Remove unnecessary blank lines

* Reduce unnecessary changes

This reduces the diff against main

* Simplefin Account Setup: Display in modal

This fixes an issue with the `X` dismiss button in the top right corner

* Removed unnecessary comment.

* removed unnecessary function.

* fixed broken links

* Removed unnecessary file

* changed to database query

* set to use UTC and gaurd against null

* set dry_run=true

* Fixed comment

* Changed to use a database-level query

* matched test name to test behavior.

* Eliminate code duplication and Time.zone dependency

* make final summary surface failures

* lint fix

* Revised timezone comment. better handle missing selectors.

* sanitized LIKE wildcards

* Fixed SimpleFin import to avoid “Currency can’t be blank” validation failures when providers return an empty currency string.

* Added helper methods for admin and self-hosted checks

* Specify exception types in rescue clauses.

* Refined logic to determine transaction dates for credit accounts.

* Refined stats calculation for `total_accounts` to track the maximum unique accounts per run instead of accumulating totals.

* Moved `unlink_all!` logic to `SimplefinItem::Unlinking` concern and deprecated `SimplefinItem::Unlinker`. Updated related references.

* Refined legacy unlinking logic, improved `current_holdings` formatting, and added ENV-based overrides for self-hosted checks.

* Enhanced `unlink_all!` with explicit error handling, improved transaction safety, and refined ENV-based self-hosted checks. Adjusted exception types and cleaned up private method handling.

* Improved currency assignment logic by adding fallback to `current_account` and `family` currencies.

* Enhanced error tracking during SimpleFin account imports by adding categorized error buckets, limiting stored errors to the last 5, and improving `stats` calculations.

* typo fix

* Didn't realize rabbit was still mad...
Refactored SimpleFin error handling and CTA logic: centralized duplicate detection and relink visibility into controller, improved task counters, adjusted redirect notices, and fixed form indexing.

* Dang rabbit never stops... Centralized SimpleFin maps logic into `MapsHelper` concern and integrated it into relevant controllers and rake tasks. Optimized queries, reduced redundancy, and improved unlinked counts and manual account checks with batch processing. Adjusted task arguments for clarity.

* Persistent rabbit. Optimized SimpleFin maps logic by implementing batch queries for manual account and unlinked count checks, reducing N+1 issues. Improved clarity of rake task argument descriptions and error messages for better usability.

* Lost a commit somehow, resolved here. Refactored transaction extra details logic by introducing `build_transaction_extra_details` helper to improve clarity, reusability, and reduce view complexity. Enhanced rake tasks with strict dry-run validation and better error handling. Updated schema to allow nullable `merchant_id` and added conditional unique indexes for recurring transactions.

* Refactored sensitive data redaction in `simplefin_unlink` task for recursive handling, optimized SQL sanitization in `simplefin_holdings_backfill`, improved error handling in `transactions_helper`, and streamlined day change calculation logic in `Holding` model.

* Lint fix

* Removed per PR comments.

* Also removing per PR comment.

* git commit -m "SimpleFIN polish: preserve #manual-accounts wrapper, unify \"manual\" scope, and correct unlinked counts
- Preserve #manual-accounts wrapper: switch non-empty updates to turbo_stream.update and background broadcast_update_to; keep empty-path replace to render <div id=\"manual-accounts\"></div>
- Unify definition of manual accounts via Account.visible_manual (visible + legacy-nil + no AccountProvider); reuse in controllers, jobs, and helper
- Correct setup/unlinked counts: SimplefinItem::Syncer#finalize_setup_counts and maps now consider AccountProvider links (legacy account AND provider must be absent)
Deleted:
- app/models/simplefin_item/relink_service.rb
- app/controllers/concerns/simplefin_items/relink_helpers.rb
- app/javascript/controllers/auto_relink_controller.js
- app/views/simplefin_items/_relink_modal.html.erb
- app/views/simplefin_items/manual_relink.html.erb
- app/views/simplefin_items/relink.html.erb
- test/services/simplefin_item/relink_service_test.rb
Refs: PR #318 unified link/unlink; PR #267 SimpleFIN; follow-up to fix wrapper ID loss and counting drift."

* Extend unlinked account check to include "Investment" type

* set SimpleFIN item for `balances`, remove redundant unpacking, and improve holdings task error

* SimpleFIN: add `errors` action + modal; do not reintroduce legacy relink actions; removed dead helper

* FIX simpleFIN linking

* Add delay back, tests benefit from it

* Put cache back in

* Remove empty `rake` task

* Small spelling fixes.

---------

Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: sokie <sokysrm@gmail.com>
Co-authored-by: Dylan Corrales <deathcamel58@gmail.com>
2025-11-17 21:51:37 +01:00
Juan José Mata
dfd467ccb5 Add "Reset account" followed by sample data preload (#163)
* Add reset with sample data option on profile settings

* No need for "member" user in preload

* Cleanup/shorten copy
2025-09-25 11:43:23 +02:00
Zach Gollwitzer
0329a5f211 Data exports (#2517)
* Import / export UI

* Data exports

* Lint fixes, brakeman update
2025-07-24 10:50:05 -04:00
Zach Gollwitzer
1aae00f586 perf(transactions): add kind to Transaction model and remove expensive Transfer joins in aggregations (#2388)
* add kind to transaction model

* Basic transfer creator

* Fix method naming conflict

* Creator form pattern

* Remove stale methods

* Tweak migration

* Remove BaseQuery, write entire query in each class for clarity

* Query optimizations

* Remove unused exchange rate query lines

* Remove temporary cache-warming strategy

* Fix test

* Update transaction search

* Decouple transactions endpoint from IncomeStatement

* Clean up transactions controller

* Update cursor rules

* Cleanup comments, logic in search

* Fix totals logic on transactions view

* Fix pagination

* Optimize search totals query

* Default to last 30 days on transactions page if no filters

* Decouple transactions list from transfer details

* Revert transfer route

* Migration reset

* Bundle update

* Fix matching logic, tests

* Remove unused code
2025-06-20 13:31:58 -04:00
Zach Gollwitzer
854a21993a Handle ITEM_NOT_FOUND errors on Plaid deletions 2025-05-25 11:52:29 -04:00
Zach Gollwitzer
e4ee06c9f6 Security resolver and health checker (#2281)
* Setup health check

* Security health checker cron

* Use resolver throughout codebase

* Use resolver for trade builder

* Add security health checks to schedule

* Handle no provider

* Lint fixes
2025-05-22 12:43:24 -04:00
Zach Gollwitzer
f82f77466a Fix Sentry context for security details exception 2025-05-18 10:52:35 -04:00
Zach Gollwitzer
9f13b5bb83 Handle stale syncs (#2257)
* Handle stale syncs

* Use `visible` sync logic in sidebar groups
2025-05-17 18:28:21 -04:00
Zach Gollwitzer
10f255a9a9 Clarify backend data pipeline naming concepts (importers, processors, materializers, calculators, and syncers) (#2255)
* Rename MarketDataSyncer to MarketDataImporter

* Materializers

* Importers

* More reference replacements
2025-05-17 16:37:16 -04:00
Zach Gollwitzer
6dc1d22672 Market data sync refinements (#2252)
* Exchange rate syncer implementation

* Security price syncer

* Fix issues with provider API

* Add back prod schedule

* Add back price and exchange rate syncs to account syncs

* Remove unused stock_exchanges table
2025-05-16 14:17:56 -04:00
Zach Gollwitzer
10dd9e061a Improve account sync performance, handle concurrent market data syncing (#2236)
* PlaidConnectable concern

* Remove bad abstraction

* Put sync implementations in own concerns

* Sync strategies

* Move sync orchestration to Sync class

* Clean up sync class, add state machine

* Basic market data sync cron

* Fix price sync

* Improve sync window column names, add timestamps

* 30 day syncs by default

* Clean up market data methods

* Report high duplicate sync counts to Sentry

* Add sync states throughout app

* account tab session

* Persistent account tab selections

* Remove manual sleep

* Add migration to clear stale syncs on self hosted apps

* Tweak sync states

* Sync completion event broadcasts

* Fix timezones in tests

* Cleanup

* More cleanup

* Plaid item UI broadcasts for sync

* Fix account ID namespace conflict

* Sync broadcasters

* Smoother account sync refreshes

* Remove test sync delay
2025-05-15 10:19:56 -04:00
Zach Gollwitzer
a51c4d2cba New onboarding, trials, Stripe integration (#2185)
* New onboarding, trials, Stripe integration

* Fix tests

* Lint fixes

* Fix subscription endpoints
2025-05-01 16:47:14 -04:00
Zach Gollwitzer
297a695d0f Transaction rules engine V1 (#1900)
* Domain model sketch

* Scaffold out rules domain

* Migrations

* Remove existing data enrichment for clean slate

* Sketch out business logic and basic tests

* Simplify rule scope building and action executions

* Get generator working again

* Basic implementation + tests

* Remove manual merchant management (rules will replace)

* Revert "Remove manual merchant management (rules will replace)"

This reverts commit 83dcbd9ff0aa7bbee211796b71aa48b71df5e57e.

* Family and Provider merchants model

* Fix brakeman warnings

* Fix notification loader

* Update notification position

* Add Rule action and condition registries

* Rule form with compound conditions and tests

* Split out notification types, add CTA type

* Rules form builder and Stimulus controller

* Clean up rule registry domain

* Clean up rules stimulus controller

* CTA message for rule when user changes transaction category

* Fix tests

* Lint updates

* Centralize notifications in Notifiable concern

* Implement category rule prompts with auto backoff and option to disable

* Fix layout bug caused by merge conflict

* Initialize rule with correct action for category CTA

* Add rule deletions, get rules working

* Complete dynamic rule form, split Stimulus controllers by resource

* Fix failing tests

* Change test password to avoid chromium conflicts

* Update integration tests

* Centralize all test password references

* Add re-apply rule action

* Rule confirm modal

* Run migrations

* Trigger rule notification after inline category updates

* Clean up rule styles

* Basic attribute locking for rules

* Apply attribute locks on user edits

* Log data enrichments, only apply rules to unlocked attributes

* Fix merge errors

* Additional merge conflict fixes

* Form UI improvements, ignore attribute locks on manual rule application

* Batch AI auto-categorization of transactions

* Auto merchant detection, ai enrichment in batches

* Fix Plaid merchant assignments

* Plaid category matching

* Cleanup 1

* Test cleanup

* Remove stale route

* Fix desktop chat UI issues

* Fix mobile nav styling issues
2025-04-18 11:39:58 -04:00
Zach Gollwitzer
8648f11413 Sync hierarchy updates (#2087)
* Add parent sync orchestration

* Pass sync object to dependents
2025-04-11 12:13:46 -04:00
Zach Gollwitzer
1061aacb0f Set AI queue 2025-03-28 13:27:50 -04:00
Zach Gollwitzer
2f6b11c18f Personal finance AI (v1) (#2022)
* AI sidebar

* Add chat and message models with associations

* Implement AI chat functionality with sidebar and messaging system

- Add chat and messages controllers
- Create chat and message views
- Implement chat-related routes
- Add message broadcasting and user interactions
- Update application layout to support chat sidebar
- Enhance user model with initials method

* Refactor AI sidebar with enhanced chat menu and interactions

- Update sidebar layout with dynamic width and improved responsiveness
- Add new chat menu Stimulus controller for toggling between chat and chat list views
- Improve chat list display with recent chats and empty state
- Extract AI avatar to a partial for reusability
- Enhance message display and interaction styling
- Add more contextual buttons and interaction hints

* Improve chat scroll behavior and message styling

- Refactor chat scroll functionality with Stimulus controller
- Optimize message scrolling in chat views
- Update message styling for better visual hierarchy
- Enhance chat container layout with flex and auto-scroll
- Simplify message rendering across different chat views

* Extract AI avatar to a shared partial for consistent styling

- Refactor AI avatar rendering across chat views
- Replace hardcoded avatar markup with a reusable partial
- Simplify avatar display in chats and messages views

* Update sidebar controller to handle right panel width dynamically

- Add conditional width class for right sidebar panel
- Ensure consistent sidebar toggle behavior for both left and right panels
- Use specific width class for right panel (w-[375px])

* Refactor chat form and AI greeting with flexible partials

- Extract message form to a reusable partial with dynamic context support
- Create flexible AI greeting partial for consistent welcome messages
- Simplify chat and sidebar views by leveraging new partials
- Add support for different form scenarios (chat, new chat, sidebar)
- Improve code modularity and reduce duplication

* Add chat clearing functionality with dynamic menu options

- Implement clear chat action in ChatsController
- Add clear chat route to support clearing messages
- Update AI sidebar with dropdown menu for chat actions
- Preserve system message when clearing chat
- Enhance chat interaction with new menu options

* Add frontmatter to project structure documentation

- Create initial frontmatter for structure.mdc file
- Include description and configuration options
- Prepare for potential dynamic documentation rendering

* Update general project rules with additional guidelines

- Add rule for using `Current.family` instead of `current_family`
- Include new guidelines for testing, API routes, and solution approach
- Expand project-specific rules for more consistent development practices

* Add OpenAI gem and AI-friendly data representations

- Add `ruby-openai` gem for AI integration
- Implement `to_ai_readable_hash` methods in BalanceSheet and IncomeStatement
- Include Promptable module in both models
- Add savings rate calculation method in IncomeStatement
- Prepare financial models for AI-powered insights and interactions

* Enhance AI Financial Assistant with Advanced Querying and Debugging Capabilities

- Implement comprehensive AI financial query system with function-based interactions
- Add detailed debug logging for AI responses and function calls
- Extend BalanceSheet and IncomeStatement models with AI-friendly methods
- Create robust error handling and fallback mechanisms for AI queries
- Update chat and message views to support debug mode and enhanced rendering
- Add AI query routes and initial test coverage for financial assistant

* Refactor AI sidebar and chat layout with improved structure and comments

- Remove inline AI chat from application layout
- Enhance AI sidebar with more semantic HTML structure
- Add descriptive comments to clarify different sections of chat view
- Improve flex layout and scrolling behavior in chat messages container
- Optimize message rendering with more explicit class names and structure

* Add Markdown rendering support for AI chat messages

- Implement `markdown` helper method in ApplicationHelper using Redcarpet
- Update message view to render AI messages with Markdown formatting
- Add comprehensive Markdown rendering options (tables, code blocks, links)
- Enhance AI Financial Assistant prompt to encourage Markdown usage
- Remove commented Markdown CSS in Tailwind application stylesheet

* Missing comma

* Enhance AI response processing with chat history context

* Improve AI debug logging with payload size limits and internal message flag

* Enhance AI chat interaction with improved thinking indicator and scrolling behavior

* Add AI consent and enable/disable functionality for AI chat

* Upgrade Biome and refactor JavaScript template literals

- Update @biomejs/biome to latest version with caret (^) notation
- Refactor AI query and chat controllers to use template literals
- Standardize npm scripts formatting in package.json

* Add beta testing usage note to AI consent modal

* Update test fixtures and configurations for AI chat functionality

- Add family association to chat fixtures and tests
- Set consistent password digest for test users
- Enable AI for test users
- Add OpenAI access token for test environment
- Update chat and user model tests to include family context

* Simplify data model and get tests passing

* Remove structure.mdc from version control

* Integrate AI chat styles into existing prose pattern

* Match Figma design spec, implement Turbo frames and actions for chats controller

* AI rules refresh

* Consolidate Stimulus controllers, thinking state, controllers, and views

* Naming, domain alignment

* Reset migrations

* Improve data model to support tool calls and message types

* Tool calling tests and fixtures

* Tool call implementation and test

* Get assistant test working again

* Test updates

* Process tool calls within provider

* Chat UI back to working state again

* Remove stale code

* Tests passing

* Update openai class naming to avoid conflicts

* Reconfigure test env

* Rebuild gemfile

* Fix naming conflicts for ChatResponse

* Message styles

* Use OpenAI conversation state management

* Assistant function base implementation

* Add back thinking messages, clean up error handling for chat

* Fix sync error when security price has bad data from provider

* Add balance sheet function to assistant

* Add better function calling error visibility

* Add income statement function

* Simplify and clean up "thinking" interactions with Turbo frames

* Remove stale data definitions from functions

* Ensure VCR fixtures working with latest code

* basic stream implementation

* Get streaming working

* Make AI sidebar wider when left sidebar is collapsed

* Get tests working with streaming responses

* Centralize provider error handling

* Provider data boundaries

---------

Co-authored-by: Josh Pigford <josh@joshpigford.com>
2025-03-28 13:08:22 -04:00
Zach Gollwitzer
19cc63c8f4 Use Redis for ActiveJob and ActionCable (#2004)
* Use Redis for ActiveJob and ActionCable

* Fix alwaysApply setting

* Update queue names and weights

* Tweak weights

* Update job queues

* Update docker setup guide

* Remove deprecated upgrade columns from users table

* Refactor Redis configuration for Sidekiq and caching in production environment

* Add Sidekiq Sentry monitoring

* queue naming fix

* Clean up schema
2025-03-19 12:36:16 -04:00
Zach Gollwitzer
f65b93a352 Data provider simplification, tests, and documentation (#1997)
* Ignore env.test from source control

* Simplification of providers interface

* Synth tests

* Update money to use new find rates method

* Remove unused issues code

* Additional issue feature removals

* Update price data fetching and tests

* Update documentation for providers

* Security test fixes

* Fix self host test

* Update synth usage data access

* Remove AI pr schema changes
2025-03-17 11:54:53 -04:00
Zach Gollwitzer
fa0248056d Show UI warning to user when they need provider data but have not setup Synth yet (#1926)
* Simplify provider concerns

* Update tests

* Add UI warning for missing Synth key if family requires external data
2025-02-28 11:35:10 -05:00
Tony Vincent
8208722247 Feat: Data "reset" button (#1913)
* feat: Allow admins to delete family data

* feat: Allow self-hosting users to delete cached data

* Remove system tests
2025-02-28 07:49:12 -05:00
Josh Pigford
68d7cb5de6 Enhance security information retrieval and handling (#1826)
* Enhance security information retrieval and handling

- Add support for operating MIC codes in security info fetching
- Update security uniqueness validation to handle unknown securities
- Improve security creation and update logic in Plaid investment sync
- Update combobox and view components to handle operating MIC codes
- Add unknown flag for securities with incomplete information

* Update schema.rb

* Refactor the need for mic codes

* Don't fetch prices unless a security has the necessary mic code

* Deduplication

* Lint

* Update Securities and Plaid Investment Sync

- Modify PlaidInvestmentSync to return plaid_security for USD cash
- Add non-null constraint to Securities ticker column
- Update Securities fixture to use exchange_operating_mic instead of exchange_mic

---------

Signed-off-by: Josh Pigford <josh@joshpigford.com>
2025-02-11 10:40:30 -06:00
Zach Gollwitzer
536c82f2aa Feature: Add the ability to "revert" a CSV import (#1814)
* Allow reverting imports

* Fix tests

* Add currency column to all imports

* Don't auto-enrich demo account
2025-02-07 15:36:05 -05:00
Josh Pigford
b84a33c09d Refactor transaction enrichment to support batch processing (#1803)
* Refactor transaction enrichment to support batch processing

- Add method to enrich transactions in batches
- Implement job scheduling for unenriched transactions
- Improve logging and error handling for transaction enrichment

* Re-enable enrichment

* Fix transaction enrichment query to use correct table references

- Update queries to explicitly join and reference account_entries and account_transactions tables
- Remove unnecessary name presence check before enrichment
- Improve query precision for unenriched transaction selection

* Optimize transaction enrichment query joins

- Refactor database joins to use explicit table references
- Improve query performance for unenriched transaction selection
- Ensure correct table aliasing in enrichment methods

* Remove deprecated data enrichment job and method

- Delete EnrichDataJob as it's no longer used
- Remove `enrich_data_later` method from Account model
- Update Account::Syncer to directly call `enrich_data` instead of scheduling a job
2025-02-05 10:34:28 -06:00
Zach Gollwitzer
3140835f28 Adjust queues to prioritize account syncs (#1682) 2025-01-24 13:39:08 -05:00
Zach Gollwitzer
fe199f2357 Add account data enrichment (#1532)
* Add data enrichment

* Make data enrichment optional for self-hosters

* Add categories to data enrichment

* Only update category and merchant if nil

* Fix name overrides

* Lint fixes
2024-12-13 17:22:27 -05:00
Zach Gollwitzer
cbba2ba675 Basic Plaid Integration (#1433)
* Basic plaid data model and linking

* Remove institutions, add plaid items

* Improve schema and Plaid provider

* Add webhook verification sketch

* Webhook verification

* Item accounts and balances sync setup

* Provide test encryption keys

* Fix test

* Only provide encryption keys in prod

* Try defining keys in test env

* Consolidate account sync logic

* Add back plaid account initialization

* Plaid transaction sync

* Sync UI overhaul for Plaid

* Add liability and investment syncing

* Handle investment webhooks and process current day holdings

* Remove logs

* Remove "all" period select for performance

* fix amount calc

* Remove todo comment

* Coming soon for investment historical data

* Document Plaid configuration

* Listen for holding updates
2024-11-15 13:49:37 -05:00
Josh Pigford
31d9d926f7 Fix for certain securities returning incorrect prices 2024-11-07 11:52:16 -06:00
Josh Pigford
e7f09e6f71 Groundwork for security info (#1396)
* Groundwork for security info

* Lint
2024-10-30 18:08:19 -04:00
Josh Pigford
cd91e66618 Initial pass at Synth-based ticker selection (#1392)
* Initial pass at Synth-based ticker selection

* Update _tickers.turbo_stream.erb

* Functional combobox display

* A few cleanup steps

* Linter

* Prevent long strings

* Another step towards functional combobox

* Deprecated files

* Custom Combobox implementation

* Lint

* Test suite fixes

* Lint

* Make direct use of mic codes

* Update splits

* Update trades_test.rb
2024-10-30 09:23:44 -04:00
Josh Pigford
aa3342b0dc Stock imports (#1363)
* Initial pass

* Marketstack data provider

* Marketstack data provider

* Refactor a bit
2024-10-24 16:36:50 -05:00
Zach Gollwitzer
c6bdf49f10 Account::Sync model and test fixture simplifications (#968)
* Add sync model

* Fresh fixtures for sync tests

* Sync tests overhaul

* Fix entry tests

* Complete remaining model test updates

* Update system tests

* Update demo data task

* Add system tests back to PR checks

* More simplifications, add empty family to fixtures for easier testing
2024-07-10 11:22:59 -04:00
Zach Gollwitzer
45ae4a9737 CSV Transaction Imports (#708)
Introduces a basic CSV import module for bulk-importing account transactions.

Changes include:

- User can load a CSV
- User can configure the column mappings for a CSV
- Imported CSV shows invalid cells
- User can clean up their data directly in the UI
- User can see a preview of the import rows and confirm import
- Layout refactor + Import nav stepper
- System test stability improvements
2024-05-17 09:09:32 -04:00
Josh Brown
19ee773d9b Add ability to delete Maybe account (#698)
* Build out user deactivation and purging workflows

* Add i18n translations for user deletion

* Add tests for user deletion

* Fix lint issue
2024-04-30 11:40:31 -04:00
Jakub Kottnauer
ad4de99f1a Add partial account sync support (#653)
* Add partial account sync support

* Fix indentation

* Use historical balance as base when doing partial sync

* Simplify controller crud tests

* Fix linter errors
2024-04-24 15:55:52 -04:00
Zach Gollwitzer
5aca2ff9b6 Add zero-config self hosting on Render (#612)
* v1 of backend implementation for self hosting

* Add docs

* Add upgrades controller

* Add global helpers for self hosting mode

* Add self host settings controller

* Conditionally show self hosting settings

* Environment and config updates

* Complete upgrade prompting flow

* Update config for forked repo

* Move configuration of github provider within class

* Add upgrades cron

* Update deploy button

* Update guides

* Fix render deployer

* Typo

* Enable auto upgrades

* Fix cron

* Make upgrade modes more clear and consistent

* Trigger new available version

* Fix logic for displaying upgrade prompts

* Finish implementation

* Fix regression

* Trigger new version

* Add i18n translations

* trigger new version

* reduce caching time for testing

* Decrease cache for testing

* trigger upgrade

* trigger upgrade

* Only trigger deploy once

* trigger upgrade

* If target is commit, always upgrade if any upgrade is available

* trigger upgrade

* trigger upgrade

* Test release

* Change back to maybe repo for defaults

* Fix lint errors

* Clearer naming

* Fix relative link

* Add abs path

* Relative link

* Update docs
2024-04-13 09:28:45 -04:00
Zach Gollwitzer
110855d077 Multi-Currency Part 2 (#543)
* Support all currencies, handle outside DB

* Remove currencies from seed

* Fix account balance namespace

* Set default currency on authentication

* Cache currency instances

* Implement multi-currency syncs with tests

* Series fallback, passing tests

* Fix conflicts

* Make value group concrete class that works with currency values

* Fix migration conflict

* Update tests to expect multi-currency results

* Update account list to use group method

* Namespace updates

* Fetch unknown exchange rates from API

* Fix date range bug

* Ensure demo data works without external API

* Enforce cascades only at DB level
2024-03-21 13:39:10 -04:00
Josh Pigford
4843cf22c6 Use Synth for exchange rates (#514)
* Switch currency seeding over to Synth

* Switch all exchange rates over to Synth
2024-03-04 10:26:20 -06:00
Zach Gollwitzer
dbf575c02a Basic Account Balance Sync Algorithm (#501)
* Sketch out sync interface

* Add basic account sync algorithm

* Update logic for final balance in series

* Remove start_date concept

* Clean up tests

* Improve clarity of test

* Update app/models/account.rb

Co-authored-by: Rob Zolkos <rob@zolkos.com>
Signed-off-by: Zach Gollwitzer <zach.gollwitzer@gmail.com>

* Update app/models/transaction.rb

Co-authored-by: Rob Zolkos <rob@zolkos.com>
Signed-off-by: Zach Gollwitzer <zach.gollwitzer@gmail.com>

* Update app/models/valuation.rb

Co-authored-by: Rob Zolkos <rob@zolkos.com>
Signed-off-by: Zach Gollwitzer <zach.gollwitzer@gmail.com>

* Re-organize code, simplify job interface

* Consolidate balance calculations

* More cleanup

---------

Signed-off-by: Zach Gollwitzer <zach.gollwitzer@gmail.com>
Co-authored-by: Rob Zolkos <rob@zolkos.com>
2024-02-29 08:32:52 -05:00
Zach Gollwitzer
7d48c01833 Prepare fixture data for account sync tests (#493)
* Rename account balance field for clarity

`original_balance` and `original_currency` may infer that these values are "original" to the account.  In reality, they represent the "current" balance and currency on the account.

* Prepare fixture data for account sync testing

* Update to new field

* Fix conflicts

* Remove local schema change
2024-02-27 12:43:49 -05:00