* Create MCP server endpoint documentation
* Add Assistant Architecture section to AI documentation
* Add Users API documentation for account reset and delete endpoints
* Document Pipelock CI security scanning in contributing guide
* fix: correct scope and error codes in Users API documentation
* Exclude `docs/hosting/ai.md` from Pipelock scan
---------
Co-authored-by: askmanu[bot] <192355599+askmanu[bot]@users.noreply.github.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
* Document admin-only reset auth in OpenAPI docs
The DELETE /api/v1/users/reset endpoint now requires admin role
(ensure_admin). Update the rswag spec to:
- Set default user role to admin so the 200 test passes
- Add a 403 response case for non-admin users with read_write scope
- Clarify the description notes admin requirement
- Add SuccessMessage schema and users paths to openapi.yaml
https://claude.ai/code/session_01Tj8ToLRmVg5HLmHwq9KKDY
* Consolidate duplicate 403 responses for reset endpoint
OpenAPI keys responses by status code, so two 403 blocks caused the
first (insufficient scope) to be silently overwritten by the second
(non-admin). Merge into a single 403 whose description covers both
causes: requires read_write scope and admin role. The test exercises
the read-only key path which hits 403 via scope check.
https://claude.ai/code/session_01Tj8ToLRmVg5HLmHwq9KKDY
* Em-dash out of messages.
* Fix tests
* Fix tests
---------
Co-authored-by: Claude <noreply@anthropic.com>
* Expand AI docs: architecture, MCP, external assistant setup, troubleshooting
- Add architecture overview explaining two independent AI pipelines
(chat assistant vs auto-categorization)
- Document MCP callback endpoint (JSON-RPC 2.0, auth, available tools)
- Add OpenClaw gateway configuration example
- Add Kubernetes network policy guidance (targetPort vs servicePort)
- Add Pipelock notes (mcpToolPolicy, NO_PROXY behavior)
- Add troubleshooting for "Failed to generate response" with external assistant
- Fix stale function list (4 tools -> 7)
- Fix incorrect env-vs-UI precedence statement
- Fix em-dashes in existing content
* Fix troubleshooting curl to use pod env vars
Use sh -c so $EXTERNAL_ASSISTANT_TOKEN and $EXTERNAL_ASSISTANT_URL
expand inside the pod, not on the local shell.
* feat(helm): add Pipelock ConfigMap, scanning config, and consolidate compose
- Add ConfigMap template rendering DLP, response scanning, MCP input/tool
scanning, and forward proxy settings from values
- Mount ConfigMap as /etc/pipelock/pipelock.yaml volume in deployment
- Add checksum/config annotation for automatic pod restart on config change
- Gate HTTPS_PROXY/HTTP_PROXY env injection on forwardProxy.enabled (skip
in MCP-only mode)
- Use hasKey for all boolean values to prevent Helm default swallowing false
- Single source of truth for ports (forwardProxy.port/mcpProxy.port)
- Pipelock-specific imagePullSecrets with fallback to app secrets
- Merge standalone compose.example.pipelock.yml into compose.example.ai.yml
- Add pipelock.example.yaml for Docker Compose users
- Add exclude-paths to CI workflow for locale file false positives
* Add external assistant support (OpenAI-compatible SSE proxy)
Allow self-hosted instances to delegate chat to an external AI agent
via an OpenAI-compatible streaming endpoint. Configurable per-family
through Settings UI or ASSISTANT_TYPE env override.
- Assistant::External::Client: SSE streaming HTTP client (no new gems)
- Settings UI with type selector, env lock indicator, config status
- Helm chart and Docker Compose env var support
- 45 tests covering client, config, routing, controller, integration
* Add session key routing, email allowlist, and config plumbing
Route to the actual OpenClaw session via x-openclaw-session-key header
instead of creating isolated sessions. Gate external assistant access
behind an email allowlist (EXTERNAL_ASSISTANT_ALLOWED_EMAILS env var).
Plumb session_key and allowedEmails through Helm chart, compose, and
env template.
* Add HTTPS_PROXY support to External::Client for Pipelock integration
Net::HTTP does not auto-read HTTPS_PROXY/HTTP_PROXY env vars (unlike
Faraday). Explicitly resolve proxy from environment in build_http so
outbound traffic to the external assistant routes through Pipelock's
forward proxy when enabled. Respects NO_PROXY for internal hosts.
* Add UI fields for external assistant config (Setting-backed with env fallback)
Follow the same pattern as OpenAI settings: database-backed Setting
fields with env var defaults. Self-hosters can now configure the
external assistant URL, token, and agent ID from the browser
(Settings > Self-Hosting > AI Assistant) instead of requiring env vars.
Fields disable when the corresponding env var is set.
* Improve external assistant UI labels and add help text
Change placeholder to generic OpenAI-compatible URL pattern. Add help
text under each field explaining where the values come from: URL from
agent provider, token for authentication, agent ID for multi-agent
routing.
* Add external assistant docs and fix URL help text
Add External AI Assistant section to docs/hosting/ai.md covering setup
(UI and env vars), how it works, Pipelock security scanning, access
control, and Docker Compose example. Drop "chat completions" jargon
from URL help text.
* Harden external assistant: retry logic, disconnect UI, error handling, and test coverage
- Add retry with backoff for transient network errors (no retry after streaming starts)
- Add disconnect button with confirmation modal in self-hosting settings
- Narrow rescue scope with fallback logging for unexpected errors
- Safe cleanup of partial responses on stream interruption
- Gate ai_available? on family assistant_type instead of OR-ing all providers
- Truncate conversation history to last 20 messages
- Proxy-aware HTTP client with NO_PROXY support
- Sanitize protocol to use generic headers (X-Agent-Id, X-Session-Key)
- Full test coverage for streaming, retries, proxy routing, config, and disconnect
* Exclude external assistant client from Pipelock scan-diff
False positive: `@token` instance variable flagged as "Credential in URL".
Temporary workaround until Pipelock supports inline suppression.
* Address review feedback: NO_PROXY boundary fix, SSE done flag, design tokens
- Fix NO_PROXY matching to require domain boundary (exact match or .suffix),
case-insensitive. Prevents badexample.com matching example.com.
- Add done flag to SSE streaming so read_body stops after [DONE]
- Move MAX_CONVERSATION_MESSAGES to class level
- Use bg-success/bg-destructive design tokens for status indicators
- Add rationale comment for pipelock scan exclusion
- Update docs last-updated date
* Address second round of review feedback
- Allowlist email comparison is now case-insensitive and nil-safe
- Cap SSE buffer at 1 MB to prevent memory blowup from malformed streams
- Don't expose upstream HTTP response body in user-facing errors (log it instead)
- Fix frozen string warning on buffer initialization
- Fix "builtin" typo in docs (should be "built-in")
* Protect completed responses from cleanup, sanitize error messages
- Don't destroy a fully streamed assistant message if post-stream
metadata update fails (only cleanup partial responses)
- Log raw connection/HTTP errors internally, show generic messages
to users to avoid leaking network/proxy details
- Update test assertions for new error message wording
* Fix SSE content guard and NO_PROXY test correctness
Use nil check instead of present? for SSE delta content to preserve
whitespace-only chunks (newlines, spaces) that can occur in code output.
Fix NO_PROXY test to use HTTP_PROXY matching the http:// client URL so
the proxy resolution and NO_PROXY bypass logic are actually exercised.
* Forward proxy credentials to Net::HTTP
Pass proxy_uri.user and proxy_uri.password to Net::HTTP.new so
authenticated proxies (http://user:pass@host:port) work correctly.
Without this, credentials parsed from the proxy URL were silently
dropped. Nil values are safe as positional args when no creds exist.
* Update pipelock integration to v0.3.1 with full scanning config
Bump Helm image tag from 0.2.7 to 0.3.1. Add missing security
sections to both the Helm ConfigMap and compose example config:
mcp_tool_policy, mcp_session_binding, and tool_chain_detection.
These protect the /mcp endpoint against tool injection, session
hijacking, and multi-step exfiltration chains.
Add version and mode fields to config files. Enable include_defaults
for DLP and response scanning to merge user patterns with the 35
built-in patterns. Remove redundant --mode CLI flag from the Helm
deployment template since mode is now in the config file.
* Pipelock Helm hardening + docs for external assistant and pipelock
Helm templates:
- ServiceMonitor for Prometheus scraping on /metrics (proxy port)
- Ingress template for MCP reverse proxy (external AI agent access)
- PodDisruptionBudget with minAvailable/maxUnavailable mutual exclusion
- topologySpreadConstraints on Deployment
- Structured logging config (format, output, include_allowed/blocked)
- extraConfig escape hatch for additional pipelock.yaml sections
- requireForExternalAssistant guard (fails when assistant enabled without pipelock)
- Component label on Service metadata for ServiceMonitor targeting
- NOTES.txt pipelock section with health, access, security, metrics info
- Bump pipelock image tag 0.3.1 -> 0.3.2
- Fix: rename _asserts.tpl -> asserts.tpl (Helm skipped _ prefixed file)
Documentation:
- Helm chart README: full Pipelock section
- docs/hosting/pipelock.md: dedicated hosting guide (Docker + Kubernetes)
- docs/hosting/docker.md: AI features section (external assistant, pipelock)
- .env.example: external assistant and MCP env vars
Infra:
- Chart.lock pinning dependency versions
- .gitignore for vendored subchart tarballs
* Fix bot comments: quote ingress host, fix sidecar wording, add code block lang
* Fail fast when pipelock ingress enabled with empty hosts
* Fail fast when pipelock ingress host has empty paths
* Messed up the conflict merge
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
* feat(helm): add Pipelock ConfigMap, scanning config, and consolidate compose
- Add ConfigMap template rendering DLP, response scanning, MCP input/tool
scanning, and forward proxy settings from values
- Mount ConfigMap as /etc/pipelock/pipelock.yaml volume in deployment
- Add checksum/config annotation for automatic pod restart on config change
- Gate HTTPS_PROXY/HTTP_PROXY env injection on forwardProxy.enabled (skip
in MCP-only mode)
- Use hasKey for all boolean values to prevent Helm default swallowing false
- Single source of truth for ports (forwardProxy.port/mcpProxy.port)
- Pipelock-specific imagePullSecrets with fallback to app secrets
- Merge standalone compose.example.pipelock.yml into compose.example.ai.yml
- Add pipelock.example.yaml for Docker Compose users
- Add exclude-paths to CI workflow for locale file false positives
* Add external assistant support (OpenAI-compatible SSE proxy)
Allow self-hosted instances to delegate chat to an external AI agent
via an OpenAI-compatible streaming endpoint. Configurable per-family
through Settings UI or ASSISTANT_TYPE env override.
- Assistant::External::Client: SSE streaming HTTP client (no new gems)
- Settings UI with type selector, env lock indicator, config status
- Helm chart and Docker Compose env var support
- 45 tests covering client, config, routing, controller, integration
* Add session key routing, email allowlist, and config plumbing
Route to the actual OpenClaw session via x-openclaw-session-key header
instead of creating isolated sessions. Gate external assistant access
behind an email allowlist (EXTERNAL_ASSISTANT_ALLOWED_EMAILS env var).
Plumb session_key and allowedEmails through Helm chart, compose, and
env template.
* Add HTTPS_PROXY support to External::Client for Pipelock integration
Net::HTTP does not auto-read HTTPS_PROXY/HTTP_PROXY env vars (unlike
Faraday). Explicitly resolve proxy from environment in build_http so
outbound traffic to the external assistant routes through Pipelock's
forward proxy when enabled. Respects NO_PROXY for internal hosts.
* Add UI fields for external assistant config (Setting-backed with env fallback)
Follow the same pattern as OpenAI settings: database-backed Setting
fields with env var defaults. Self-hosters can now configure the
external assistant URL, token, and agent ID from the browser
(Settings > Self-Hosting > AI Assistant) instead of requiring env vars.
Fields disable when the corresponding env var is set.
* Improve external assistant UI labels and add help text
Change placeholder to generic OpenAI-compatible URL pattern. Add help
text under each field explaining where the values come from: URL from
agent provider, token for authentication, agent ID for multi-agent
routing.
* Add external assistant docs and fix URL help text
Add External AI Assistant section to docs/hosting/ai.md covering setup
(UI and env vars), how it works, Pipelock security scanning, access
control, and Docker Compose example. Drop "chat completions" jargon
from URL help text.
* Harden external assistant: retry logic, disconnect UI, error handling, and test coverage
- Add retry with backoff for transient network errors (no retry after streaming starts)
- Add disconnect button with confirmation modal in self-hosting settings
- Narrow rescue scope with fallback logging for unexpected errors
- Safe cleanup of partial responses on stream interruption
- Gate ai_available? on family assistant_type instead of OR-ing all providers
- Truncate conversation history to last 20 messages
- Proxy-aware HTTP client with NO_PROXY support
- Sanitize protocol to use generic headers (X-Agent-Id, X-Session-Key)
- Full test coverage for streaming, retries, proxy routing, config, and disconnect
* Exclude external assistant client from Pipelock scan-diff
False positive: `@token` instance variable flagged as "Credential in URL".
Temporary workaround until Pipelock supports inline suppression.
* Address review feedback: NO_PROXY boundary fix, SSE done flag, design tokens
- Fix NO_PROXY matching to require domain boundary (exact match or .suffix),
case-insensitive. Prevents badexample.com matching example.com.
- Add done flag to SSE streaming so read_body stops after [DONE]
- Move MAX_CONVERSATION_MESSAGES to class level
- Use bg-success/bg-destructive design tokens for status indicators
- Add rationale comment for pipelock scan exclusion
- Update docs last-updated date
* Address second round of review feedback
- Allowlist email comparison is now case-insensitive and nil-safe
- Cap SSE buffer at 1 MB to prevent memory blowup from malformed streams
- Don't expose upstream HTTP response body in user-facing errors (log it instead)
- Fix frozen string warning on buffer initialization
- Fix "builtin" typo in docs (should be "built-in")
* Protect completed responses from cleanup, sanitize error messages
- Don't destroy a fully streamed assistant message if post-stream
metadata update fails (only cleanup partial responses)
- Log raw connection/HTTP errors internally, show generic messages
to users to avoid leaking network/proxy details
- Update test assertions for new error message wording
* Fix SSE content guard and NO_PROXY test correctness
Use nil check instead of present? for SSE delta content to preserve
whitespace-only chunks (newlines, spaces) that can occur in code output.
Fix NO_PROXY test to use HTTP_PROXY matching the http:// client URL so
the proxy resolution and NO_PROXY bypass logic are actually exercised.
* Forward proxy credentials to Net::HTTP
Pass proxy_uri.user and proxy_uri.password to Net::HTTP.new so
authenticated proxies (http://user:pass@host:port) work correctly.
Without this, credentials parsed from the proxy URL were silently
dropped. Nil values are safe as positional args when no creds exist.
* Update pipelock integration to v0.3.1 with full scanning config
Bump Helm image tag from 0.2.7 to 0.3.1. Add missing security
sections to both the Helm ConfigMap and compose example config:
mcp_tool_policy, mcp_session_binding, and tool_chain_detection.
These protect the /mcp endpoint against tool injection, session
hijacking, and multi-step exfiltration chains.
Add version and mode fields to config files. Enable include_defaults
for DLP and response scanning to merge user patterns with the 35
built-in patterns. Remove redundant --mode CLI flag from the Helm
deployment template since mode is now in the config file.
* Wire ui layout and AI flags into mobile auth
Include ui_layout and ai_enabled in mobile login/signup/SSO payloads,
add an authenticated endpoint to enable AI from Flutter, and gate
mobile navigation based on intro layout and AI consent flow.
* Linter
* Ensure write scope on enable_ai
* Make sure AI is available before enabling it
* Test improvements
* PR comment
* Fix review issues: test assertion bug, missing coverage, and Dart defaults (#985)
- Fix login test to use ai_enabled? (method) instead of ai_enabled (column)
to match what mobile_user_payload actually serializes
- Add test for enable_ai when ai_available? returns false (403 path)
- Default aiEnabled to false when user is null in AuthProvider to avoid
showing AI as available before authentication completes
- Remove extra blank lines in auth_provider.dart and auth_service.dart
https://claude.ai/code/session_01LEYYmtsDBoqizyihFtkye4
Co-authored-by: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude <noreply@anthropic.com>
Add rswag request specs for merchants index/show and define a MerchantDetail schema used by the docs. Update the generated OpenAPI document with merchants paths and schema.
* Add SearchFamilyImportedFiles assistant function with vector store support
Implement per-Family document search using OpenAI vector stores, allowing
the AI assistant to search through uploaded financial documents (tax returns,
statements, contracts, etc.). The architecture is modular with a provider-
agnostic VectorStoreConcept interface so other RAG backends can be added.
Key components:
- Assistant::Function::SearchFamilyImportedFiles - tool callable from any LLM
- Provider::VectorStoreConcept - abstract vector store interface
- Provider::Openai vector store methods (create, upload, search, delete)
- Family::VectorSearchable concern with document management
- FamilyDocument model for tracking uploaded files
- Migration adding vector_store_id to families and family_documents table
https://claude.ai/code/session_01TSkKc7a9Yu2ugm1RvSf4dh
* Extract VectorStore adapter layer for swappable backends
Replace the Provider::VectorStoreConcept mixin with a standalone adapter
architecture under VectorStore::. This cleanly separates vector store
concerns from the LLM provider and makes it trivial to swap backends.
Components:
- VectorStore::Base — abstract interface (create/delete/upload/remove/search)
- VectorStore::Openai — uses ruby-openai gem's native vector_stores.search
- VectorStore::Pgvector — skeleton for local pgvector + embedding model
- VectorStore::Qdrant — skeleton for Qdrant vector DB
- VectorStore::Registry — resolves adapter from VECTOR_STORE_PROVIDER env
- VectorStore::Response — success/failure wrapper (like Provider::Response)
Consumers updated to go through VectorStore.adapter:
- Family::VectorSearchable
- Assistant::Function::SearchFamilyImportedFiles
- FamilyDocument
Removed: Provider::VectorStoreConcept, vector store methods from Provider::Openai
https://claude.ai/code/session_01TSkKc7a9Yu2ugm1RvSf4dh
* Add Vector Store configuration docs to ai.md
Documents how to configure the document search feature, covering all
three supported backends (OpenAI, pgvector, Qdrant), environment
variables, Docker Compose examples, supported file types, and privacy
considerations.
https://claude.ai/code/session_01TSkKc7a9Yu2ugm1RvSf4dh
* No need to specify `imported` in code
* Missed a couple more places
* Tiny reordering for the human OCD
* Update app/models/assistant/function/search_family_files.rb
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
* PR comments
* More PR comments
---------
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Add REST API for holdings and trades (Discussion #905)
- Trades: GET index (filter by account_id, account_ids, start_date, end_date),
GET show, POST create (buy/sell with security_id or ticker), PATCH update,
DELETE destroy. Create restricted to accounts that support trades (investment
or crypto exchange). Uses existing Trade::CreateForm for creation.
- Holdings: GET index (filter by account_id, account_ids, date, start_date,
end_date, security_id), GET show. Read-only; scoped to family.
- Auth: read scope for index/show; write scope for create/update/destroy.
- Responses: JSON via jbuilder (trade: id, date, amount, qty, price, account,
security, category; holding: id, date, qty, price, amount, account, security,
avg_cost). Pagination for index endpoints (page, per_page).
Co-authored-by: Cursor <cursoragent@cursor.com>
* API v1 holdings & trades: validation, docs, specs
- Holdings: validate date params, return 400 for invalid dates (parse_date!)
- Trades: validate start_date/end_date, return 422 for invalid dates
- Trades: accept buy/sell and inflow/outflow in update (trade_sell_from_type_or_nature?)
- Trades view: nil guard for trade.security
- Trades apply_filters: single join(:entry) when filtering
- OpenAPI: add Trade/TradeCollection schemas, ErrorResponse.errors
- Add spec/requests/api/v1/holdings_spec.rb and trades_spec.rb (rswag)
- Regenerate docs/api/openapi.yaml
Co-authored-by: Cursor <cursoragent@cursor.com>
* CI: fix Brakeman and test rate-limit failures
- Disable Rack::Attack in test (use existing enabled flag) so parallel
API tests no longer hit 429 from shared api_ip throttle
- Add Brakeman ignore for trades_controller trade_params mass-assignment
(account_id/security_id validated in create/update)
- Trades/holdings API and OpenAPI spec updates
Co-authored-by: Cursor <cursoragent@cursor.com>
* Trades: partial qty/price update fallback; fix PATCH OpenAPI schema
- Fall back to existing trade qty/price when only one is supplied so sign
normalisation and amount recalculation always run
- OpenAPI: remove top-level qty, price, investment_activity_label,
category_id from PATCH body; document entryable_attributes only
Co-authored-by: Cursor <cursoragent@cursor.com>
* Trades: fix update/DELETE OpenAPI and avoid sell-trade corruption
- Only run qty/price normalisation when client sends qty or price; preserve
existing trade direction when type/nature omitted
- OpenAPI: remove duplicate PATCH path param; add 422 for PATCH; document
DELETE 200 body (DeleteResponse)
Co-authored-by: Cursor <cursoragent@cursor.com>
* API: flat trade update params, align holdings errors, spec/OpenAPI fixes
- Trades update: accept flat params (qty, price, type, etc.), build
entryable_attributes in build_entry_params_for_update (match transactions)
- Holdings: ArgumentError → 422 validation_failed; parse_date!(value, name)
with safe message; extract render_validation_error, log_and_render_error
- Specs: path id required (trades, holdings); trades delete 200 DeleteResponse;
remove holdings 500; trades update body flat; holdings 422 invalid date
- OpenAPI: PATCH trade request body flat
Co-authored-by: Cursor <cursoragent@cursor.com>
* OpenAPI: add 422 invalid date filter to holdings index
Co-authored-by: Cursor <cursoragent@cursor.com>
* API consistency and RSwag doc-only fixes
- Trades: use render_validation_error in all 4 validation paths; safe_per_page_param case/when
- Holdings: set_holding to family.holdings.find; price as Money.format in API; safe_per_page_param case/when
- Swagger: Holding qty/price descriptions (Quantity of shares held, Formatted price per share)
- RSwag: trades delete and valuations 201 use bare run_test! (documentation only, no expect)
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix index-vs-show visibility inconsistencies and preserve custom activity labels
- Add account status filter to set_holding to match index behavior
- Add visible scope to set_trade to match index behavior
- Preserve existing investment_activity_label when updating qty/price
Co-authored-by: Cursor <cursoragent@cursor.com>
* Trades: clearer validation for non-numeric qty/price
Return 'must be valid numbers' when qty or price is non-numeric (e.g. abc)
instead of misleading 'must be present and positive'.
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: add SSL_CA_FILE and SSL_VERIFY environment variables to support self-signed certificates in self-hosted environments
* fix: NoMethodError by defining SSL helper methods before configure block executes
* refactor: Refactor SessionsController to use shared SslConfigurable module and simplify SSL initializer redundant checks
* refactor: improve SSL configuration robustness and error detection accuracy
* fix:HTTParty SSL options, add file validation guards, prevent Tempfile GC, and redact URLs in error logs
* fix: Fix SSL concern indentation and stub Simplefin POST correctly in tests
* fix: normalize ssl_verify to always return boolean instead of nil
* fix: solve failing SimpleFin test
* refactor: trim unused error-handling code from SslConfigurable, replace Tempfile with fixed-path CA bundle, fix namespace pollution in initializers, and add unit tests for core SSL configuration and Langfuse CRL callback.
* fix: added require ileutils in the initializer and require ostruct in the test file.
* fix: solve autoload conflict that broke provider loading, validate all certs in PEM bundles, and add missing requires.
* feat: Add PDF import with AI-powered document analysis
This enhances the import functionality to support PDF files with AI-powered
document analysis. When a PDF is uploaded, it is processed by AI to:
- Identify the document type (bank statement, credit card statement, etc.)
- Generate a summary of the document contents
- Extract key metadata (institution, dates, balances, transaction count)
After processing, an email is sent to the user asking for next steps.
Key changes:
- Add PdfImport model for handling PDF document imports
- Add Provider::Openai::PdfProcessor for AI document analysis
- Add ProcessPdfJob for async PDF processing
- Add PdfImportMailer for user notification emails
- Update imports controller to detect and handle PDF uploads
- Add PDF import option to the new import page
- Add i18n translations for all new strings
- Add comprehensive tests for the new functionality
* Add bank statement import with AI extraction
- Create ImportBankStatement assistant function for MCP
- Add BankStatementExtractor with chunked processing for small context windows
- Register function in assistant configurable
- Make PdfImport#pdf_file_content public for extractor access
- Increase OpenAI request timeout to 600s for slow local models
- Increase DB connection pool to 20 for concurrent operations
Tested with M-Pesa bank statement via remote Ollama (qwen3:8b):
- Successfully extracted 18 transactions
- Generated CSV and created TransactionImport
- Works with 3000 char chunks for small context windows
* Add pdf-reader gem dependency
The BankStatementExtractor uses PDF::Reader to parse bank statement
PDFs, but the gem was not properly declared in the Gemfile. This would
cause NameError in production when processing bank statements.
Added pdf-reader ~> 2.12 to Gemfile dependencies.
* Fix transaction deduplication to preserve legitimate duplicates
The previous deduplication logic removed ALL duplicate transactions based
on [date, amount, name], which would drop legitimate same-day duplicates
like multiple ATM withdrawals or card authorizations.
Changed to only deduplicate transactions that appear in consecutive chunks
(chunking artifacts) while preserving all legitimate duplicates within the
same chunk or non-adjacent chunks.
* Refactor bank statement extraction to use public provider method
Address code review feedback:
- Add public extract_bank_statement method to Provider::Openai
- Remove direct access to private client via send(:client)
- Update ImportBankStatement to use new public method
- Add require 'set' to BankStatementExtractor
- Remove PII-sensitive content from error logs
- Add defensive check for nil response.error
- Handle oversized PDF pages in chunking logic
- Remove unused process_native and process_generic methods
- Update email copy to reflect feature availability
- Add guard for nil document_type in email template
- Document pdf-reader gem rationale in Gemfile
Tested with both OpenAI (gpt-4o) and Ollama (qwen3:8b):
- OpenAI: 49 transactions extracted in 30s
- Ollama: 40 transactions extracted in 368s
- All encapsulation and error handling working correctly
* Update schema.rb with ai_summary and document_type columns
* Address PR #808 review comments
- Rename :csv_file to :import_file across controllers/views/tests
- Add PDF test fixture (sample_bank_statement.pdf)
- Add supports_pdf_processing? method for graceful degradation
- Revert unrelated database.yml pool change (600->3)
- Remove month_start_day schema bleed from other PR
- Fix PdfProcessor: use .strip instead of .strip_heredoc
- Add server-side PDF magic byte validation
- Conditionally show PDF import option when AI provider available
- Fix ProcessPdfJob: sanitize errors, handle update failure
- Move pdf_file attachment from Import to PdfImport
- Document deduplication logic limitations
- Fix ImportBankStatement: catch specific exceptions only
- Remove unnecessary require 'set'
- Remove dead json_schema method from PdfProcessor
- Reduce default OpenAI timeout from 600s to 60s
- Fix nil guard in text mailer template
- Add require 'csv' to ImportBankStatement
- Remove Gemfile pdf-reader comment
* Fix RuboCop indentation in ProcessPdfJob
* Refactor PDF import check to use model predicate method
Replace is_a?(PdfImport) type check with requires_csv_workflow? predicate
that leverages STI inheritance for cleaner controller logic.
* Fix missing 'unknown' locale key and schema version mismatch
- Add 'unknown: Unknown Document' to document_types locale
- Fix schema version to match latest migration (2026_01_24_180211)
* Document OPENAI_REQUEST_TIMEOUT env variable
Added to .env.local.example and docs/hosting/ai.md
* Rename ALLOWED_MIME_TYPES to ALLOWED_CSV_MIME_TYPES for clarity
* Add comment explaining requires_csv_workflow? predicate
* Remove redundant required_column_keys from PdfImport
Base class already returns [] by default
* Add ENV toggle to disable PDF processing for non-vision endpoints
OPENAI_SUPPORTS_PDF_PROCESSING=false can be used for OpenAI-compatible
endpoints (e.g., Ollama) that don't support vision/PDF processing.
* Wire up transaction extraction for PDF bank statements
- Add extracted_data JSONB column to imports
- Add extract_transactions method to PdfImport
- Call extraction in ProcessPdfJob for bank statements
- Store transactions in extracted_data for later review
* Fix ProcessPdfJob retry logic, sanitize and localize errors
- Allow retries after partial success (classification ok, extraction failed)
- Log sanitized error message instead of raw message to avoid data leakage
- Use i18n for user-facing error messages
* Add vision-capable model validation for PDF processing
* Fix drag-and-drop test to use correct field name csv_file
* Schema bleedover from another branch
* Fix drag-drop import form field name to match controller
* Add vision capability guard to process_pdf method
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: mkdev11 <jaysmth689+github@users.noreply.github.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
* feat: add valuations API endpoints for managing account reconciliations
* refactor: formatting
* fix: make account extraction clearer
* feat: validation and error handling improvements
* feat: transaction
* feat: error handling
* Add API documentation LLM context
* Make it easier for people
* feat: transaction in creation
* feat: add OpenAPI spec for Valuations API
* fix: update notes validation to check for key presence
* Prevent double render
* All other docs use `apiKeyAuth`
* More `apiKeyAuth`
* Remove testing assertions from API doc specs
* fix: correct valuation entry references
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Suggest to use .env.example
it's easier to start with a pre-done file than search for info about each configuration
* added paragraph to tell how to activate https
* fix type.
As there was 2 space, I assumed there should be a line break
* Corrects minor typos in HTTPS documentation
Addresses a couple of minor typographical errors in the HTTPS documentation section, improving clarity and readability for users setting up HTTPS.
* Clarifies .env setup in Docker documentation
Corrects grammar and improves clarity in the Docker documentation regarding obtaining the `.env.example` file for initial configuration.
* Fixes typo in HTTPS documentation
Corrects a grammatical error in the Docker HTTPS setup guide, improving clarity for users configuring SSL.
* Implement API v1 Imports controller
- Add Api::V1::ImportsController with index, show, and create actions
- Add Jbuilder views for index and show
- Add integration tests
- Implement row generation logic in create action
- Update routes
* Validate import account belongs to family
- Add validation to Import model to ensure account belongs to the same family
- Add regression test case in Api::V1::ImportsControllerTest
* updating docs to be more detailed
* Rescue StandardError instead of bare rescue in ImportsController
* Optimize Imports API and fix documentation
- Implement rows_count counter cache for Imports
- Preload rows in Api::V1::ImportsController#show
- Update documentation to show correct OAuth scopes
* Fix formatting in ImportsControllerTest
* Permit all import parameters and fix unknown attribute error
* Restore API routes for auth, chats, and messages
* removing pr summary
* Fix trailing whitespace and configured? test failure
- Update Import#configured? to use rows_count for performance and consistency
- Mock rows_count in TransactionImportTest
- Fix trailing whitespace in migration
* Harden security and fix mass assignment in ImportsController
- Handle type and account_id explicitly in create action
- Rename import_params to import_config_params for clarity
- Validate type against Import::TYPES
* Fix MintImport rows_count update and migration whitespace
- Update MintImport#generate_rows_from_csv to update rows_count counter cache
- Fix trailing whitespace and final newline in AddRowsCountToImports migration
* Implement full-screen Drag and Drop CSV import on Transactions page
- Add DragAndDropImport Stimulus controller listening on document
- Add full-screen overlay with icon and text to Transactions index
- Update ImportsController to handle direct file uploads via create action
- Add system test for drag and drop functionality
* Implement Drag and Drop CSV upload on Import Upload page
- Add drag-and-drop-import controller to import/uploads/show
- Add full-screen overlay to import/uploads/show
- Annotate upload form and input with drag-and-drop targets
- Add PR_SUMMARY.md
* removing pr summary
* Add file validation to ImportsController
- Validate file size (max 10MB) and MIME type in create action
- Prevent memory exhaustion and invalid file processing
- Defined MAX_CSV_SIZE and ALLOWED_MIME_TYPES in Import model
* Refactor dragLeave logic with counter pattern to prevent flickering
* Extract shared drag-and-drop overlay partial
- Create app/views/imports/_drag_drop_overlay.html.erb
- Update transactions/index and import/uploads/show to use the partial
- Reduce code duplication in views
* Update Brakeman and harden ImportsController security
- Update brakeman to 7.1.2
- Explicitly handle type assignment in ImportsController#create to avoid mass assignment
- Remove :type from permitted import parameters
* Fix trailing whitespace in DragAndDropImportTest
* Don't commit LLM comments as file
* FIX add api validation
---------
Co-authored-by: Carlos Adames <cj@Carloss-MacBook-Air.local>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: sokie <sokysrm@gmail.com>
* Document CSV import processing delay issue
Added troubleshooting information for CSV import delays.
Signed-off-by: zenaufa <zenaufa@hotmail.com>
* Small edits suggested by LLM
---------
Signed-off-by: zenaufa <zenaufa@hotmail.com>
Co-authored-by: Juan José Mata <jjmata@jjmata.com>
Multi-provider SSO support:
- Database-backed SSO provider management with admin UI
- Support for OpenID Connect, Google OAuth2, GitHub, and SAML 2.0
- Flipper feature flag (db_sso_providers) for dynamic provider loading
- ProviderLoader service for YAML or database configuration
Admin functionality:
- Admin::SsoProvidersController for CRUD operations
- Admin::UsersController for super_admin role management
- Pundit policies for authorization
- Test connection endpoint for validating provider config
User provisioning improvements:
- JIT (just-in-time) account creation with configurable default role
- Changed default JIT role from admin to member (security)
- User attribute sync on each SSO login
- Group/role mapping from IdP claims
SSO identity management:
- Settings::SsoIdentitiesController for users to manage connected accounts
- Issuer validation for OIDC identities
- Unlink protection when no password set
Audit logging:
- SsoAuditLog model tracking login, logout, link, unlink, JIT creation
- Captures IP address, user agent, and metadata
Advanced OIDC features:
- Custom scopes per provider
- Configurable prompt parameter (login, consent, select_account, none)
- RP-initiated logout (federated logout to IdP)
- id_token storage for logout
SAML 2.0 support:
- omniauth-saml gem integration
- IdP metadata URL or manual configuration
- Certificate and fingerprint validation
- NameID format configuration
* Add configuration and logic for dynamic SSO provider support and stricter JIT account creation
- Introduced `config/auth.yml` for centralized auth configuration and documentation.
- Added support for multiple SSO providers, including Google, GitHub, and OpenID Connect.
- Implemented stricter JIT SSO account creation modes (`create_and_link` vs `link_only`).
- Enabled optional restriction of JIT creation by allowed email domains.
- Enhanced OmniAuth initializer for dynamic provider setup and better configurability.
- Refined login UI to handle local login disabling and emergency super-admin override.
- Updated account creation flow to respect JIT mode and domain checks.
- Added tests for SSO account creation, login form visibility, and emergency overrides.
# Conflicts:
# app/controllers/sessions_controller.rb
* remove non-translation
* Refactor authentication views to use translation keys and update locale files
- Extracted hardcoded strings in `oidc_accounts/link.html.erb` and `sessions/new.html.erb` into translation keys for better localization support.
- Added missing translations for English and Spanish in `sessions` and `oidc_accounts` locale files.
* Enhance OmniAuth provider configuration and refine local login override logic
- Updated OmniAuth initializer to support dynamic provider configuration with `name` and scoped parameters for Google and GitHub.
- Improved local login logic to enforce stricter handling of super-admin override when local login is disabled.
- Added test for invalid super-admin override credentials.
* Document Google sign-in configuration for local development and self-hosted environments
---------
Co-authored-by: Josh Waldrep <joshua.waldrep5+github@gmail.com>
* Add RSwag coverage for chat API
* Linter
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
* Add transaction rswag
* FIX linter
---------
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: sokie <sokysrm@gmail.com>
* Add OpenID Connect login support
* Add docs for OIDC config with Google Auth
* Use Google styles for log in
- Add support for linking existing account
- Force users to sign-in with passoword first, when linking existing accounts
- Add support to create new user when using OIDC
- Add identities to user to prevent account take-ver
- Make tests mocking instead of being integration tests
- Manage session handling correctly
- use OmniAuth.config.mock_auth instead of passing auth data via request env
* Conditionally render Oauth button
- Set a config item `configuration.x.auth.oidc_enabled`
- Hide button if disabled
---------
Signed-off-by: Juan José Mata <juanjo.mata@gmail.com>
Signed-off-by: soky srm <sokysrm@gmail.com>
Co-authored-by: sokie <sokysrm@gmail.com>
* Add comprehensive AI/LLM configuration documentation
* Fix Chat.start! to use default model when model is nil or empty
* Ensure all controllers use Chat.default_model for consistency
* Move AI doc inside `hosting/`
* Probably too much error handling
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: jjmata <187772+jjmata@users.noreply.github.com>
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* Remove orphan function
* Add centralized branding helpers and update locales
* Remove _plus and add (proper) brand
* No longer Sure, configurable
* Consistency with compose file naming
* Missed `product_name` mapping
* Fix brand/product name in mailers
* Product name in email reset flow
* Fix i18n errors/tests
* Fix password mailer brand/product name (again)
* Missed hardcoded `Sure` in onboarding goals
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
* PR nitpick on documentation
* Missing interpolation key for invited UI
* Orphan assets
* New logos
---------
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
* Replace Maybe for Sure in select code areas
* Make sure passwords are consistent
* Remove (admin|member) from demo data first name
* Database and schema names finally to `sure`
* Fix broken test
* Another (benchmarking) database name to `sure_*`
* More rebranding to Sure
* Missed this Maybe mention in the same page
* Random nitpicks and more Maybes
* Demo data accounts and more Maybes
* Test data account updates
* Impersonation test accounts
* Consistency with `compose.example.yml`
* maybe#2419 was recreated as sure#1 but message wasn't removed.
* Information in maybe#2419 gives more details that are helpful for
users wanting to setup Plaid
* add initial pages for Bank Sync and Lunch Flow
* update breadcrumbs on Lunch Flow page
* update content for Lunch Flow page
* add norefeerrer to Github link
* update lunch flow url
* nest lunch_flow resource under bank_sync
* add a provider link partial
* remove trailing whitespaces
* update providers style to match merchants page
* remove separate lunch flow page
* fix hover on dark mode
* point lunch flow to custom sure landing page
* [i18n] Bank Sync label
* [i18n] API Keys
* [i18n] Self-Hosting consistency
* Security breadcrum, not "securities" default
---------
Co-authored-by: Juan José Mata <juanjo.mata@gmail.com>
* feat: Add Brand Fetch logo link for logos
* docs: brand fetch integration docs
* Document CLIENT_ID location?
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
---------
Signed-off-by: Juan José Mata <jjmata@jjmata.com>
Co-authored-by: Vincent Teo <vinteo@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
* First cut of smallest rebrand, pending icons
* Leave SQL schema tokens/user names the same for now
* First batch of logos
* Release notes/what's new
* /releases missing
* redirect_uri for sureapp://
* Padded logo
* Test the correct /releases URL
* Missed a few mobile URIs
* Some icons/asssets from /website/ repo
* Seed/sample data user @sure.local now
* New screenshot
* Want to keep their legal "boilerplate" from the upstream repo
