diff --git a/.github/workflows/pipelock.yml b/.github/workflows/pipelock.yml index 3668c0a49..f7e5bd897 100644 --- a/.github/workflows/pipelock.yml +++ b/.github/workflows/pipelock.yml @@ -24,3 +24,4 @@ jobs: test-vectors: 'false' exclude-paths: | config/locales/views/reports/ + docs/hosting/ai.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e032cd731..c44b64524 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -40,3 +40,12 @@ To get setup for local development, you have two options: 7. Before requesting a review, please make sure that all [Github Checks](https://docs.github.com/en/rest/checks?apiVersion=2022-11-28) have passed and your branch is up-to-date with the `main` branch. After doing so, request a review and wait for a maintainer's approval. All PRs should target the `main` branch. + +### Automated Security Scanning + +Every pull request to the `main` branch automatically runs a Pipelock security scan. This scan analyzes your PR diff for: + +- Leaked secrets (API keys, tokens, credentials) +- Agent security risks (misconfigurations, exposed credentials, missing controls) + +The scan runs as part of the CI pipeline and typically completes in ~30 seconds. If security issues are found, the CI check will fail. You don't need to configure anything—the security scanning is automatic and zero-configuration. diff --git a/docs/api/users.md b/docs/api/users.md new file mode 100644 index 000000000..d7f4f1c12 --- /dev/null +++ b/docs/api/users.md @@ -0,0 +1,117 @@ +# Users API Documentation + +The Users API allows external applications to manage user account data within Sure. The OpenAPI description is generated directly from executable request specs, ensuring it always reflects the behaviour of the running Rails application. + +## Generated OpenAPI specification + +- The source of truth for the documentation lives in [`spec/requests/api/v1/users_spec.rb`](../../spec/requests/api/v1/users_spec.rb). These specs authenticate against the Rails stack, exercise every user endpoint, and capture real response shapes. +- Regenerate the OpenAPI document with: + + ```sh + RAILS_ENV=test bundle exec rake rswag:specs:swaggerize + ``` + + The task compiles the request specs and writes the result to [`docs/api/openapi.yaml`](openapi.yaml). + +- Run just the documentation specs with: + + ```sh + bundle exec rspec spec/requests/api/v1/users_spec.rb + ``` + +## Authentication requirements + +All user endpoints require an OAuth2 access token or API key that grants the `read_write` scope. + +## Available endpoints + +| Endpoint | Scope | Description | +| --- | --- | --- | +| `DELETE /api/v1/users/reset` | `read_write` | Reset account data while preserving the user account. | +| `DELETE /api/v1/users/me` | `read_write` | Permanently delete the user account. | + +Refer to the generated [`openapi.yaml`](openapi.yaml) for request/response schemas, reusable components (errors), and security definitions. + +## Reset account + +`DELETE /api/v1/users/reset` + +Resets all financial data (accounts, categories, merchants, tags, transactions, etc.) for the current user's family while keeping the user account intact. The reset runs asynchronously in the background. + +### Request + +No request body required. + +### Response + +```json +{ + "message": "Account reset has been initiated" +} +``` + +### Use cases + +- Clear all financial data to start fresh +- Remove test data after initial setup +- Reset to a clean state for new imports + +## Delete account + +`DELETE /api/v1/users/me` + +Permanently deactivates the current user account and all associated data. This action cannot be undone. + +### Request + +No request body required. + +### Response + +```json +{ + "message": "Account has been deleted" +} +``` + +### Error responses + +In addition to standard error codes (`unauthorized`, `insufficient_scope`), the delete endpoint may return: + +**422 Unprocessable Entity** + +```json +{ + "error": "Failed to delete account", + "details": ["Cannot deactivate admin with other users"] +} +``` + +This occurs when the user cannot be deactivated (for example, an admin user with other active users in the family). + +## Security considerations + +- Both endpoints require the `read_write` scope. Read-only API keys cannot access these endpoints. +- Deactivated users cannot access these endpoints. +- The reset operation preserves the user account, allowing you to continue using Sure with a clean slate. +- The delete operation is permanent and removes the user account entirely. + +## Error responses + +Errors conform to the shared `ErrorResponse` schema in the OpenAPI document: + +```json +{ + "error": "error_code", + "message": "Human readable error message", + "details": ["Optional array of extra context"] +} +``` + +Common error codes include: + +| Code | Description | +| --- | --- | +| `unauthorized` | Missing or invalid API key | +| `insufficient_scope` | API key lacks required `read_write` scope | +| `Failed to delete account` | Account deletion failed (see details field) | \ No newline at end of file diff --git a/docs/hosting/ai.md b/docs/hosting/ai.md index d32bd1b6e..60a5b5b8b 100644 --- a/docs/hosting/ai.md +++ b/docs/hosting/ai.md @@ -511,6 +511,238 @@ x-rails-env: &rails_env Or configure the assistant via the Settings UI after startup (MCP env vars are still required for callback). +## Assistant Architecture + +Sure's AI assistant system uses a modular architecture that allows different assistant implementations to be plugged in based on configuration. This section explains the architecture for contributors who want to understand or extend the system. + +### Overview + +The assistant system evolved from a monolithic class to a module-based architecture with a registry pattern. This allows Sure to support multiple assistant types (builtin, external) and makes it easy to add new implementations. + +**Key benefits:** +- **Extensible:** Add new assistant types without modifying existing code +- **Configurable:** Choose assistant type per family or globally +- **Isolated:** Each implementation has its own logic and dependencies +- **Testable:** Implementations are independent and can be tested separately + +### Component Hierarchy + +#### `Assistant` Module + +The main entry point for all assistant operations. Located in `app/models/assistant.rb`. + +**Key methods:** + +| Method | Description | +|--------|-------------| +| `.for_chat(chat)` | Returns the appropriate assistant instance for a chat | +| `.config_for(chat)` | Returns configuration for builtin assistants | +| `.available_types` | Lists all registered assistant types | +| `.function_classes` | Returns all available function/tool classes | + +**Example usage:** + +```ruby +# Get an assistant for a chat +assistant = Assistant.for_chat(chat) + +# Respond to a message +assistant.respond_to(message) +``` + +#### `Assistant::Base` + +Abstract base class that all assistant implementations inherit from. Located in `app/models/assistant/base.rb`. + +**Contract:** +- Must implement `respond_to(message)` instance method +- Includes `Assistant::Broadcastable` for real-time updates +- Receives the `chat` object in the initializer + +**Example implementation:** + +```ruby +class Assistant::MyCustom < Assistant::Base + def respond_to(message) + # Your custom logic here + assistant_message = AssistantMessage.new(chat: chat, content: "Response") + assistant_message.save! + end +end +``` + +#### `Assistant::Builtin` + +The default implementation that uses the configured OpenAI-compatible LLM provider. Located in `app/models/assistant/builtin.rb`. + +**Features:** +- Uses `Assistant::Provided` for LLM provider selection +- Uses `Assistant::Configurable` for system prompts and function configuration +- Supports function calling via `Assistant::FunctionToolCaller` +- Streams responses in real-time + +**Key methods:** + +| Method | Description | +|--------|-------------| +| `.for_chat(chat)` | Creates a new builtin assistant with config | +| `#respond_to(message)` | Processes a message using the LLM | + +#### `Assistant::External` + +Implementation for delegating chat to a remote AI agent. Located in `app/models/assistant/external.rb`. + +**Features:** +- Sends conversation to external agent via OpenAI-compatible API +- Agent calls back to Sure's `/mcp` endpoint for financial data +- Supports access control via email allowlist +- Streams responses from the agent + +**Configuration:** + +```ruby +config = Assistant::External.config +# => # +``` + +### Registry Pattern + +The `Assistant` module uses a registry to map type names to implementation classes: + +```ruby +REGISTRY = { + "builtin" => Assistant::Builtin, + "external" => Assistant::External +}.freeze +``` + +**Type selection logic:** + +1. Check `ENV["ASSISTANT_TYPE"]` (global override) +2. Check `chat.user.family.assistant_type` (per-family setting) +3. Default to `"builtin"` + +**Example:** + +```ruby +# Global override +ENV["ASSISTANT_TYPE"] = "external" +Assistant.for_chat(chat) # => Assistant::External instance + +# Per-family setting +family.update(assistant_type: "external") +Assistant.for_chat(chat) # => Assistant::External instance + +# Default +Assistant.for_chat(chat) # => Assistant::Builtin instance +``` + +### Function Registry + +The `Assistant.function_classes` method centralizes all available financial tools: + +```ruby +def self.function_classes + [ + Function::GetTransactions, + Function::GetAccounts, + Function::GetHoldings, + Function::GetBalanceSheet, + Function::GetIncomeStatement, + Function::ImportBankStatement, + Function::SearchFamilyFiles + ] +end +``` + +These functions are: +- Used by builtin assistants for LLM function calling +- Exposed via the MCP endpoint for external agents +- Defined in `app/models/assistant/function/` + +### Adding a New Assistant Type + +To add a custom assistant implementation: + +#### 1. Create the implementation class + +```ruby +# app/models/assistant/my_custom.rb +class Assistant::MyCustom < Assistant::Base + class << self + def for_chat(chat) + new(chat) + end + end + + def respond_to(message) + # Your implementation here + # Must create and save an AssistantMessage + assistant_message = AssistantMessage.new( + chat: chat, + content: "My custom response" + ) + assistant_message.save! + end +end +``` + +#### 2. Register the implementation + +```ruby +# app/models/assistant.rb +REGISTRY = { + "builtin" => Assistant::Builtin, + "external" => Assistant::External, + "my_custom" => Assistant::MyCustom +}.freeze +``` + +#### 3. Add validation + +```ruby +# app/models/family.rb +ASSISTANT_TYPES = %w[builtin external my_custom].freeze +``` + +#### 4. Use the new type + +```bash +# Global override +ASSISTANT_TYPE=my_custom + +# Or set per-family in the database +family.update(assistant_type: "my_custom") +``` + +### Integration Points + +#### Pipelock Integration + +For external assistants, Pipelock can scan traffic: +- **Outbound:** Sure -> agent (via `HTTPS_PROXY`) +- **Inbound:** Agent -> Sure /mcp (via MCP reverse proxy on port 8889) + +See the [External AI Assistant](#external-ai-assistant) and [Pipelock](pipelock.md) documentation for configuration. + +#### OpenClaw/WebSocket Support + +The `Assistant::External` implementation currently uses HTTP streaming. Future implementations could use WebSocket connections via OpenClaw or other gateways. + +**Example future implementation:** + +```ruby +class Assistant::WebSocket < Assistant::Base + def respond_to(message) + # Connect via WebSocket + # Stream bidirectional communication + # Handle tool calls via MCP + end +end +``` + +Register it in the `REGISTRY` and add to `Family::ASSISTANT_TYPES` to activate. + ## AI Cache Management Sure caches AI-generated results (like auto-categorization and merchant detection) to avoid redundant API calls and costs. However, there are situations where you may want to clear this cache. diff --git a/docs/hosting/mcp.md b/docs/hosting/mcp.md new file mode 100644 index 000000000..671feb700 --- /dev/null +++ b/docs/hosting/mcp.md @@ -0,0 +1,338 @@ +# MCP Server for External AI Assistants + +Sure includes a Model Context Protocol (MCP) server endpoint that allows external AI assistants like Claude Desktop, GPT agents, or custom AI clients to query your financial data. + +## What is MCP? + +[Model Context Protocol](https://modelcontextprotocol.io/) is a JSON-RPC 2.0 protocol that enables AI assistants to access structured data and tools from external applications. Instead of copying and pasting financial data into a chat window, your AI assistant can directly query Sure's data through a secure API. + +This is useful when: +- You want to use an external AI assistant (Claude, GPT, custom agents) to analyze your Sure financial data +- You prefer to keep your LLM provider separate from Sure +- You're building custom AI agents that need access to financial tools + +## Prerequisites + +To enable the MCP endpoint, you need to set two environment variables: + +| Variable | Description | Example | +|----------|-------------|---------| +| `MCP_API_TOKEN` | Bearer token for authentication | `your-secret-token-here` | +| `MCP_USER_EMAIL` | Email of the Sure user whose data the assistant can access | `user@example.com` | + +Both variables are **required**. The endpoint will not activate if either is missing. + +### Generating a secure token + +Generate a random token for `MCP_API_TOKEN`: + +```bash +# macOS/Linux +openssl rand -base64 32 + +# Or use any secure password generator +``` + +### Choosing the user + +The `MCP_USER_EMAIL` must match an existing Sure user's email address. The AI assistant will have access to all financial data for that user's family. + +> [!CAUTION] +> The AI assistant will have **read access to all financial data** for the specified user. Only set this for users you trust with your AI provider. + +## Configuration + +### Docker Compose + +Add the environment variables to your `compose.yml`: + +```yaml +x-rails-env: &rails_env + MCP_API_TOKEN: your-secret-token-here + MCP_USER_EMAIL: user@example.com +``` + +Both `web` and `worker` services inherit this configuration. + +### Kubernetes (Helm) + +Add the variables to your `values.yaml` or set them via Secrets: + +```yaml +env: + MCP_API_TOKEN: your-secret-token-here + MCP_USER_EMAIL: user@example.com +``` + +Or create a Secret and reference it: + +```yaml +envFrom: + - secretRef: + name: sure-mcp-credentials +``` + +## Protocol Details + +The MCP endpoint is available at: + +``` +POST /mcp +``` + +### Authentication + +All requests must include the `MCP_API_TOKEN` as a Bearer token: + +``` +Authorization: Bearer +``` + +### Supported Methods + +Sure implements the following JSON-RPC 2.0 methods: + +| Method | Description | +|--------|-------------| +| `initialize` | Protocol handshake, returns server info and capabilities | +| `tools/list` | Lists available financial tools with schemas | +| `tools/call` | Executes a tool with provided arguments | + +### Available Tools + +The MCP endpoint exposes these financial tools: + +| Tool | Description | +|------|-------------| +| `get_transactions` | Retrieve transaction history with filtering | +| `get_accounts` | Get account information and balances | +| `get_holdings` | Query investment holdings | +| `get_balance_sheet` | Current financial position (assets, liabilities, net worth) | +| `get_income_statement` | Income and expenses over a period | +| `import_bank_statement` | Import bank statement data | +| `search_family_files` | Search uploaded documents in the vault | + +These are the same tools used by Sure's builtin AI assistant. + +## Example Requests + +### Initialize + +Handshake to verify protocol version and capabilities: + +```bash +curl -X POST https://your-sure-instance/mcp \ + -H "Authorization: Bearer your-secret-token" \ + -H "Content-Type: application/json" \ + -d '{ + "jsonrpc": "2.0", + "id": 1, + "method": "initialize" + }' +``` + +Response: + +```json +{ + "jsonrpc": "2.0", + "id": 1, + "result": { + "protocolVersion": "2025-03-26", + "capabilities": { + "tools": {} + }, + "serverInfo": { + "name": "sure", + "version": "1.0" + } + } +} +``` + +### List Tools + +Get available tools with their schemas: + +```bash +curl -X POST https://your-sure-instance/mcp \ + -H "Authorization: Bearer your-secret-token" \ + -H "Content-Type: application/json" \ + -d '{ + "jsonrpc": "2.0", + "id": 2, + "method": "tools/list" + }' +``` + +Response includes tool names, descriptions, and JSON schemas for parameters. + +### Call a Tool + +Execute a tool to get transactions: + +```bash +curl -X POST https://your-sure-instance/mcp \ + -H "Authorization: Bearer your-secret-token" \ + -H "Content-Type: application/json" \ + -d '{ + "jsonrpc": "2.0", + "id": 3, + "method": "tools/call", + "params": { + "name": "get_transactions", + "arguments": { + "start_date": "2024-01-01", + "end_date": "2024-01-31" + } + } + }' +``` + +Response: + +```json +{ + "jsonrpc": "2.0", + "id": 3, + "result": { + "content": [ + { + "type": "text", + "text": "[{\"id\":\"...\",\"amount\":-45.99,\"date\":\"2024-01-15\",\"name\":\"Coffee Shop\"}]" + } + ] + } +} +``` + +## Security Considerations + +### Transient Session Isolation + +The MCP controller creates a **transient session** for each request. This prevents session state leaks that could expose other users' data if the Sure instance is using impersonation features. + +Each MCP request: +1. Authenticates the token +2. Loads the user specified in `MCP_USER_EMAIL` +3. Creates a temporary session scoped to that user +4. Executes the tool call +5. Discards the session + +This ensures the AI assistant can only access data for the intended user. + +### Pipelock Security Scanning + +For production deployments, we recommend using [Pipelock](https://github.com/luckyPipewrench/pipelock) to scan MCP traffic for security threats. + +Pipelock provides: +- **DLP scanning**: Detects secrets being exfiltrated through tool calls +- **Prompt injection detection**: Identifies attempts to manipulate the AI +- **Tool poisoning detection**: Prevents malicious tool call sequences +- **Policy enforcement**: Block or warn on suspicious patterns + +See the [Pipelock documentation](pipelock.md) and the example configuration in `compose.example.pipelock.yml` for setup instructions. + +### Network Security + +The `/mcp` endpoint is exposed on the same port as the web UI (default 3000). For hardened deployments: + +**Docker Compose:** +- The MCP endpoint is protected by the `MCP_API_TOKEN` but is reachable on port 3000 +- For additional security, use Pipelock's MCP reverse proxy (port 8889) which adds scanning +- See `compose.example.ai.yml` for a Pipelock configuration + +**Kubernetes:** +- Use NetworkPolicies to restrict access to the MCP endpoint +- Route external agents through Pipelock's MCP reverse proxy +- See the [Helm chart documentation](../../charts/sure/README.md) for Pipelock ingress setup + +## Production Deployment + +For a production-ready setup with security scanning: + +1. **Download the example configuration:** + + ```bash + curl -o compose.ai.yml https://raw.githubusercontent.com/we-promise/sure/main/compose.example.ai.yml + curl -o pipelock.example.yaml https://raw.githubusercontent.com/we-promise/sure/main/pipelock.example.yaml + ``` + +2. **Set your MCP credentials in `.env`:** + + ```bash + MCP_API_TOKEN=your-secret-token + MCP_USER_EMAIL=user@example.com + ``` + +3. **Start the stack:** + + ```bash + docker compose -f compose.ai.yml up -d + ``` + +4. **Connect your AI assistant to the Pipelock MCP proxy:** + + ``` + http://your-server:8889 + ``` + +The Pipelock proxy (port 8889) scans all MCP traffic before forwarding to Sure's `/mcp` endpoint. + +## Connecting AI Assistants + +### Claude Desktop + +Configure Claude Desktop to use Sure's MCP server: + +1. Open Claude Desktop settings +2. Add a new MCP server +3. Set the endpoint to `http://your-server:8889` (if using Pipelock) or `http://your-server:3000/mcp` +4. Add the authorization header: `Authorization: Bearer your-secret-token` + +### Custom Agents + +Any AI agent that supports JSON-RPC 2.0 can connect to the MCP endpoint. The agent should: + +1. Send a POST request to `/mcp` +2. Include the `Authorization: Bearer ` header +3. Use the JSON-RPC 2.0 format for requests +4. Handle the protocol methods: `initialize`, `tools/list`, `tools/call` + +## Troubleshooting + +### "MCP endpoint not configured" error + +**Symptom:** Requests return HTTP 503 with "MCP endpoint not configured" + +**Fix:** Ensure both `MCP_API_TOKEN` and `MCP_USER_EMAIL` are set as environment variables and restart Sure. + +### "unauthorized" error + +**Symptom:** Requests return HTTP 401 with "unauthorized" + +**Fix:** Verify the `Authorization` header contains the correct token: `Bearer ` + +### "MCP user not configured" error + +**Symptom:** Requests return HTTP 503 with "MCP user not configured" + +**Fix:** The `MCP_USER_EMAIL` does not match an existing user. Check that: +- The email is correct +- The user exists in the database +- There are no typos or extra spaces + +### Pipelock connection refused + +**Symptom:** AI assistant cannot connect to Pipelock's MCP proxy (port 8889) + +**Fix:** +1. Verify Pipelock is running: `docker compose ps pipelock` +2. Check Pipelock health: `docker compose exec pipelock /pipelock healthcheck --addr 127.0.0.1:8888` +3. Verify the port is exposed in your `compose.yml` + +## See Also + +- [External AI Assistant Configuration](ai.md#external-ai-assistant) - Configure Sure's chat to use an external agent +- [Pipelock Security Proxy](pipelock.md) - Set up security scanning for MCP traffic +- [Model Context Protocol Specification](https://modelcontextprotocol.io/) - Official MCP documentation