mirror of
https://github.com/we-promise/sure.git
synced 2026-04-07 22:34:47 +00:00
* feat(helm): add Pipelock ConfigMap, scanning config, and consolidate compose - Add ConfigMap template rendering DLP, response scanning, MCP input/tool scanning, and forward proxy settings from values - Mount ConfigMap as /etc/pipelock/pipelock.yaml volume in deployment - Add checksum/config annotation for automatic pod restart on config change - Gate HTTPS_PROXY/HTTP_PROXY env injection on forwardProxy.enabled (skip in MCP-only mode) - Use hasKey for all boolean values to prevent Helm default swallowing false - Single source of truth for ports (forwardProxy.port/mcpProxy.port) - Pipelock-specific imagePullSecrets with fallback to app secrets - Merge standalone compose.example.pipelock.yml into compose.example.ai.yml - Add pipelock.example.yaml for Docker Compose users - Add exclude-paths to CI workflow for locale file false positives * Add external assistant support (OpenAI-compatible SSE proxy) Allow self-hosted instances to delegate chat to an external AI agent via an OpenAI-compatible streaming endpoint. Configurable per-family through Settings UI or ASSISTANT_TYPE env override. - Assistant::External::Client: SSE streaming HTTP client (no new gems) - Settings UI with type selector, env lock indicator, config status - Helm chart and Docker Compose env var support - 45 tests covering client, config, routing, controller, integration * Add session key routing, email allowlist, and config plumbing Route to the actual OpenClaw session via x-openclaw-session-key header instead of creating isolated sessions. Gate external assistant access behind an email allowlist (EXTERNAL_ASSISTANT_ALLOWED_EMAILS env var). Plumb session_key and allowedEmails through Helm chart, compose, and env template. * Add HTTPS_PROXY support to External::Client for Pipelock integration Net::HTTP does not auto-read HTTPS_PROXY/HTTP_PROXY env vars (unlike Faraday). Explicitly resolve proxy from environment in build_http so outbound traffic to the external assistant routes through Pipelock's forward proxy when enabled. Respects NO_PROXY for internal hosts. * Add UI fields for external assistant config (Setting-backed with env fallback) Follow the same pattern as OpenAI settings: database-backed Setting fields with env var defaults. Self-hosters can now configure the external assistant URL, token, and agent ID from the browser (Settings > Self-Hosting > AI Assistant) instead of requiring env vars. Fields disable when the corresponding env var is set. * Improve external assistant UI labels and add help text Change placeholder to generic OpenAI-compatible URL pattern. Add help text under each field explaining where the values come from: URL from agent provider, token for authentication, agent ID for multi-agent routing. * Add external assistant docs and fix URL help text Add External AI Assistant section to docs/hosting/ai.md covering setup (UI and env vars), how it works, Pipelock security scanning, access control, and Docker Compose example. Drop "chat completions" jargon from URL help text. * Harden external assistant: retry logic, disconnect UI, error handling, and test coverage - Add retry with backoff for transient network errors (no retry after streaming starts) - Add disconnect button with confirmation modal in self-hosting settings - Narrow rescue scope with fallback logging for unexpected errors - Safe cleanup of partial responses on stream interruption - Gate ai_available? on family assistant_type instead of OR-ing all providers - Truncate conversation history to last 20 messages - Proxy-aware HTTP client with NO_PROXY support - Sanitize protocol to use generic headers (X-Agent-Id, X-Session-Key) - Full test coverage for streaming, retries, proxy routing, config, and disconnect * Exclude external assistant client from Pipelock scan-diff False positive: `@token` instance variable flagged as "Credential in URL". Temporary workaround until Pipelock supports inline suppression. * Address review feedback: NO_PROXY boundary fix, SSE done flag, design tokens - Fix NO_PROXY matching to require domain boundary (exact match or .suffix), case-insensitive. Prevents badexample.com matching example.com. - Add done flag to SSE streaming so read_body stops after [DONE] - Move MAX_CONVERSATION_MESSAGES to class level - Use bg-success/bg-destructive design tokens for status indicators - Add rationale comment for pipelock scan exclusion - Update docs last-updated date * Address second round of review feedback - Allowlist email comparison is now case-insensitive and nil-safe - Cap SSE buffer at 1 MB to prevent memory blowup from malformed streams - Don't expose upstream HTTP response body in user-facing errors (log it instead) - Fix frozen string warning on buffer initialization - Fix "builtin" typo in docs (should be "built-in") * Protect completed responses from cleanup, sanitize error messages - Don't destroy a fully streamed assistant message if post-stream metadata update fails (only cleanup partial responses) - Log raw connection/HTTP errors internally, show generic messages to users to avoid leaking network/proxy details - Update test assertions for new error message wording * Fix SSE content guard and NO_PROXY test correctness Use nil check instead of present? for SSE delta content to preserve whitespace-only chunks (newlines, spaces) that can occur in code output. Fix NO_PROXY test to use HTTP_PROXY matching the http:// client URL so the proxy resolution and NO_PROXY bypass logic are actually exercised. * Forward proxy credentials to Net::HTTP Pass proxy_uri.user and proxy_uri.password to Net::HTTP.new so authenticated proxies (http://user:pass@host:port) work correctly. Without this, credentials parsed from the proxy URL were silently dropped. Nil values are safe as positional args when no creds exist. * Update pipelock integration to v0.3.1 with full scanning config Bump Helm image tag from 0.2.7 to 0.3.1. Add missing security sections to both the Helm ConfigMap and compose example config: mcp_tool_policy, mcp_session_binding, and tool_chain_detection. These protect the /mcp endpoint against tool injection, session hijacking, and multi-step exfiltration chains. Add version and mode fields to config files. Enable include_defaults for DLP and response scanning to merge user patterns with the 35 built-in patterns. Remove redundant --mode CLI flag from the Helm deployment template since mode is now in the config file. * Pipelock Helm hardening + docs for external assistant and pipelock Helm templates: - ServiceMonitor for Prometheus scraping on /metrics (proxy port) - Ingress template for MCP reverse proxy (external AI agent access) - PodDisruptionBudget with minAvailable/maxUnavailable mutual exclusion - topologySpreadConstraints on Deployment - Structured logging config (format, output, include_allowed/blocked) - extraConfig escape hatch for additional pipelock.yaml sections - requireForExternalAssistant guard (fails when assistant enabled without pipelock) - Component label on Service metadata for ServiceMonitor targeting - NOTES.txt pipelock section with health, access, security, metrics info - Bump pipelock image tag 0.3.1 -> 0.3.2 - Fix: rename _asserts.tpl -> asserts.tpl (Helm skipped _ prefixed file) Documentation: - Helm chart README: full Pipelock section - docs/hosting/pipelock.md: dedicated hosting guide (Docker + Kubernetes) - docs/hosting/docker.md: AI features section (external assistant, pipelock) - .env.example: external assistant and MCP env vars Infra: - Chart.lock pinning dependency versions - .gitignore for vendored subchart tarballs * Fix bot comments: quote ingress host, fix sidecar wording, add code block lang * Fail fast when pipelock ingress enabled with empty hosts * Fail fast when pipelock ingress host has empty paths * Messed up the conflict merge --------- Signed-off-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <juanjo.mata@gmail.com> Co-authored-by: Juan José Mata <jjmata@jjmata.com>
271 lines
10 KiB
Markdown
271 lines
10 KiB
Markdown
# Self Hosting Sure with Docker
|
|
|
|
This guide will help you setup, update, and maintain your self-hosted Sure application with Docker Compose. Docker Compose is the most popular and recommended way to self-host the Sure app.
|
|
|
|
## Setup Guide
|
|
|
|
Follow the guide below to get your app running.
|
|
|
|
### Step 1: Install Docker
|
|
|
|
Complete the following steps:
|
|
|
|
1. Install Docker Engine by following [the official guide](https://docs.docker.com/engine/install/)
|
|
2. Start the Docker service on your machine
|
|
3. Verify that Docker is installed correctly and is running by opening up a terminal and running the following command:
|
|
|
|
```bash
|
|
# If Docker is setup correctly, this command will succeed
|
|
docker run hello-world
|
|
```
|
|
|
|
### Step 2: Configure your Docker Compose file and environment
|
|
|
|
#### Create a directory for your app to run
|
|
|
|
Open your terminal and create a directory where your app will run. Below is an example command with a recommended directory:
|
|
|
|
```bash
|
|
# Create a directory on your computer for Docker files (name it whatever you like)
|
|
mkdir -p ~/docker-apps/sure
|
|
|
|
# Once created, navigate your current working directory to the new folder
|
|
cd ~/docker-apps/sure
|
|
```
|
|
|
|
#### Copy our sample Docker Compose file
|
|
|
|
Make sure you are in the directory you just created and run the following command:
|
|
|
|
```bash
|
|
# Download the sample compose.yml file from the GitHub repository
|
|
curl -o compose.yml https://raw.githubusercontent.com/we-promise/sure/main/compose.example.yml
|
|
```
|
|
|
|
This command will do the following:
|
|
|
|
1. Fetch the sample docker compose file from our public Github repository
|
|
2. Creates a file in your current directory called `compose.yml` with the contents of the example file
|
|
|
|
At this point, the only file in your current working directory should be `compose.yml`.
|
|
|
|
### Step 3 (optional): Configure your environment
|
|
|
|
By default, our `compose.example.yml` file runs without any configuration.
|
|
That said, if you would like extra security (important if you're running outside of a local network), you can follow the steps below to set things up.
|
|
|
|
If you're running the app locally and don't care much about security, you can skip this step.
|
|
|
|
#### Create your environment file
|
|
|
|
In order to configure the app, you will need to create a file called `.env`, which is where Docker will read environment variables from.
|
|
|
|
To do this, you should get our .env.example as a starting point:
|
|
|
|
```bash
|
|
curl -o .env https://raw.githubusercontent.com/we-promise/sure/main/.env.example
|
|
```
|
|
|
|
#### Generate the app secret key
|
|
|
|
The app requires an environment variable called `SECRET_KEY_BASE` to run.
|
|
|
|
We will first need to generate this in the terminal. If you have `openssl` installed on your computer, you can generate it with the following command:
|
|
|
|
```bash
|
|
openssl rand -hex 64
|
|
```
|
|
|
|
_Alternatively_, you can generate a key without openssl or any external dependencies by pasting the following bash command in your terminal and running it:
|
|
|
|
```bash
|
|
head -c 64 /dev/urandom | od -An -tx1 | tr -d ' \n' && echo
|
|
```
|
|
|
|
Once you have generated a key, save it and move on to the next step.
|
|
|
|
#### Fill in your environment file
|
|
|
|
Open the file named `.env` that we created in a prior step using your favorite text editor.
|
|
|
|
Fill in this file with the following variables:
|
|
|
|
```txt
|
|
SECRET_KEY_BASE="replacemewiththegeneratedstringfromthepriorstep"
|
|
POSTGRES_PASSWORD="replacemewithyourdesireddatabasepassword"
|
|
```
|
|
|
|
#### Using HTTPS
|
|
|
|
Assuming you want to access your instance from the internet, you should have secured your URL address with an SSL certificate.
|
|
The Docker instance runs in plain HTTP and you need to tell it that you are redirecting your HTTPS stream to the HTTP one.
|
|
To do this, edit the `compose.yml` file and find the line stating:
|
|
|
|
```yaml
|
|
RAILS_ASSUME_SSL: "false"
|
|
```
|
|
|
|
and change it to `true`
|
|
|
|
```yaml
|
|
RAILS_ASSUME_SSL: "true"
|
|
```
|
|
|
|
### Step 4: Run the app
|
|
|
|
You are now ready to run the app. Start with the following command to make sure everything is working:
|
|
|
|
```bash
|
|
docker compose up
|
|
```
|
|
|
|
This will pull our official Docker image and start the app. You will see logs in your terminal.
|
|
|
|
Open your browser, and navigate to `http://localhost:3000`.
|
|
|
|
If everything is working, you will see the Sure login screen.
|
|
|
|
### Step 5: Create your account
|
|
|
|
The first time you run the app, you will need to register a new account by hitting "create your account" on the login page.
|
|
|
|
1. Enter your email
|
|
2. Enter a password
|
|
|
|
### Step 6: Run the app in the background
|
|
|
|
Most self-hosting users will want the Sure app to run in the background on their computer so they can access it at all times. To do this, hit `Ctrl+C` to stop the running process, and then run the following command:
|
|
|
|
```bash
|
|
docker compose up -d
|
|
```
|
|
|
|
The `-d` flag will run Docker Compose in "detached" mode. To verify it is running, you can run the following command:
|
|
|
|
```
|
|
docker compose ls
|
|
```
|
|
|
|
### Step 7: Enjoy!
|
|
|
|
Your app is now set up. You can visit it at `http://localhost:3000` in your browser.
|
|
|
|
If you find bugs or have a feature request, be sure to read through our [contributing guide here](https://github.com/we-promise/sure/wiki/How-to-Contribute-Effectively-to-Sure).
|
|
|
|
## AI features, external assistant, and Pipelock
|
|
|
|
Sure ships with a separate compose file for AI-related features: `compose.example.ai.yml`. It adds:
|
|
|
|
- **Pipelock** (always on): AI agent security proxy that scans outbound LLM calls and inbound MCP traffic
|
|
- **Ollama + Open WebUI** (optional `--profile ai`): local LLM inference
|
|
|
|
### Using the AI compose file
|
|
|
|
```bash
|
|
# Download both compose files
|
|
curl -o compose.yml https://raw.githubusercontent.com/we-promise/sure/main/compose.example.yml
|
|
curl -o compose.ai.yml https://raw.githubusercontent.com/we-promise/sure/main/compose.example.ai.yml
|
|
curl -o pipelock.example.yaml https://raw.githubusercontent.com/we-promise/sure/main/pipelock.example.yaml
|
|
|
|
# Run with Pipelock (no local LLM)
|
|
docker compose -f compose.ai.yml up -d
|
|
|
|
# Run with Pipelock + Ollama
|
|
docker compose -f compose.ai.yml --profile ai up -d
|
|
```
|
|
|
|
### Setting up the external AI assistant
|
|
|
|
The external assistant delegates chat to a remote AI agent instead of calling LLMs directly. The agent calls back to Sure's `/mcp` endpoint for financial data (accounts, transactions, balance sheet).
|
|
|
|
1. Set the MCP endpoint credentials in your `.env`:
|
|
```bash
|
|
MCP_API_TOKEN=generate-a-random-token-here
|
|
MCP_USER_EMAIL=your@email.com # must match an existing Sure user
|
|
```
|
|
|
|
2. Set the external assistant connection:
|
|
```bash
|
|
EXTERNAL_ASSISTANT_URL=https://your-agent/v1/chat/completions
|
|
EXTERNAL_ASSISTANT_TOKEN=your-agent-api-token
|
|
```
|
|
|
|
3. Choose how to activate:
|
|
- **Per-family (UI):** Go to Settings > Self-Hosting > AI Assistant, select "External"
|
|
- **Global (env):** Set `ASSISTANT_TYPE=external` to force all families to use external
|
|
|
|
See [docs/hosting/ai.md](ai.md) for full configuration details including agent ID, session keys, and email allowlisting.
|
|
|
|
### Pipelock security proxy
|
|
|
|
Pipelock sits between Sure and external services, scanning AI traffic for:
|
|
|
|
- **Secret exfiltration** (DLP): catches API keys, tokens, or personal data leaking in prompts
|
|
- **Prompt injection**: detects attempts to override system instructions
|
|
- **Tool poisoning**: validates MCP tool calls against known-safe patterns
|
|
|
|
When using `compose.example.ai.yml`, Pipelock is always running. External AI agents should connect to port 8889 (MCP reverse proxy) instead of directly to Sure's `/mcp` on port 3000.
|
|
|
|
For full Pipelock configuration, see [docs/hosting/pipelock.md](pipelock.md).
|
|
|
|
## How to update your app
|
|
|
|
The mechanism that updates your self-hosted Sure app is the GHCR (Github Container Registry) Docker image that you see in the `compose.yml` file:
|
|
|
|
```yml
|
|
image: ghcr.io/we-promise/sure:latest
|
|
```
|
|
|
|
We recommend using one of the following images, but you can pin your app to whatever version you'd like (see [packages](https://github.com/we-promise/sure/pkgs/container/sure)):
|
|
|
|
- `ghcr.io/we-promise/sure:latest` (latest `alpha`)
|
|
- `ghcr.io/we-promise/sure:stable` (latest release)
|
|
|
|
By default, your app _will NOT_ automatically update. To update your self-hosted app, run the following commands in your terminal:
|
|
|
|
```bash
|
|
cd ~/docker-apps/sure # Navigate to whatever directory you configured the app in
|
|
docker compose pull # This pulls the "latest" published image from GHCR
|
|
docker compose build # This rebuilds the app with updates
|
|
docker compose up --no-deps -d web worker # This restarts the app using the newest version
|
|
```
|
|
|
|
## How to change which updates your app receives
|
|
|
|
If you'd like to pin the app to a specific version or tag, all you need to do is edit the `compose.yml` file:
|
|
|
|
```yml
|
|
image: ghcr.io/we-promise/sure:stable
|
|
```
|
|
|
|
After doing this, make sure and restart the app:
|
|
|
|
```bash
|
|
docker compose pull # This pulls the "latest" published image from GHCR
|
|
docker compose build # This rebuilds the app with updates
|
|
docker compose up --no-deps -d web worker # This restarts the app using the newest version
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### ActiveRecord::DatabaseConnectionError
|
|
|
|
If you are trying to get Sure started for the **first time** and run into database connection issues, it is likely because Docker has already initialized the Postgres database with a _different_ default role (usually from a previous attempt to start the app).
|
|
|
|
If you run into this issue, you can optionally **reset the database**.
|
|
|
|
**PLEASE NOTE: this will delete any existing data that you have in your Sure database, so proceed with caution.** For first-time users of the app just trying to get started, you're generally safe to run the commands below.
|
|
|
|
By running the commands below, you will delete your existing Sure database and "reset" it.
|
|
|
|
```
|
|
docker compose down
|
|
docker volume rm sure_postgres-data # this is the name of the volume the DB is mounted to
|
|
docker compose up
|
|
docker compose exec db psql -U sure_user -d sure_development -c "SELECT 1;" # This will verify that the issue is fixed
|
|
```
|
|
|
|
### Slow `.csv` import (processing rows taking longer than expected)
|
|
|
|
Importing comma-separated-value file(s) requires the `sure-worker` container to communicate with Redis. Check your worker logs for any unexpected errors, such as connection timeouts or Redis communication failures.
|