From a53a131c466ee6151f6e9411673e6e7883f890db Mon Sep 17 00:00:00 2001 From: LPW Date: Tue, 3 Mar 2026 10:32:35 -0500 Subject: [PATCH] Add Pipelock operational templates, docs, and config hardening (#1102) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(helm): add Pipelock ConfigMap, scanning config, and consolidate compose - Add ConfigMap template rendering DLP, response scanning, MCP input/tool scanning, and forward proxy settings from values - Mount ConfigMap as /etc/pipelock/pipelock.yaml volume in deployment - Add checksum/config annotation for automatic pod restart on config change - Gate HTTPS_PROXY/HTTP_PROXY env injection on forwardProxy.enabled (skip in MCP-only mode) - Use hasKey for all boolean values to prevent Helm default swallowing false - Single source of truth for ports (forwardProxy.port/mcpProxy.port) - Pipelock-specific imagePullSecrets with fallback to app secrets - Merge standalone compose.example.pipelock.yml into compose.example.ai.yml - Add pipelock.example.yaml for Docker Compose users - Add exclude-paths to CI workflow for locale file false positives * Add external assistant support (OpenAI-compatible SSE proxy) Allow self-hosted instances to delegate chat to an external AI agent via an OpenAI-compatible streaming endpoint. Configurable per-family through Settings UI or ASSISTANT_TYPE env override. - Assistant::External::Client: SSE streaming HTTP client (no new gems) - Settings UI with type selector, env lock indicator, config status - Helm chart and Docker Compose env var support - 45 tests covering client, config, routing, controller, integration * Add session key routing, email allowlist, and config plumbing Route to the actual OpenClaw session via x-openclaw-session-key header instead of creating isolated sessions. Gate external assistant access behind an email allowlist (EXTERNAL_ASSISTANT_ALLOWED_EMAILS env var). Plumb session_key and allowedEmails through Helm chart, compose, and env template. * Add HTTPS_PROXY support to External::Client for Pipelock integration Net::HTTP does not auto-read HTTPS_PROXY/HTTP_PROXY env vars (unlike Faraday). Explicitly resolve proxy from environment in build_http so outbound traffic to the external assistant routes through Pipelock's forward proxy when enabled. Respects NO_PROXY for internal hosts. * Add UI fields for external assistant config (Setting-backed with env fallback) Follow the same pattern as OpenAI settings: database-backed Setting fields with env var defaults. Self-hosters can now configure the external assistant URL, token, and agent ID from the browser (Settings > Self-Hosting > AI Assistant) instead of requiring env vars. Fields disable when the corresponding env var is set. * Improve external assistant UI labels and add help text Change placeholder to generic OpenAI-compatible URL pattern. Add help text under each field explaining where the values come from: URL from agent provider, token for authentication, agent ID for multi-agent routing. * Add external assistant docs and fix URL help text Add External AI Assistant section to docs/hosting/ai.md covering setup (UI and env vars), how it works, Pipelock security scanning, access control, and Docker Compose example. Drop "chat completions" jargon from URL help text. * Harden external assistant: retry logic, disconnect UI, error handling, and test coverage - Add retry with backoff for transient network errors (no retry after streaming starts) - Add disconnect button with confirmation modal in self-hosting settings - Narrow rescue scope with fallback logging for unexpected errors - Safe cleanup of partial responses on stream interruption - Gate ai_available? on family assistant_type instead of OR-ing all providers - Truncate conversation history to last 20 messages - Proxy-aware HTTP client with NO_PROXY support - Sanitize protocol to use generic headers (X-Agent-Id, X-Session-Key) - Full test coverage for streaming, retries, proxy routing, config, and disconnect * Exclude external assistant client from Pipelock scan-diff False positive: `@token` instance variable flagged as "Credential in URL". Temporary workaround until Pipelock supports inline suppression. * Address review feedback: NO_PROXY boundary fix, SSE done flag, design tokens - Fix NO_PROXY matching to require domain boundary (exact match or .suffix), case-insensitive. Prevents badexample.com matching example.com. - Add done flag to SSE streaming so read_body stops after [DONE] - Move MAX_CONVERSATION_MESSAGES to class level - Use bg-success/bg-destructive design tokens for status indicators - Add rationale comment for pipelock scan exclusion - Update docs last-updated date * Address second round of review feedback - Allowlist email comparison is now case-insensitive and nil-safe - Cap SSE buffer at 1 MB to prevent memory blowup from malformed streams - Don't expose upstream HTTP response body in user-facing errors (log it instead) - Fix frozen string warning on buffer initialization - Fix "builtin" typo in docs (should be "built-in") * Protect completed responses from cleanup, sanitize error messages - Don't destroy a fully streamed assistant message if post-stream metadata update fails (only cleanup partial responses) - Log raw connection/HTTP errors internally, show generic messages to users to avoid leaking network/proxy details - Update test assertions for new error message wording * Fix SSE content guard and NO_PROXY test correctness Use nil check instead of present? for SSE delta content to preserve whitespace-only chunks (newlines, spaces) that can occur in code output. Fix NO_PROXY test to use HTTP_PROXY matching the http:// client URL so the proxy resolution and NO_PROXY bypass logic are actually exercised. * Forward proxy credentials to Net::HTTP Pass proxy_uri.user and proxy_uri.password to Net::HTTP.new so authenticated proxies (http://user:pass@host:port) work correctly. Without this, credentials parsed from the proxy URL were silently dropped. Nil values are safe as positional args when no creds exist. * Update pipelock integration to v0.3.1 with full scanning config Bump Helm image tag from 0.2.7 to 0.3.1. Add missing security sections to both the Helm ConfigMap and compose example config: mcp_tool_policy, mcp_session_binding, and tool_chain_detection. These protect the /mcp endpoint against tool injection, session hijacking, and multi-step exfiltration chains. Add version and mode fields to config files. Enable include_defaults for DLP and response scanning to merge user patterns with the 35 built-in patterns. Remove redundant --mode CLI flag from the Helm deployment template since mode is now in the config file. * Pipelock Helm hardening + docs for external assistant and pipelock Helm templates: - ServiceMonitor for Prometheus scraping on /metrics (proxy port) - Ingress template for MCP reverse proxy (external AI agent access) - PodDisruptionBudget with minAvailable/maxUnavailable mutual exclusion - topologySpreadConstraints on Deployment - Structured logging config (format, output, include_allowed/blocked) - extraConfig escape hatch for additional pipelock.yaml sections - requireForExternalAssistant guard (fails when assistant enabled without pipelock) - Component label on Service metadata for ServiceMonitor targeting - NOTES.txt pipelock section with health, access, security, metrics info - Bump pipelock image tag 0.3.1 -> 0.3.2 - Fix: rename _asserts.tpl -> asserts.tpl (Helm skipped _ prefixed file) Documentation: - Helm chart README: full Pipelock section - docs/hosting/pipelock.md: dedicated hosting guide (Docker + Kubernetes) - docs/hosting/docker.md: AI features section (external assistant, pipelock) - .env.example: external assistant and MCP env vars Infra: - Chart.lock pinning dependency versions - .gitignore for vendored subchart tarballs * Fix bot comments: quote ingress host, fix sidecar wording, add code block lang * Fail fast when pipelock ingress enabled with empty hosts * Fail fast when pipelock ingress host has empty paths * Messed up the conflict merge --------- Signed-off-by: Juan José Mata Co-authored-by: Juan José Mata Co-authored-by: Juan José Mata --- .env.example | 15 ++ charts/sure/.gitignore | 2 + charts/sure/CHANGELOG.md | 16 +- charts/sure/Chart.lock | 9 + charts/sure/README.md | 108 +++++++++ charts/sure/templates/NOTES.txt | 35 ++- charts/sure/templates/_asserts.tpl | 7 - charts/sure/templates/asserts.tpl | 23 ++ charts/sure/templates/pipelock-configmap.yaml | 22 ++ .../sure/templates/pipelock-deployment.yaml | 2 + charts/sure/templates/pipelock-ingress.yaml | 42 ++++ charts/sure/templates/pipelock-pdb.yaml | 21 ++ charts/sure/templates/pipelock-service.yaml | 1 + .../templates/pipelock-servicemonitor.yaml | 21 ++ charts/sure/values.yaml | 51 +++- docs/hosting/docker.md | 56 +++++ docs/hosting/pipelock.md | 219 ++++++++++++++++++ 17 files changed, 640 insertions(+), 10 deletions(-) create mode 100644 charts/sure/.gitignore create mode 100644 charts/sure/Chart.lock delete mode 100644 charts/sure/templates/_asserts.tpl create mode 100644 charts/sure/templates/asserts.tpl create mode 100644 charts/sure/templates/pipelock-ingress.yaml create mode 100644 charts/sure/templates/pipelock-pdb.yaml create mode 100644 charts/sure/templates/pipelock-servicemonitor.yaml create mode 100644 docs/hosting/pipelock.md diff --git a/.env.example b/.env.example index 548bd1971..2928bcc1c 100644 --- a/.env.example +++ b/.env.example @@ -25,6 +25,21 @@ OPENAI_ACCESS_TOKEN= OPENAI_MODEL= OPENAI_URI_BASE= +# Optional: External AI Assistant — delegates chat to a remote AI agent +# instead of calling LLMs directly. The agent calls back to Sure's /mcp endpoint. +# See docs/hosting/ai.md for full details. +# ASSISTANT_TYPE=external +# EXTERNAL_ASSISTANT_URL=https://your-agent-host/v1/chat/completions +# EXTERNAL_ASSISTANT_TOKEN=your-api-token +# EXTERNAL_ASSISTANT_AGENT_ID=main +# EXTERNAL_ASSISTANT_SESSION_KEY=agent:main:main +# EXTERNAL_ASSISTANT_ALLOWED_EMAILS=user@example.com + +# Optional: MCP server endpoint — enables /mcp for external AI assistants. +# Both values are required. MCP_USER_EMAIL must match an existing user's email. +# MCP_API_TOKEN=your-random-bearer-token +# MCP_USER_EMAIL=user@example.com + # Optional: Langfuse config LANGFUSE_HOST=https://cloud.langfuse.com LANGFUSE_PUBLIC_KEY= diff --git a/charts/sure/.gitignore b/charts/sure/.gitignore new file mode 100644 index 000000000..58f68018c --- /dev/null +++ b/charts/sure/.gitignore @@ -0,0 +1,2 @@ +# Vendored subchart tarballs (regenerated by `helm dependency build`) +charts/ diff --git a/charts/sure/CHANGELOG.md b/charts/sure/CHANGELOG.md index 7aa613010..0d1842c6b 100644 --- a/charts/sure/CHANGELOG.md +++ b/charts/sure/CHANGELOG.md @@ -5,7 +5,7 @@ All notable changes to the Sure Helm chart will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). -## [0.6.9-alpha] - 2026-03-01 +## [0.6.9-alpha] - 2026-03-02 ### Added - **Pipelock security proxy** (`pipelock.enabled=true`): Separate Deployment + Service that provides two scanning layers @@ -20,11 +20,25 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Boolean safety: uses `hasKey` to prevent Helm's `default` from swallowing explicit `false` - Configurable ports via `forwardProxy.port` and `mcpProxy.port` (single source of truth across Service, Deployment, and env vars) - `pipelock.example.yaml` reference config for Docker Compose deployments +- **Pipelock operational hardening**: + - `pipelock.serviceMonitor`: Prometheus Operator ServiceMonitor for /metrics on the proxy port + - `pipelock.ingress`: Ingress template for MCP reverse proxy (external AI assistant access in k8s) + - `pipelock.pdb`: PodDisruptionBudget with minAvailable/maxUnavailable mutual exclusion guard + - `pipelock.topologySpreadConstraints`: Pod spread across nodes + - `pipelock.logging`: Structured logging config (format, output, include_allowed, include_blocked) + - `pipelock.extraConfig`: Escape hatch for additional pipelock.yaml config sections + - `pipelock.requireForExternalAssistant`: Helm guard that fails when externalAssistant is enabled without pipelock + - Component label (`app.kubernetes.io/component: pipelock`) on Service metadata for selector targeting + - NOTES.txt: Pipelock health check commands, MCP access info, security notes, metrics status ### Changed +- Bumped `pipelock.image.tag` from `0.3.1` to `0.3.2` - Consolidated `compose.example.pipelock.yml` into `compose.example.ai.yml` — Pipelock now runs alongside Ollama in one compose file with health checks, config volume mount, and MCP env vars (`MCP_API_TOKEN`, `MCP_USER_EMAIL`) - CI: Pipelock scan `fail-on-findings` changed from `false` to `true`; added `exclude-paths` for locale help text false positives +### Fixed +- Renamed `_asserts.tpl` to `asserts.tpl` — Helm's `_` prefix convention prevented guards from executing + ## [0.6.7-alpha] - 2026-01-10 ### Added diff --git a/charts/sure/Chart.lock b/charts/sure/Chart.lock new file mode 100644 index 000000000..5cdb2cfcc --- /dev/null +++ b/charts/sure/Chart.lock @@ -0,0 +1,9 @@ +dependencies: +- name: cloudnative-pg + repository: https://cloudnative-pg.github.io/charts + version: 0.27.1 +- name: redis-operator + repository: https://ot-container-kit.github.io/helm-charts + version: 0.23.0 +digest: sha256:5ffa5c535cb5feea62a29665045a79da8a5d058c3ba11c4db37a4afa97563e3e +generated: "2026-03-02T21:16:32.757224371-05:00" diff --git a/charts/sure/README.md b/charts/sure/README.md index 6a004d153..bf30f3071 100644 --- a/charts/sure/README.md +++ b/charts/sure/README.md @@ -12,6 +12,7 @@ Official Helm chart for deploying the Sure Rails application on Kubernetes. It s - Optional subcharts - CloudNativePG (operator) + Cluster CR for PostgreSQL with HA support - OT-CONTAINER-KIT redis-operator for Redis HA (replication by default, optional Sentinel) +- Optional Pipelock AI agent security proxy (forward proxy + MCP reverse proxy with DLP, prompt injection, and tool poisoning detection) - Security best practices: runAsNonRoot, readOnlyRootFilesystem, optional existingSecret, no hardcoded secrets - Scalability - Replicas (web/worker), resources, topology spread constraints @@ -637,6 +638,112 @@ hpa: targetCPUUtilizationPercentage: 70 ``` +## Pipelock (AI agent security proxy) + +Pipelock is an optional sidecar that scans AI agent traffic for secret exfiltration, prompt injection, and tool poisoning. It runs as a separate Deployment with two listeners: + +- **Forward proxy** (port 8888): Scans outbound HTTPS from Faraday-based AI clients. Auto-injected via `HTTPS_PROXY` env vars when enabled. +- **MCP reverse proxy** (port 8889): Scans inbound MCP traffic from external AI assistants. + +### Enabling Pipelock + +```yaml +pipelock: + enabled: true + image: + tag: "0.3.2" + mode: balanced # strict, balanced, or audit +``` + +### Exposing MCP to external AI assistants + +When running in Kubernetes, external AI agents need network access to the MCP reverse proxy port. Enable the Pipelock Ingress: + +```yaml +pipelock: + enabled: true + ingress: + enabled: true + className: nginx + annotations: + cert-manager.io/cluster-issuer: letsencrypt + hosts: + - host: pipelock.example.com + paths: + - path: / + pathType: Prefix + tls: + - hosts: [pipelock.example.com] + secretName: pipelock-tls +``` + +Security: The Ingress routes to port `mcp` (8889). Ensure `MCP_API_TOKEN` is set so the MCP endpoint requires authentication. The Ingress itself does not add auth. + +### Metrics (Prometheus) + +Pipelock exposes `/metrics` on the forward proxy port. Enable scraping with a ServiceMonitor: + +```yaml +pipelock: + serviceMonitor: + enabled: true + interval: 30s + portName: proxy # matches Service port name for 8888 + additionalLabels: + release: prometheus # match your Prometheus Operator selector +``` + +### PodDisruptionBudget + +Protect Pipelock from node drains: + +```yaml +pipelock: + pdb: + enabled: true + maxUnavailable: 1 # safe for single-replica; use minAvailable when replicas > 1 +``` + +Note: Setting `minAvailable` with `replicas=1` blocks eviction entirely. Use `maxUnavailable` for single-replica deployments. + +### Structured logging + +```yaml +pipelock: + logging: + format: json # json or text + output: stdout + includeAllowed: false + includeBlocked: true +``` + +### Extra config (escape hatch) + +For Pipelock config sections not covered by structured values (session profiling, data budgets, kill switch, etc.), use `extraConfig`: + +```yaml +pipelock: + extraConfig: + session_profiling: + enabled: true + max_sessions: 1000 + data_budget: + max_bytes_per_session: 10485760 +``` + +These are appended verbatim to `pipelock.yaml`. Do not duplicate keys already rendered by the chart. + +### Requiring Pipelock for external assistants + +To enforce that Pipelock is enabled whenever the external AI assistant feature is active: + +```yaml +pipelock: + requireForExternalAssistant: true +``` + +This causes `helm template` / `helm install` to fail if `rails.externalAssistant.enabled=true` and `pipelock.enabled=false`. Note: this only guards the `externalAssistant` path. Direct MCP access via `MCP_API_TOKEN` is configured through env vars and not detectable from Helm values. + ## Security Notes - Never commit secrets in `values.yaml`. Use `rails.existingSecret` or a tool like Sealed Secrets. @@ -660,6 +767,7 @@ See `values.yaml` for the complete configuration surface, including: - `migrations.*`: strategy job or initContainer - `simplefin.encryption.*`: enable + backfill options - `cronjobs.*`: custom CronJobs +- `pipelock.*`: AI agent security proxy (forward proxy, MCP reverse proxy, DLP, injection scanning, logging, serviceMonitor, ingress, PDB, extraConfig) - `service.*`, `ingress.*`, `serviceMonitor.*`, `hpa.*` ## Helm tests diff --git a/charts/sure/templates/NOTES.txt b/charts/sure/templates/NOTES.txt index f4c340b9d..e9e46850f 100644 --- a/charts/sure/templates/NOTES.txt +++ b/charts/sure/templates/NOTES.txt @@ -41,7 +41,40 @@ Troubleshooting - For CloudNativePG, verify the RW service exists and the primary is Ready. - For redis-operator, verify the RedisSentinel CR reports Ready and that the master service resolves. +{{- if .Values.pipelock.enabled }} + +Pipelock (AI agent security proxy) +----------------------------------- +5) Verify pipelock is running: + kubectl rollout status deploy/{{ include "sure.fullname" . }}-pipelock -n {{ .Release.Namespace }} + kubectl logs deploy/{{ include "sure.fullname" . }}-pipelock -n {{ .Release.Namespace }} --tail=20 + +6) MCP access for external AI assistants: +{{- if .Values.pipelock.ingress.enabled }} +{{- range .Values.pipelock.ingress.hosts }} + - Ingress: http{{ if $.Values.pipelock.ingress.tls }}s{{ end }}://{{ .host }} +{{- end }} +{{- else }} + - No Ingress configured. Port-forward for local access: + kubectl port-forward -n {{ .Release.Namespace }} svc/{{ include "sure.fullname" . }}-pipelock 8889:{{ .Values.pipelock.mcpProxy.port | default 8889 }} +{{- end }} + + Security: Enable TLS on the pipelock Ingress and ensure MCP_API_TOKEN is set. + The MCP endpoint requires authentication but the Ingress does not add it. + +7) Metrics: +{{- if .Values.pipelock.serviceMonitor.enabled }} + - ServiceMonitor enabled — Prometheus will scrape /metrics on port {{ .Values.pipelock.serviceMonitor.portName }}. +{{- else }} + - ServiceMonitor not enabled. Metrics are available at http://:{{ .Values.pipelock.forwardProxy.port | default 8888 }}/metrics + Enable with: pipelock.serviceMonitor.enabled=true +{{- end }} +{{- end }} + Security reminder ----------------- - For production, prefer immutable image tags (for example, image.tag=v1.2.3) instead of 'latest'. -- Provide secrets via an existing Kubernetes Secret or a secret manager (External Secrets, Sealed Secrets). \ No newline at end of file +- Provide secrets via an existing Kubernetes Secret or a secret manager (External Secrets, Sealed Secrets). +{{- if .Values.pipelock.enabled }} +- When exposing MCP to external AI assistants, always enable pipelock to scan inbound traffic. +{{- end }} \ No newline at end of file diff --git a/charts/sure/templates/_asserts.tpl b/charts/sure/templates/_asserts.tpl deleted file mode 100644 index de1cf0bbb..000000000 --- a/charts/sure/templates/_asserts.tpl +++ /dev/null @@ -1,7 +0,0 @@ -{{/* -Mutual exclusivity and configuration guards -*/}} - -{{- if and .Values.redisOperator.managed.enabled .Values.redisSimple.enabled -}} -{{- fail "Invalid configuration: Both redisOperator.managed.enabled and redisSimple.enabled are true. Enable only one in-cluster Redis provider." -}} -{{- end -}} diff --git a/charts/sure/templates/asserts.tpl b/charts/sure/templates/asserts.tpl new file mode 100644 index 000000000..1d481c0e9 --- /dev/null +++ b/charts/sure/templates/asserts.tpl @@ -0,0 +1,23 @@ +{{/* +Mutual exclusivity and configuration guards +*/}} + +{{- if and .Values.redisOperator.managed.enabled .Values.redisSimple.enabled -}} +{{- fail "Invalid configuration: Both redisOperator.managed.enabled and redisSimple.enabled are true. Enable only one in-cluster Redis provider." -}} +{{- end -}} + +{{- $extEnabled := false -}} +{{- if .Values.rails -}}{{- if .Values.rails.externalAssistant -}}{{- if .Values.rails.externalAssistant.enabled -}} +{{- $extEnabled = true -}} +{{- end -}}{{- end -}}{{- end -}} +{{- $plEnabled := false -}} +{{- if .Values.pipelock -}}{{- if .Values.pipelock.enabled -}} +{{- $plEnabled = true -}} +{{- end -}}{{- end -}} +{{- $requirePL := false -}} +{{- if .Values.pipelock -}}{{- if .Values.pipelock.requireForExternalAssistant -}} +{{- $requirePL = true -}} +{{- end -}}{{- end -}} +{{- if and $extEnabled (not $plEnabled) $requirePL -}} +{{- fail "pipelock.requireForExternalAssistant is true but pipelock.enabled is false. Enable pipelock (pipelock.enabled=true) when using rails.externalAssistant, or set pipelock.requireForExternalAssistant=false." -}} +{{- end -}} diff --git a/charts/sure/templates/pipelock-configmap.yaml b/charts/sure/templates/pipelock-configmap.yaml index 7b39a6726..8962d67fb 100644 --- a/charts/sure/templates/pipelock-configmap.yaml +++ b/charts/sure/templates/pipelock-configmap.yaml @@ -64,6 +64,20 @@ {{- $chainAction = .Values.pipelock.toolChainDetection.action | default "warn" -}} {{- $chainWindow = int (.Values.pipelock.toolChainDetection.windowSize | default 20) -}} {{- $chainGap = int (.Values.pipelock.toolChainDetection.maxGap | default 3) -}} +{{- end -}} +{{- $logFormat := "json" -}} +{{- $logOutput := "stdout" -}} +{{- $logIncludeAllowed := false -}} +{{- $logIncludeBlocked := true -}} +{{- if .Values.pipelock.logging -}} +{{- $logFormat = .Values.pipelock.logging.format | default "json" -}} +{{- $logOutput = .Values.pipelock.logging.output | default "stdout" -}} +{{- if hasKey .Values.pipelock.logging "includeAllowed" -}} +{{- $logIncludeAllowed = .Values.pipelock.logging.includeAllowed -}} +{{- end -}} +{{- if hasKey .Values.pipelock.logging "includeBlocked" -}} +{{- $logIncludeBlocked = .Values.pipelock.logging.includeBlocked -}} +{{- end -}} {{- end }} apiVersion: v1 kind: ConfigMap @@ -116,4 +130,12 @@ data: action: {{ $chainAction }} window_size: {{ $chainWindow }} max_gap: {{ $chainGap }} + logging: + format: {{ $logFormat }} + output: {{ $logOutput }} + include_allowed: {{ $logIncludeAllowed }} + include_blocked: {{ $logIncludeBlocked }} +{{- if .Values.pipelock.extraConfig }} + {{- toYaml .Values.pipelock.extraConfig | nindent 4 }} +{{- end }} {{- end }} diff --git a/charts/sure/templates/pipelock-deployment.yaml b/charts/sure/templates/pipelock-deployment.yaml index 99732fb0c..bf57ec8ab 100644 --- a/charts/sure/templates/pipelock-deployment.yaml +++ b/charts/sure/templates/pipelock-deployment.yaml @@ -96,4 +96,6 @@ spec: {{- toYaml (.Values.pipelock.affinity | default dict) | nindent 8 }} tolerations: {{- toYaml (.Values.pipelock.tolerations | default list) | nindent 8 }} + topologySpreadConstraints: + {{- toYaml (.Values.pipelock.topologySpreadConstraints | default (list)) | nindent 8 }} {{- end }} diff --git a/charts/sure/templates/pipelock-ingress.yaml b/charts/sure/templates/pipelock-ingress.yaml new file mode 100644 index 000000000..49c3e7ef8 --- /dev/null +++ b/charts/sure/templates/pipelock-ingress.yaml @@ -0,0 +1,42 @@ +{{- if and .Values.pipelock.enabled .Values.pipelock.ingress.enabled }} +apiVersion: networking.k8s.io/v1 +kind: Ingress +metadata: + name: {{ include "sure.fullname" . }}-pipelock + labels: + {{- include "sure.labels" . | nindent 4 }} + {{- with .Values.pipelock.ingress.annotations }} + annotations: + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + {{- if .Values.pipelock.ingress.className }} + ingressClassName: {{ .Values.pipelock.ingress.className }} + {{- end }} + {{- if .Values.pipelock.ingress.hosts }} + rules: + {{- range .Values.pipelock.ingress.hosts }} + {{- if not .paths }} + {{- fail "each entry in pipelock.ingress.hosts must include at least one paths item" }} + {{- end }} + - host: {{ .host | quote }} + http: + paths: + {{- range .paths }} + - path: {{ .path }} + pathType: {{ .pathType }} + backend: + service: + name: {{ include "sure.fullname" $ }}-pipelock + port: + name: mcp + {{- end }} + {{- end }} + {{- else }} + {{- fail "pipelock.ingress.enabled=true requires at least one entry in pipelock.ingress.hosts" }} + {{- end }} + {{- if .Values.pipelock.ingress.tls }} + tls: + {{- toYaml .Values.pipelock.ingress.tls | nindent 4 }} + {{- end }} +{{- end }} diff --git a/charts/sure/templates/pipelock-pdb.yaml b/charts/sure/templates/pipelock-pdb.yaml new file mode 100644 index 000000000..59f7da34a --- /dev/null +++ b/charts/sure/templates/pipelock-pdb.yaml @@ -0,0 +1,21 @@ +{{- if and .Values.pipelock.enabled .Values.pipelock.pdb.enabled }} +{{- if and .Values.pipelock.pdb.minAvailable .Values.pipelock.pdb.maxUnavailable }} +{{- fail "pipelock.pdb: set either minAvailable or maxUnavailable, not both." -}} +{{- end }} +apiVersion: policy/v1 +kind: PodDisruptionBudget +metadata: + name: {{ include "sure.fullname" . }}-pipelock + labels: + {{- include "sure.labels" . | nindent 4 }} +spec: + {{- if .Values.pipelock.pdb.minAvailable }} + minAvailable: {{ .Values.pipelock.pdb.minAvailable }} + {{- else if .Values.pipelock.pdb.maxUnavailable }} + maxUnavailable: {{ .Values.pipelock.pdb.maxUnavailable }} + {{- end }} + selector: + matchLabels: + app.kubernetes.io/component: pipelock + {{- include "sure.selectorLabels" . | nindent 6 }} +{{- end }} diff --git a/charts/sure/templates/pipelock-service.yaml b/charts/sure/templates/pipelock-service.yaml index 01be758c7..c20cac0a4 100644 --- a/charts/sure/templates/pipelock-service.yaml +++ b/charts/sure/templates/pipelock-service.yaml @@ -13,6 +13,7 @@ metadata: name: {{ include "sure.fullname" . }}-pipelock labels: {{- include "sure.labels" . | nindent 4 }} + app.kubernetes.io/component: pipelock spec: type: {{ (.Values.pipelock.service).type | default "ClusterIP" }} selector: diff --git a/charts/sure/templates/pipelock-servicemonitor.yaml b/charts/sure/templates/pipelock-servicemonitor.yaml new file mode 100644 index 000000000..dfe2d2c54 --- /dev/null +++ b/charts/sure/templates/pipelock-servicemonitor.yaml @@ -0,0 +1,21 @@ +{{- if and .Values.pipelock.enabled .Values.pipelock.serviceMonitor.enabled }} +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: {{ include "sure.fullname" . }}-pipelock + labels: + {{- include "sure.labels" . | nindent 4 }} + {{- with .Values.pipelock.serviceMonitor.additionalLabels }} + {{- toYaml . | nindent 4 }} + {{- end }} +spec: + selector: + matchLabels: + app.kubernetes.io/component: pipelock + {{- include "sure.selectorLabels" . | nindent 6 }} + endpoints: + - interval: {{ .Values.pipelock.serviceMonitor.interval }} + scrapeTimeout: {{ .Values.pipelock.serviceMonitor.scrapeTimeout }} + path: {{ .Values.pipelock.serviceMonitor.path }} + port: {{ .Values.pipelock.serviceMonitor.portName }} +{{- end }} diff --git a/charts/sure/values.yaml b/charts/sure/values.yaml index ae3b34b3d..f6d473044 100644 --- a/charts/sure/values.yaml +++ b/charts/sure/values.yaml @@ -488,7 +488,7 @@ pipelock: enabled: false image: repository: ghcr.io/luckypipewrench/pipelock - tag: "0.3.1" + tag: "0.3.2" pullPolicy: IfNotPresent imagePullSecrets: [] replicas: 1 @@ -541,3 +541,52 @@ pipelock: nodeSelector: {} tolerations: [] affinity: {} + topologySpreadConstraints: [] + + # Prometheus Operator ServiceMonitor for /metrics on the proxy port + serviceMonitor: + enabled: false + interval: 30s + scrapeTimeout: 10s + path: /metrics + portName: proxy # matches Service port name "proxy" (8888) + additionalLabels: {} + + # Ingress for MCP reverse proxy (port 8889) — external AI assistants need this in k8s + ingress: + enabled: false + className: "" + annotations: {} + hosts: + - host: pipelock.local + paths: + - path: / + pathType: Prefix + tls: [] + + # PodDisruptionBudget — protects pipelock during node drains. + # WARNING: minAvailable with replicas=1 blocks eviction entirely. + # Use maxUnavailable: 1 for single-replica deployments, or increase replicas. + pdb: + enabled: false + minAvailable: "" # set to 1 when replicas > 1 + maxUnavailable: 1 # safe default: allows 1 pod to be evicted + + # Structured logging for k8s log aggregation + logging: + format: json + output: stdout + includeAllowed: false + includeBlocked: true + + # Escape hatch: ADDITIONAL config sections appended to pipelock.yaml. + # Use for sections not covered by structured values above (session_profiling, + # data_budget, adaptive_enforcement, kill_switch, internal CIDRs, etc.) + # Do NOT duplicate keys already rendered above — behavior is parser-dependent. + extraConfig: {} + + # Hard-fail helm template when externalAssistant is enabled without pipelock. + # NOTE: This only guards the rails.externalAssistant path. Direct MCP access + # (/mcp endpoint with MCP_API_TOKEN) is not detectable from Helm values. + # For full coverage, also ensure pipelock is enabled whenever MCP_API_TOKEN is set. + requireForExternalAssistant: false diff --git a/docs/hosting/docker.md b/docs/hosting/docker.md index 8fd6a25cf..09114ed09 100644 --- a/docs/hosting/docker.md +++ b/docs/hosting/docker.md @@ -152,6 +152,62 @@ Your app is now set up. You can visit it at `http://localhost:3000` in your brow If you find bugs or have a feature request, be sure to read through our [contributing guide here](https://github.com/we-promise/sure/wiki/How-to-Contribute-Effectively-to-Sure). +## AI features, external assistant, and Pipelock + +Sure ships with a separate compose file for AI-related features: `compose.example.ai.yml`. It adds: + +- **Pipelock** (always on): AI agent security proxy that scans outbound LLM calls and inbound MCP traffic +- **Ollama + Open WebUI** (optional `--profile ai`): local LLM inference + +### Using the AI compose file + +```bash +# Download both compose files +curl -o compose.yml https://raw.githubusercontent.com/we-promise/sure/main/compose.example.yml +curl -o compose.ai.yml https://raw.githubusercontent.com/we-promise/sure/main/compose.example.ai.yml +curl -o pipelock.example.yaml https://raw.githubusercontent.com/we-promise/sure/main/pipelock.example.yaml + +# Run with Pipelock (no local LLM) +docker compose -f compose.ai.yml up -d + +# Run with Pipelock + Ollama +docker compose -f compose.ai.yml --profile ai up -d +``` + +### Setting up the external AI assistant + +The external assistant delegates chat to a remote AI agent instead of calling LLMs directly. The agent calls back to Sure's `/mcp` endpoint for financial data (accounts, transactions, balance sheet). + +1. Set the MCP endpoint credentials in your `.env`: + ```bash + MCP_API_TOKEN=generate-a-random-token-here + MCP_USER_EMAIL=your@email.com # must match an existing Sure user + ``` + +2. Set the external assistant connection: + ```bash + EXTERNAL_ASSISTANT_URL=https://your-agent/v1/chat/completions + EXTERNAL_ASSISTANT_TOKEN=your-agent-api-token + ``` + +3. Choose how to activate: + - **Per-family (UI):** Go to Settings > Self-Hosting > AI Assistant, select "External" + - **Global (env):** Set `ASSISTANT_TYPE=external` to force all families to use external + +See [docs/hosting/ai.md](ai.md) for full configuration details including agent ID, session keys, and email allowlisting. + +### Pipelock security proxy + +Pipelock sits between Sure and external services, scanning AI traffic for: + +- **Secret exfiltration** (DLP): catches API keys, tokens, or personal data leaking in prompts +- **Prompt injection**: detects attempts to override system instructions +- **Tool poisoning**: validates MCP tool calls against known-safe patterns + +When using `compose.example.ai.yml`, Pipelock is always running. External AI agents should connect to port 8889 (MCP reverse proxy) instead of directly to Sure's `/mcp` on port 3000. + +For full Pipelock configuration, see [docs/hosting/pipelock.md](pipelock.md). + ## How to update your app The mechanism that updates your self-hosted Sure app is the GHCR (Github Container Registry) Docker image that you see in the `compose.yml` file: diff --git a/docs/hosting/pipelock.md b/docs/hosting/pipelock.md new file mode 100644 index 000000000..622253999 --- /dev/null +++ b/docs/hosting/pipelock.md @@ -0,0 +1,219 @@ +# Pipelock: AI Agent Security Proxy + +[Pipelock](https://github.com/luckyPipewrench/pipelock) is an optional security proxy that scans AI agent traffic flowing through Sure. It protects against secret exfiltration, prompt injection, and tool poisoning. + +## What Pipelock does + +Pipelock runs as a separate proxy service alongside Sure with two listeners: + +| Listener | Port | Direction | What it scans | +|----------|------|-----------|---------------| +| Forward proxy | 8888 | Outbound (Sure to LLM) | DLP (secrets in prompts), response injection | +| MCP reverse proxy | 8889 | Inbound (agent to Sure /mcp) | Prompt injection, tool poisoning, DLP | + +### Forward proxy (outbound) + +When `HTTPS_PROXY=http://pipelock:8888` is set, outbound HTTPS requests from Faraday-based clients (like `ruby-openai`) are routed through Pipelock. It scans request bodies for leaked secrets and response bodies for prompt injection. + +**Covered:** OpenAI API calls via ruby-openai (uses Faraday). +**Not covered:** SimpleFIN, Coinbase, Plaid, or anything using Net::HTTP/HTTParty directly. These bypass `HTTPS_PROXY`. + +### MCP reverse proxy (inbound) + +External AI assistants that call Sure's `/mcp` endpoint should connect through Pipelock on port 8889 instead of directly to port 3000. Pipelock scans: + +- Tool call arguments (DLP, shell obfuscation detection) +- Tool responses (injection payloads) +- Session binding (detects tool inventory manipulation) +- Tool call chains (multi-step attack patterns like recon then exfil) + +## Docker Compose setup + +The `compose.example.ai.yml` file includes Pipelock. To use it: + +1. Download the compose file and Pipelock config: + ```bash + curl -o compose.ai.yml https://raw.githubusercontent.com/we-promise/sure/main/compose.example.ai.yml + curl -o pipelock.example.yaml https://raw.githubusercontent.com/we-promise/sure/main/pipelock.example.yaml + ``` + +2. Start the stack: + ```bash + docker compose -f compose.ai.yml up -d + ``` + +3. Verify Pipelock is healthy: + ```bash + docker compose -f compose.ai.yml ps pipelock + # Should show "healthy" + ``` + +### Connecting external AI agents + +External agents should use the MCP reverse proxy port: + +```text +http://your-server:8889 +``` + +The agent must include the `MCP_API_TOKEN` as a Bearer token in requests. Set this in your `.env`: + +```bash +MCP_API_TOKEN=generate-a-random-token +MCP_USER_EMAIL=your@email.com +``` + +### Running without Pipelock + +To use `compose.example.ai.yml` without Pipelock, remove the `pipelock` service and its `depends_on` entries from `web` and `worker`, then unset the proxy env vars (`HTTPS_PROXY`, `HTTP_PROXY`). + +Or use the standard `compose.example.yml` which does not include Pipelock. + +## Helm (Kubernetes) setup + +Enable Pipelock in your Helm values: + +```yaml +pipelock: + enabled: true + image: + tag: "0.3.2" + mode: balanced +``` + +This creates a separate Deployment, Service, and ConfigMap. The chart auto-injects `HTTPS_PROXY`/`HTTP_PROXY`/`NO_PROXY` into web and worker pods. + +### Exposing MCP to external agents (Kubernetes) + +In Kubernetes, external agents cannot reach the MCP port by default. Enable the Pipelock Ingress: + +```yaml +pipelock: + enabled: true + ingress: + enabled: true + className: nginx + hosts: + - host: pipelock.example.com + paths: + - path: / + pathType: Prefix + tls: + - hosts: [pipelock.example.com] + secretName: pipelock-tls +``` + +Or port-forward for testing: + +```bash +kubectl port-forward svc/sure-pipelock 8889:8889 -n sure +``` + +### Monitoring + +Enable the ServiceMonitor for Prometheus scraping: + +```yaml +pipelock: + serviceMonitor: + enabled: true + interval: 30s + additionalLabels: + release: prometheus +``` + +Metrics are available at `/metrics` on the forward proxy port (8888). + +### Eviction protection + +For production, enable the PodDisruptionBudget: + +```yaml +pipelock: + pdb: + enabled: true + maxUnavailable: 1 +``` + +See the [Helm chart README](../../charts/sure/README.md#pipelock-ai-agent-security-proxy) for all configuration options. + +## Pipelock configuration file + +The `pipelock.example.yaml` file (Docker Compose) or ConfigMap (Helm) controls scanning behavior. Key sections: + +| Section | What it controls | +|---------|-----------------| +| `mode` | `strict` (block threats), `balanced` (warn + block critical), `audit` (log only) | +| `forward_proxy` | Outbound HTTPS scanning (tunnel timeouts, idle timeouts) | +| `dlp` | Data loss prevention (scan env vars, built-in patterns) | +| `response_scanning` | Scan LLM responses for prompt injection | +| `mcp_input_scanning` | Scan inbound MCP requests | +| `mcp_tool_scanning` | Validate tool calls, detect drift | +| `mcp_tool_policy` | Pre-execution rules (shell obfuscation, etc.) | +| `mcp_session_binding` | Pin tool inventory, detect manipulation | +| `tool_chain_detection` | Multi-step attack patterns | +| `websocket_proxy` | WebSocket frame scanning (disabled by default) | +| `logging` | Output format (json/text), verbosity | + +For the Helm chart, most sections are configurable via `values.yaml`. For additional sections not covered by structured values (session profiling, data budgets, kill switch), use the `extraConfig` escape hatch: + +```yaml +pipelock: + extraConfig: + session_profiling: + enabled: true + max_sessions: 1000 +``` + +## Modes + +| Mode | Behavior | Use case | +|------|----------|----------| +| `strict` | Block all detected threats | Production with sensitive data | +| `balanced` | Warn on low-severity, block on high-severity | Default; good for most deployments | +| `audit` | Log everything, block nothing | Initial rollout, testing | + +Start with `audit` mode to see what Pipelock detects without blocking anything. Review the logs, then switch to `balanced` or `strict`. + +## Limitations + +- Forward proxy only covers Faraday-based HTTP clients. Net::HTTP, HTTParty, and other libraries ignore `HTTPS_PROXY`. +- Docker Compose has no egress network policies. The `/mcp` endpoint on port 3000 is still reachable directly (auth token required). For enforcement, use Kubernetes NetworkPolicies. +- Pipelock scans text content. Binary payloads (images, file uploads) are passed through by default. + +## Troubleshooting + +### Pipelock container not starting + +Check the config file is mounted correctly: +```bash +docker compose -f compose.ai.yml logs pipelock +``` + +Common issues: +- Missing `pipelock.example.yaml` file +- YAML syntax errors in config +- Port conflicts (8888 or 8889 already in use) + +### LLM calls failing with proxy errors + +If AI chat stops working after enabling Pipelock: +```bash +# Check Pipelock logs for blocked requests +docker compose -f compose.ai.yml logs pipelock --tail=50 +``` + +If requests are being incorrectly blocked, switch to `audit` mode in the config file and restart: +```yaml +mode: audit +``` + +### MCP requests not reaching Sure + +Verify the MCP upstream is configured correctly: +```bash +# Test from inside the Pipelock container +docker compose -f compose.ai.yml exec pipelock /pipelock healthcheck --addr 127.0.0.1:8888 +``` + +Check that `MCP_API_TOKEN` and `MCP_USER_EMAIL` are set in your `.env` file and that the email matches an existing Sure user.