Juan José MataandClaude f78303ebbe Add function-calling probe to detect tool-use support (#3255)
* feat(ai-health): name the missing function calling behind an opaque chat error

The assistant reads accounts, transactions and holdings through function
calls, so every chat request carries a `tools` payload. A model without
function-calling support rejects it — OpenRouter answers a bare 404 — and
the operator sees only that status code, with nothing pointing at the
model. Both earlier attempts at this guessed from the chat-time error;
the AI status page already runs live probes, so let it answer the
question directly instead.

`AiHealth::Probe#function_calling` asks the configured model for one
trivial tool call the way the assistant asks for its own: chat
completions with `tools` for OpenAI-compatible endpoints, the Responses
API for hosted OpenAI, and `messages.create` with `tools` for Anthropic,
carrying the same strict schema `Provider::Openai` sends. Reading it
against the plain LLM probe is what makes the verdict sound rather than
a guess at 404s: plain chat passing while the same request with tools
fails means the model has no function calling; a response that carries
no tool call means the endpoint took the tools but the model ignored
them; both failing, or a timeout, stays an ordinary probe failure.

The AI status card gains a Function calling (tools) row, an alert
naming the fix for each of the two bad outcomes, and a failure reason.
The hosting settings model field now says the assistant needs a
tools-capable model and links super admins to the check.

Refs #830

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DWCmRQ1ry26JnhA79s1ZKH

* fix(ai-health): only call a refusal a refusal, and probe the route chat takes

Two findings from review of the function-calling probe.

A tools request can fail for reasons that say nothing about tool support:
a 429, a 500, a dropped connection, an unreadable body. Reading any
non-timeout failure as `:unsupported` sent the operator hunting for a new
model over a transient blip. Only a 4xx the service chose to answer with
— excluding the ones that mean "not now" or "not you" — is a refusal of
the tools payload; everything else stays an ordinary probe failure. The
bare 404 from OpenRouter that this page exists to explain still reads as
missing function calling.

`Provider::Openai#supports_responses_endpoint?` is the real routing
decision and `OPENAI_SUPPORTS_RESPONSES_ENDPOINT` can flip it either way,
so choosing the API from "is the endpoint custom" could probe Chat
Completions while chat uses Responses, or the reverse — reporting on a
path the assistant never takes. Ask the provider instead, and cache the
two routes under separate keys.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DWCmRQ1ry26JnhA79s1ZKH

* fix(ai-health): confirm a tools refusal before blaming the model

A client error on the tools request can mean "your tools payload" or "your
request, tools or not" — an invalid schema, a route the endpoint does not
serve, a model it will not run. Splitting those on the status code alone
still put a 422 from an endpoint contract on the model's account and told
the operator to go find another one.

The probe now confirms it: when the tools request comes back a client
error, it asks again with the tools taken off. Only if that lands is the
tools payload what was turned down, and the probe says so with its own
failure code — provider-agnostic, and no reading of error text for the
word "tool", which would only ever fit the provider it was written
against. Statuses that mean "not now" or "not you" (401, 402, 403, 408,
429) never get a second ask. `AiHealth` now just reports the probe's
verdict instead of inferring one from the status.

The troubleshooting fix no longer points at an OpenRouter free tier:
those providers commonly log prompts and completions for training, and
every assistant tool call carries accounts, transactions, and holdings.
It points at the model recommendations already in this doc, and says why
free tiers are the wrong place to look.

Co-Authored-By: Claude <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DWCmRQ1ry26JnhA79s1ZKH

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-08-31 19:47:48 +02:00
2024-02-02 09:05:04 -06:00
2024-02-02 09:05:04 -06:00
2024-02-02 09:05:04 -06:00
2025-05-20 13:31:05 -05:00
2024-02-02 09:05:04 -06:00
2024-08-23 10:06:24 -04:00
2026-06-15 23:29:36 +02:00
2024-02-02 09:05:04 -06:00
2026-06-15 23:29:36 +02:00
2024-02-02 09:05:04 -06:00
2025-09-24 00:19:51 +02:00
2026-04-13 13:44:37 +02:00
2024-02-02 09:05:04 -06:00
2026-06-05 15:16:41 +02:00

Ask DeepWiki View performance data on Skylight Dosu Pipelock Security Scan

sure_shot

Deutsch | Español | Français | 日本語 | 한국어 | Português | Русский | 中文

Sure: The personal finance app for everyone

Get involved: DiscordWebsiteIssues

Important

This repository is a community fork of the now-abandoned Maybe Finance project.
Learn more in their final release doc.

Backstory

The Maybe Finance (archived/abandoned repo) team spent most of 20212022 building a full-featured personal finance and wealth management app. It even included an “Ask an Advisor” feature that connected users with a real CFP/CFA — all included with your subscription.

The business end of things didn't work out, and so they stopped developing the app in mid-2023.

After spending nearly $1 million on development (employees, contractors, data providers, infra, etc.), the team open-sourced the app. Their goal was to let users self-host it for free — and eventually launch a hosted version for a small fee.

They actually did launch that hosted version … briefly.

That also didnt work out — at least not as a sustainable B2C business — so now here we are: hosting a community-maintained fork to keep the codebase alive and see where this can go next.

Join us!

Hosting Sure

Sure is a fully working personal finance app that can be self hosted with Docker. Sure can be accessed from a browser, the macOS desktop app, the mobile app, API clients, and LLM agents. See Sure Clients for an overview.

Forking and Attribution

This repo is a community fork of the archived Maybe Finance repo. Youre free to fork it under the AGPLv3 license — but wed love it if you stuck around and contributed here instead.

To stay compliant and avoid trademark issues:

  • Be sure to include the original AGPLv3 license and clearly state in your README that your fork is based on Maybe Finance but is not affiliated with or endorsed by Maybe Finance Inc.
  • "Maybe" is a trademark of Maybe Finance Inc. and therefore, use of it is NOT allowed in forked repositories (or the logo)

Performance Issues

With data-heavy apps, inevitably, there are performance issues. We've set up a public dashboard showing the problematic requests seen on the demo site, along with the stacktraces to help debug them.

https://www.skylight.io/app/applications/s6PEZSKwcklL/recent/6h/endpoints

Any contributions that help improve performance are very much welcome.

Local Development Setup

If you are trying to self-host the app, read this guide to get started.

The instructions below are for developers to get started with contributing to the app.

Requirements

  • See .ruby-version file for required Ruby version
  • PostgreSQL >9.3 (latest stable version recommended)
  • Redis > 5.4 (latest stable version recommended)

Getting Started

cd sure
cp .env.local.example .env.local
bin/setup
bin/dev

# Optionally, load demo data
rake demo_data:default

Visit http://localhost:3000 to view the app.

If you loaded the optional demo data, log in with these credentials:

  • Email: user@example.com
  • Password: Password1!

For further instructions, see guides below.

Setup Guides

One-click Install

Run on PikaPods

Deploy on Railway

Deploy on Hostim

Managed OpenClaw for Sure Finances

Managed OpenClaw for Sure Finances

License and Trademarks

Maybe and Sure are both distributed under an AGPLv3 license.

  • "Maybe" is a trademark of Maybe Finance, Inc.
  • "Sure" is not, and refers to this community fork.
S
Description
No description provided
Readme AGPL-3.0
149 MiB
Languages
Ruby 77.5%
HTML 12.9%
Dart 5.7%
JavaScript 3.1%
Rust 0.3%
Other 0.2%