QT/sure
1
0
Fork 0
mirror of https://github.com/we-promise/sure.git synced 2026-05-29 15:34:58 +00:00
Code Issues Packages Projects Releases Wiki Activity

test(chat): make default_model tests resilient to ENV model overrides

Browse Source 
CodeRabbit flagged on PR review: the new default_model tests asserted
against Provider::*::DEFAULT_MODEL, but Chat.default_model actually
returns Provider::*.effective_model.presence (which reads
OPENAI_MODEL / ANTHROPIC_MODEL from the environment). With either env
var set, the tests would fail intermittently even though routing was
correct.

- New default_model tests now assert against the provider's
  effective_model directly, so they verify the routing decision
  (which provider's value wins) without coupling to the constant.
- Pre-existing "creates with default model" assertions had the same
  brittleness; switch them to compare against Chat.default_model so
  the chosen model is whatever the env / Setting cascade resolves to.

Verified by running `ANTHROPIC_MODEL=claude-haiku-4-5 OPENAI_MODEL=gpt-4o
bin/rails test test/models/chat_test.rb` — 16 runs, 0 failures
(previously 2 pre-existing failures + 0 from the new tests).
This commit is contained in:
Guillem Arias
2026-05-25 19:58:30 +02:00
parent 714cf0bbb4
commit a0c552cb38
1 changed files with 12 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
test/models/chat_test.rb
View File

@@ -47,7 +47,7 @@ class ChatTest < ActiveSupport::TestCase
chat = @user.chats.start!(prompt, model: nil)
assert_equal 2, chat.messages.count
assert_equal Provider::Openai::DEFAULT_MODEL, chat.messages.find_by!(type: "UserMessage").ai_model
assert_equal Chat.default_model, chat.messages.find_by!(type: "UserMessage").ai_model
end
end
@@ -58,31 +58,35 @@ class ChatTest < ActiveSupport::TestCase
chat = @user.chats.start!(prompt, model: "")
assert_equal 2, chat.messages.count
assert_equal Provider::Openai::DEFAULT_MODEL, chat.messages.find_by!(type: "UserMessage").ai_model
assert_equal Chat.default_model, chat.messages.find_by!(type: "UserMessage").ai_model
end
end
test "default_model returns claude when LLM_PROVIDER=anthropic and Anthropic is configured" do
# These three tests assert routing (which provider's effective_model wins),
# not the constant value itself — the assertion side reads through
# Provider::*.effective_model so ENV overrides like ANTHROPIC_MODEL /
# OPENAI_MODEL don't make the tests flake.
test "default_model returns Anthropic's effective_model when LLM_PROVIDER=anthropic and Anthropic is configured" do
Provider::Anthropic.stubs(:configured?).returns(true)
Setting.stubs(:llm_provider).returns("anthropic")
assert_equal Provider::Anthropic::DEFAULT_MODEL, Chat.default_model
assert_equal Provider::Anthropic.effective_model, Chat.default_model
end
test "default_model falls back to OpenAI when Anthropic is preferred but unconfigured" do
test "default_model falls back to OpenAI's effective_model when Anthropic is preferred but unconfigured" do
Provider::Anthropic.stubs(:configured?).returns(false)
Provider::Openai.stubs(:configured?).returns(true)
Setting.stubs(:llm_provider).returns("anthropic")
assert_equal Provider::Openai::DEFAULT_MODEL, Chat.default_model
assert_equal Provider::Openai.effective_model, Chat.default_model
end
test "default_model uses Anthropic when OpenAI is unconfigured" do
test "default_model uses Anthropic's effective_model when OpenAI is unconfigured" do
Provider::Anthropic.stubs(:configured?).returns(true)
Provider::Openai.stubs(:configured?).returns(false)
Setting.stubs(:llm_provider).returns("openai")
assert_equal Provider::Anthropic::DEFAULT_MODEL, Chat.default_model
assert_equal Provider::Anthropic.effective_model, Chat.default_model
end
test "creates with configured model when OPENAI_MODEL env is set" do