Files
sure/app/controllers/settings/hostings_controller.rb
kianrafiee d267288a45 Add RentCast & Realie integration for automatic property data and valuations (#2727)
* Add RentCast and Realie AVM providers for property valuation

Adds Automated Valuation Model (AVM) provider support so self-hosted
users can create property accounts from a US address lookup instead of
entering details manually:

- Provider::Rentcast and Provider::Realie clients, each fetching the
  property record (type, year built, square footage) and value estimate
  in a single API request, registered under a new :property_valuations
  registry concept
- API key fields (encrypted, ENV-overridable) with monthly usage display
  in Settings > Self-Hosting under "Property Valuation Providers"
- New property flow: when a provider key is configured, the method
  selector offers "Add via RentCast/Realie" alongside manual entry;
  the lookup form reuses the manual flow's localized address fields
  and creates the account active with fetched attributes, valuation as
  balance, and the address saved
- SyncPropertyValuationsJob refreshes linked property valuations once a
  day (never hourly) via config/schedule.yml; each provider enforces its
  monthly request cap (RentCast 50, Realie 25) with a calendar-month
  counter shared between creation lookups and refreshes

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Allow ENV override of AVM provider monthly request caps

The RentCast (50/month) and Realie (25/month) budgets default to the
free tier limits but can now be raised for paid plans via
RENTCAST_MAX_REQUESTS_PER_MONTH / REALIE_MAX_REQUESTS_PER_MONTH,
mirroring the AlphaVantage and Tiingo request limit overrides. The
settings descriptions and usage display reflect the effective cap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address review feedback: durable request counters, safer refresh job

- Move monthly AVM request counters from Rails.cache to a new
  provider_request_counts table with atomic upsert increments, so the
  hard budget caps survive cache eviction, restarts, and Redis flushes
- SyncPropertyValuationsJob: only mark a property synced when the
  balance update succeeds (wrapped in a transaction), process stalest
  valuations first so a tight budget goes where it matters, and report
  failures via DebugLogEntry.capture instead of Rails.logger
- Realie: reject lookups whose returned city/ZIP contradict the entered
  address (street+state queries can match the wrong city); document that
  numeric use codes are unpublished and leave the subtype unset
- Add Faraday open/request timeouts to both provider clients
- Localize all user-facing provider error messages
  (config/locales/models/provider/en.yml)
- Add a check constraint restricting properties.avm_provider to known
  providers
- Use the min-h-80 scale token instead of min-h-[320px]
- Tests: exact-argument expectations on the lookup stub, provider stubs
  in the no-provider test, RentCast/Realie API key update tests,
  ProviderRequestCount unit tests, failed-balance regression test

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Stub AVM provider registry lookups in settings system test

Settings::HostingsController#show now resolves the RentCast and Realie
providers for usage display; the system test's partial get_provider
stubbing treats the new lookups as unexpected invocations without these.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Address second-round review: candidate scan, provider reuse, shared row partial

- Realie: when the address lookup returns multiple candidates, pick the
  first one consistent with the entered city/ZIP instead of judging only
  the first array element; the location-mismatch error now only fires
  when no candidate matches
- SyncPropertyValuationsJob: resolve one provider instance per key for
  the whole run, so the per-instance request throttle actually spaces
  requests across properties instead of resetting on each iteration
- Extract the AVM method selector row into a shared partial so the
  manual and provider options render one shape

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add retry middleware and sync index from review feedback

- Both AVM clients now retry transient connection failures (same
  Faraday retry config as Tiingo) so a network blip doesn't burn one of
  the tight monthly budget's requests
- Partial index on properties(avm_provider, avm_last_synced_on) backing
  the daily sync job's filter and stalest-first ordering

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Key the AVM sync index on avm_last_synced_on

The daily job orders by avm_last_synced_on ASC NULLS FIRST across all
AVM-linked properties, so leading the partial index with avm_provider
prevented it from serving the sort. Rekeyed on avm_last_synced_on with
matching null ordering; the partial predicate still covers the filter.
Amended in place since the migration is unmerged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Map Realie numeric use codes; skip incomplete addresses in sync job

- Realie documents its numeric use codes at
  docs.realie.ai/api-reference/feature-key (earlier skip reason was
  wrong — the table exists). Map the residential/land/agricultural codes
  to Property subtypes, leaving codes without a subtype equivalent
  (e.g. 1006 mobile/manufactured) unset like the RentCast mapping.
- The daily job now skips properties whose address is missing a street
  or state before resolving a provider, logging a warn-level
  DebugLogEntry, so a malformed address can't burn a monthly-budget
  request every day.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Guard currency mismatches in refresh; reset AVM keys in test teardown

- The daily job now skips (with a warn-level DebugLogEntry) properties
  whose account currency no longer matches the provider's valuation
  currency, checked via the concept's valuation_currency before spending
  a request — writing a USD valuation into a re-currencied account would
  corrupt the balance
- Hostings controller test teardown now clears rentcast_api_key and
  realie_api_key alongside the other cached global settings

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Validate AVM lookup inputs before spending a provider request

The form marks name and address fields required, but a forged or
JS-less submission bypasses that and would burn one of the tight
monthly-budget requests on a lookup that can't produce a property.
Property::AvmImport now validates name and the full address locally
(localized error) before calling the provider.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Add preview-and-confirm step to the AVM property lookup

RentCast's AVM endpoint is location-based: a plausible but nonexistent
address still geocodes and returns an area-derived estimate with no
property record behind it. Instead of silently creating an account from
that, the lookup now shows what the provider returned — type, year
built, area (each "Unknown" when missing), and the estimated value —
with an explicit notice when no property record was found, and only the
user's confirmation creates the account.

The confirm step reuses the fetched data via the form (sanitized
server-side), so the flow still costs exactly one provider request.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Require a complete US address before daily valuation refresh

The job's guard only required street and state, but a later-blanked
city or ZIP silently disables the Realie wrong-city check (blank
entered fields count as "no mismatch"), so a refresh could accept
another property's valuation. The guard now mirrors the import-time
completeness check and also skips addresses edited to a non-US country.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Sign the AVM preview payload; blur valuation in privacy mode

- The confirm step now rebuilds everything from a signed, 1-hour
  message-verifier token generated at lookup time, proving the lookup
  (and its counted provider request) actually ran — a direct confirm
  POST with fabricated data can no longer create provider-linked
  properties that the daily refresh job would then spend quota on.
  Forged/expired tokens re-render the lookup form with a clear error.
- The preview's market value now carries the privacy-sensitive class so
  privacy mode blurs it like other account amounts.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Treat zero Realie valuations as absent; bind preview token to family

Review follow-ups from PR #2727:
- Realie returns modelValue: 0 when it can't produce an AVM estimate;
  zero now falls back to the assessed market value instead of syncing
  a $0 balance, and an all-zero record errors out.
- Monetary values now parse via BigDecimal(value.to_s) rather than
  Float#to_d, avoiding binary float rounding artifacts in balances.
- The signed AVM preview token's purpose is scoped to the family that
  ran the lookup, so a leaked token can't be replayed cross-family.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* Refresh AVM property valuations monthly instead of daily

Per @jjmata's review on #2727: valuations change slowly and providers
enforce tight monthly request caps, so a daily cron adds no value.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-07-25 04:47:58 +02:00

329 lines
14 KiB
Ruby

class Settings::HostingsController < ApplicationController
layout "settings"
# Minimum accepted value for each configurable LLM budget field. Mirrors the
# `min:` attribute on the form inputs in `_openai_settings.html.erb` so the
# controller rejects what the browser-side validator would reject.
LLM_BUDGET_MINIMUMS = {
llm_context_window: 256,
llm_max_response_tokens: 64,
llm_max_items_per_call: 1
}.freeze
guard_feature unless: -> { self_hosted? }
before_action :ensure_admin, only: [ :update, :clear_cache, :disconnect_external_assistant ]
before_action :ensure_super_admin_for_onboarding, only: :update
def show
@breadcrumbs = [
[ t("breadcrumbs.home"), root_path ],
[ t("breadcrumbs.self_hosting"), nil ]
]
# Determine which providers are currently selected
exchange_rate_provider = ENV["EXCHANGE_RATE_PROVIDER"].presence || Setting.exchange_rate_provider
enabled_securities = Setting.enabled_securities_providers
# Show provider settings if used for FX or enabled for securities
@show_twelve_data_settings = exchange_rate_provider == "twelve_data" || enabled_securities.include?("twelve_data")
@show_yahoo_finance_settings = exchange_rate_provider == "yahoo_finance" || enabled_securities.include?("yahoo_finance")
@show_tiingo_settings = enabled_securities.include?("tiingo")
@show_eodhd_settings = enabled_securities.include?("eodhd")
@show_alpha_vantage_settings = enabled_securities.include?("alpha_vantage")
# T-Invest doubles as a brand-logo source consulted regardless of the price
# provider, so its token is useful even when it's not enabled for prices.
# Always surface the token field, decoupled from the securities checklist.
@show_tinkoff_invest_settings = true
# Only fetch provider data if we're showing the section
if @show_twelve_data_settings
twelve_data_provider = Provider::Registry.get_provider(:twelve_data)
@twelve_data_usage = twelve_data_provider&.usage
@plan_restricted_securities = Current.family.securities_with_plan_restrictions(provider: "TwelveData")
end
if @show_yahoo_finance_settings
@yahoo_finance_provider = Provider::Registry.get_provider(:yahoo_finance)
@yahoo_finance_health_status = @yahoo_finance_provider&.health_status || :unknown
end
# Property valuation (AVM) providers — usage is shown against their tight
# monthly request caps when a key is configured
@rentcast_usage = Provider::Registry.get_provider(:rentcast)&.usage
@realie_usage = Provider::Registry.get_provider(:realie)&.usage
end
def update
if hosting_params.key?(:onboarding_state)
onboarding_state = hosting_params[:onboarding_state].to_s
Setting.onboarding_state = onboarding_state
end
if hosting_params.key?(:require_email_confirmation)
Setting.require_email_confirmation = hosting_params[:require_email_confirmation]
end
if hosting_params.key?(:invite_only_default_family_id)
value = hosting_params[:invite_only_default_family_id].presence
Setting.invite_only_default_family_id = value
end
if hosting_params.key?(:brand_fetch_client_id)
Setting.brand_fetch_client_id = hosting_params[:brand_fetch_client_id]
end
if hosting_params.key?(:brand_fetch_high_res_logos)
Setting.brand_fetch_high_res_logos = hosting_params[:brand_fetch_high_res_logos] == "1"
end
update_encrypted_setting(:twelve_data_api_key)
if hosting_params.key?(:exchange_rate_provider)
Setting.exchange_rate_provider = hosting_params[:exchange_rate_provider]
end
if hosting_params.key?(:securities_provider)
Setting.securities_provider = hosting_params[:securities_provider]
end
if hosting_params.key?(:securities_providers)
new_providers = Array(hosting_params[:securities_providers]).reject(&:blank?) & Security.valid_price_providers
old_providers = Setting.enabled_securities_providers
Setting.securities_providers = new_providers.join(",")
# Clear the legacy singular setting so the fallback in
# enabled_securities_providers doesn't re-enable a provider
# the user just unchecked.
Setting.securities_provider = nil if new_providers.empty?
# Mark securities linked to removed providers as offline so they aren't
# silently queried against an incompatible fallback provider (e.g. MFAPI
# scheme codes sent to TwelveData). The price_provider is preserved so
# provider_status can report :provider_unavailable.
removed = old_providers - new_providers
removed.each do |removed_provider|
Security.where(price_provider: removed_provider, offline: false)
.in_batches.update_all(offline: true, offline_reason: "provider_disabled")
end
# Bring securities back online when their provider is re-enabled — but only
# those that were taken offline by a provider toggle, not by health checks.
added = new_providers - old_providers
added.each do |added_provider|
Security.where(price_provider: added_provider, offline: true, offline_reason: "provider_disabled")
.in_batches.update_all(offline: false, offline_reason: nil, failed_fetch_count: 0, failed_fetch_at: nil)
end
end
update_encrypted_setting(:tiingo_api_key)
update_encrypted_setting(:eodhd_api_key)
update_encrypted_setting(:alpha_vantage_api_key)
update_encrypted_setting(:tinkoff_invest_api_key)
update_encrypted_setting(:rentcast_api_key)
update_encrypted_setting(:realie_api_key)
if hosting_params.key?(:syncs_include_pending)
Setting.syncs_include_pending = hosting_params[:syncs_include_pending] == "1"
end
sync_settings_changed = false
if hosting_params.key?(:auto_sync_enabled)
Setting.auto_sync_enabled = hosting_params[:auto_sync_enabled] == "1"
sync_settings_changed = true
end
if hosting_params.key?(:auto_sync_time)
time_value = hosting_params[:auto_sync_time]
unless Setting.valid_auto_sync_time?(time_value)
flash[:alert] = t(".invalid_sync_time")
return redirect_to settings_hosting_path
end
Setting.auto_sync_time = time_value
Setting.auto_sync_timezone = current_user_timezone
sync_settings_changed = true
end
if sync_settings_changed
sync_auto_sync_scheduler!
end
if hosting_params.key?(:openai_access_token)
token_param = hosting_params[:openai_access_token].to_s.strip
# Ignore blanks and redaction placeholders to prevent accidental overwrite
unless token_param.blank? || token_param == "********"
Setting.openai_access_token = token_param
end
end
# Validate OpenAI configuration before updating
if hosting_params.key?(:openai_uri_base) || hosting_params.key?(:openai_model)
Setting.validate_openai_config!(
uri_base: hosting_params[:openai_uri_base],
model: hosting_params[:openai_model]
)
end
if hosting_params.key?(:openai_uri_base)
Setting.openai_uri_base = hosting_params[:openai_uri_base]
end
if hosting_params.key?(:openai_model)
Setting.openai_model = hosting_params[:openai_model]
end
if hosting_params.key?(:openai_json_mode)
Setting.openai_json_mode = hosting_params[:openai_json_mode].presence
end
if hosting_params.key?(:anthropic_access_token)
token_param = hosting_params[:anthropic_access_token].to_s.strip
unless token_param.blank? || token_param == "********"
Setting.anthropic_access_token = token_param
end
end
if hosting_params.key?(:anthropic_base_url)
raw_base_url = hosting_params[:anthropic_base_url].to_s.strip
if raw_base_url.blank?
Setting.anthropic_base_url = nil
else
parsed = URI.parse(raw_base_url) rescue nil
unless parsed.is_a?(URI::HTTP)
raise Setting::ValidationError, t(".invalid_anthropic_base_url")
end
# A custom Anthropic-compatible endpoint requires a model — Provider::Anthropic
# raises without one. Validate the pair together (mirrors the OpenAI branch), using
# the submitted model when present so a blanked model field is caught too.
effective_model =
if hosting_params.key?(:anthropic_model)
hosting_params[:anthropic_model].to_s.strip
else
Setting.anthropic_model.to_s.strip
end
if effective_model.blank?
raise Setting::ValidationError, t(".anthropic_model_required_for_base_url")
end
Setting.anthropic_base_url = raw_base_url
end
end
if hosting_params.key?(:anthropic_model)
Setting.anthropic_model = hosting_params[:anthropic_model].presence
end
if hosting_params.key?(:llm_provider)
provider = hosting_params[:llm_provider].to_s
if %w[openai anthropic].include?(provider)
Setting.llm_provider = provider
end
end
LLM_BUDGET_MINIMUMS.each do |key, minimum|
next unless hosting_params.key?(key)
raw = hosting_params[key].to_s.strip
if raw.blank?
Setting.public_send("#{key}=", nil)
next
end
parsed = Integer(raw, 10) rescue nil
if parsed.nil? || parsed < minimum
label = t("settings.hostings.openai_settings.#{key}_label")
raise Setting::ValidationError, t(".invalid_llm_budget", field: label, minimum: minimum)
end
Setting.public_send("#{key}=", parsed)
end
if hosting_params.key?(:external_assistant_url)
Setting.external_assistant_url = hosting_params[:external_assistant_url]
end
if hosting_params.key?(:external_assistant_token)
token_param = hosting_params[:external_assistant_token].to_s.strip
unless token_param.blank? || token_param == "********"
Setting.external_assistant_token = token_param
end
end
if hosting_params.key?(:external_assistant_agent_id)
Setting.external_assistant_agent_id = hosting_params[:external_assistant_agent_id]
end
update_assistant_type
redirect_to settings_hosting_path, notice: t(".success")
rescue Setting::ValidationError => error
# Preserve user-submitted OpenAI config so the form re-renders with their
# input intact (issue #1824). The form auto-submits on blur, so a partial
# entry (e.g. URI base before model) hits validation and would otherwise
# be wiped because the view reads from the unchanged Setting.* values.
@openai_uri_base_input = hosting_params[:openai_uri_base] if hosting_params.key?(:openai_uri_base)
@openai_model_input = hosting_params[:openai_model] if hosting_params.key?(:openai_model)
@anthropic_base_url_input = hosting_params[:anthropic_base_url] if hosting_params.key?(:anthropic_base_url)
@anthropic_model_input = hosting_params[:anthropic_model] if hosting_params.key?(:anthropic_model)
flash.now[:alert] = error.message
render :show, status: :unprocessable_entity
end
def clear_cache
DataCacheClearJob.perform_later(Current.family)
redirect_to settings_hosting_path, notice: t(".cache_cleared")
end
def disconnect_external_assistant
Setting.external_assistant_url = nil
Setting.external_assistant_token = nil
Setting.external_assistant_agent_id = nil
Current.family.update!(assistant_type: "builtin") unless ENV["ASSISTANT_TYPE"].present?
redirect_to settings_hosting_path, notice: t(".external_assistant_disconnected")
rescue => e
Rails.logger.error("[External Assistant] Disconnect failed: #{e.message}")
redirect_to settings_hosting_path, alert: t("settings.hostings.update.failure")
end
private
def hosting_params
return ActionController::Parameters.new unless params.key?(:setting)
params.require(:setting).permit(:onboarding_state, :require_email_confirmation, :invite_only_default_family_id, :brand_fetch_client_id, :brand_fetch_high_res_logos, :twelve_data_api_key, :tiingo_api_key, :eodhd_api_key, :alpha_vantage_api_key, :tinkoff_invest_api_key, :rentcast_api_key, :realie_api_key, :openai_access_token, :openai_uri_base, :openai_model, :openai_json_mode, :anthropic_access_token, :anthropic_base_url, :anthropic_model, :llm_provider, :llm_context_window, :llm_max_response_tokens, :llm_max_items_per_call, :exchange_rate_provider, :securities_provider, :syncs_include_pending, :auto_sync_enabled, :auto_sync_time, :external_assistant_url, :external_assistant_token, :external_assistant_agent_id, securities_providers: [])
end
def update_assistant_type
return unless params[:family].present? && params[:family][:assistant_type].present?
return if ENV["ASSISTANT_TYPE"].present?
assistant_type = params[:family][:assistant_type]
Current.family.update!(assistant_type: assistant_type) if Family::ASSISTANT_TYPES.include?(assistant_type)
end
def ensure_admin
redirect_to settings_hosting_path, alert: t(".not_authorized") unless Current.user.admin?
end
def ensure_super_admin_for_onboarding
onboarding_params = %i[onboarding_state invite_only_default_family_id]
return unless onboarding_params.any? { |p| hosting_params.key?(p) }
redirect_to settings_hosting_path, alert: t(".not_authorized") unless Current.user.super_admin?
end
def sync_auto_sync_scheduler!
AutoSyncScheduler.sync!
rescue StandardError => error
Rails.logger.error("[AutoSyncScheduler] Failed to sync scheduler: #{error.message}")
Rails.logger.error(error.backtrace.join("\n"))
flash[:alert] = t(".scheduler_sync_failed")
end
def update_encrypted_setting(param_key)
return unless hosting_params.key?(param_key)
value = hosting_params[param_key].to_s.strip
Setting.public_send(:"#{param_key}=", value) unless value.blank? || value == "********"
end
def current_user_timezone
Current.family&.timezone.presence || "UTC"
end
end