QT/sure
1
0
Fork 0
mirror of https://github.com/we-promise/sure.git synced 2026-04-19 12:04:08 +00:00
Code Issues Packages Projects Releases Wiki Activity

Provider investment fixes (#600)

Browse Source 
* FIX issue with stock price retrieval on weekend

* make weekend provisional and increase lookback

* FIX query error

* fix gap fill

The bug: When a price is provisional but the provider doesn't return a new value (weekends), we fall back to the existing DB value instead of gap-filling from Friday's correct price.

* Update importer.rb

Align provider fetch to use PROVISIONAL_LOOKBACK_DAYS for consistency. In the DB fallback, derive currency from provider_prices or db_prices and filter the query accordingly.

* Update 20260110122603_mark_suspicious_prices_provisional.rb

* Delete db/migrate/20260110122603_mark_suspicious_prices_provisional.rb

Signed-off-by: soky srm <sokysrm@gmail.com>

* Update importer.rb

* FIX tests

* FIX last tests

* Update importer_test.rb

The test doesn't properly force effective_start_date to skip old dates because there are many missing dates between the old date and recent dates. Let me fix it to properly test the subset processing scenario.

---------

Signed-off-by: soky srm <sokysrm@gmail.com>
This commit is contained in:
soky srm
2026-01-10 15:43:07 +01:00
committed by GitHub
parent 2f4e2eab8d
commit 5750e69acf
8 changed files with 180 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
app/models/security/price.rb
View File

@@ -4,13 +4,12 @@ class Security::Price < ApplicationRecord
validates :date, :price, :currency, presence: true
validates :date, uniqueness: { scope: %i[security_id currency] }
# Provisional prices from recent weekdays that should be re-fetched
# Provisional prices from recent days that should be re-fetched
# - Must be provisional (gap-filled)
# - Must be from the last few days (configurable, default 3)
# - Must be a weekday (Saturday = 6, Sunday = 0 in PostgreSQL DOW)
scope :refetchable_provisional, ->(lookback_days: 3) {
# - Must be from the last few days (configurable, default 7)
# - Includes weekends: they get fixed via cascade when weekday prices are fetched
scope :refetchable_provisional, ->(lookback_days: 7) {
where(provisional: true)
.where(date: lookback_days.days.ago.to_date..Date.current)
.where("EXTRACT(DOW FROM date) NOT IN (0, 6)")
}
end

72
app/models/security/price/importer.rb
View File

@@ -2,7 +2,7 @@ class Security::Price::Importer
MissingSecurityPriceError = Class.new(StandardError)
MissingStartPriceError = Class.new(StandardError)
PROVISIONAL_LOOKBACK_DAYS = 3
PROVISIONAL_LOOKBACK_DAYS = 7
def initialize(security:, security_provider:, start_date:, end_date:, clear_cache: false)
@security = security
@@ -26,6 +26,7 @@ class Security::Price::Importer
end
prev_price_value = start_price_value
prev_currency = prev_price_currency || db_price_currency || "USD"
unless prev_price_value.present?
Rails.logger.error("Could not find a start price for #{security.ticker} on or before #{start_date}")
@@ -49,22 +50,32 @@ class Security::Price::Importer
provider_currency = provider_price&.currency
has_provider_price = provider_price_value.present? && provider_price_value.to_f > 0
has_db_price = db_price_value.present? && db_price_value.to_f > 0
is_provisional = db_price&.provisional
chosen_price = if clear_cache || is_provisional
provider_price_value || db_price_value # overwrite when possible (or when provisional)
# Choose price and currency from the same source to avoid mismatches
chosen_price, chosen_currency = if clear_cache || is_provisional
# For provisional/cache clear: only use provider price, let gap-fill handle missing
# This ensures stale DB values don't persist when provider has no weekend data
[ provider_price_value, provider_currency ]
elsif has_db_price
# For non-provisional with valid DB price: preserve existing value (user edits)
[ db_price_value, db_price&.currency ]
else
db_price_value || provider_price_value # fill gaps
# Fill gaps with provider data
[ provider_price_value, provider_currency ]
end
# Gap-fill using LOCF (last observation carried forward)
# Treat nil or zero prices as invalid and use previous price
# Treat nil or zero prices as invalid and use previous price/currency
used_locf = false
if chosen_price.nil? || chosen_price.to_f <= 0
chosen_price = prev_price_value
chosen_currency = prev_currency
used_locf = true
end
prev_price_value = chosen_price
prev_currency = chosen_currency || prev_currency
provisional = determine_provisional_status(
date: date,
@@ -77,7 +88,7 @@ class Security::Price::Importer
security_id: security.id,
date: date,
price: chosen_price,
currency: provider_currency || prev_price_currency || db_price_currency || "USD",
currency: chosen_currency || "USD",
provisional: provisional
}
end
@@ -90,7 +101,7 @@ class Security::Price::Importer
def provider_prices
@provider_prices ||= begin
provider_fetch_start_date = effective_start_date - 5.days
provider_fetch_start_date = effective_start_date - PROVISIONAL_LOOKBACK_DAYS.days
response = security_provider.fetch_security_prices(
symbol: security.ticker,
@@ -151,22 +162,53 @@ class Security::Price::Importer
end
def start_price_value
provider_price_value = provider_prices.select { |date, _| date <= start_date }
.max_by { |date, _| date }
&.last&.price
db_price_value = db_prices[start_date]&.price
provider_price_value || db_price_value
# When processing full range (first sync), use original behavior
if effective_start_date == start_date
provider_price_value = provider_prices.select { |date, _| date <= start_date }
.max_by { |date, _| date }
&.last&.price
db_price_value = db_prices[start_date]&.price
return provider_price_value if provider_price_value.present? && provider_price_value.to_f > 0
return db_price_value if db_price_value.present? && db_price_value.to_f > 0
return nil
end
# For partial range (effective_start_date > start_date), use recent data
# This prevents stale prices from old trade dates propagating to current gap-fills
cutoff_date = effective_start_date
# First try provider prices (most recent before cutoff)
provider_price_value = provider_prices
.select { |date, _| date < cutoff_date }
.max_by { |date, _| date }
&.last&.price
return provider_price_value if provider_price_value.present? && provider_price_value.to_f > 0
# Fall back to most recent DB price before cutoff
currency = prev_price_currency || db_price_currency
Security::Price
.where(security_id: security.id)
.where("date < ?", cutoff_date)
.where("price > 0")
.where(provisional: false)
.then { |q| currency.present? ? q.where(currency: currency) : q }
.order(date: :desc)
.limit(1)
.pick(:price)
end
def determine_provisional_status(date:, has_provider_price:, used_locf:, existing_provisional:)
# Provider returned real price => NOT provisional
return false if has_provider_price
# Gap-filled (LOCF) => provisional only if recent weekday
# Gap-filled (LOCF) => provisional if recent (including weekends)
# Weekend prices inherit uncertainty from Friday and get fixed via cascade
# when the next weekday sync fetches correct Friday price
if used_locf
is_weekday = !date.saturday? && !date.sunday?
is_recent = date >= PROVISIONAL_LOOKBACK_DAYS.days.ago.to_date
return is_weekday && is_recent
return is_recent
end
# Otherwise preserve existing status