sure/docs/llm-guides/adding-a-securities-provider.md

Adding a New Securities Price Provider

This guide covers every step needed to add a new securities price provider (like Tiingo, EODHD, etc.) to the application.

Architecture Overview

User searches ticker in combobox
  → SecuritiesController#index
    → Security.search_provider (queries all enabled providers concurrently)
      → Provider::YourProvider#search_securities
        → Returns results with provider key attached
          → User selects one → price_provider stored on Security record

Account sync / price fetch
  → Security#price_data_provider (looks up provider by security.price_provider)
    → Provider::YourProvider#fetch_security_prices
      → Security::Price::Importer gap-fills and upserts into DB

Key files:

• Provider class: app/models/provider/your_provider.rb
• Registry: app/models/provider/registry.rb
• Settings: app/models/setting.rb
• Provider resolution: app/models/security/provided.rb
• Price import: app/models/security/price/importer.rb
• Market data sync: app/models/account/market_data_importer.rb

Step 1: Create the Provider Class

Create app/models/provider/your_provider.rb:

class Provider::YourProvider < Provider
  include SecurityConcept

  # Include if your provider has rate limits
  include RateLimitable

  # Custom error classes
  Error = Class.new(Provider::Error)
  RateLimitError = Class.new(Error)

  # Rate limiting (only if you included RateLimitable)
  MIN_REQUEST_INTERVAL = 1.0 # seconds between requests

  def initialize(api_key)
    @api_key = api_key # pipelock:ignore
  end

  # --- Required Methods ---

  def search_securities(symbol, country_code: nil, exchange_operating_mic: nil)
    with_provider_response do
      response = client.get("#{base_url}/search", params: { q: symbol })
      parsed = JSON.parse(response.body)

      parsed.map do |result|
        SecurityConcept::Security.new(
          symbol:                 result["ticker"],
          name:                   result["name"],
          logo_url:               result["logo"],
          exchange_operating_mic: map_exchange_to_mic(result["exchange"]),
          country_code:           result["country"],
          currency:               result["currency"]
        )
      end
    end
  end

  def fetch_security_info(symbol:, exchange_operating_mic:)
    with_provider_response do
      response = client.get("#{base_url}/info/#{symbol}")
      parsed = JSON.parse(response.body)

      SecurityConcept::SecurityInfo.new(
        symbol:                 parsed["ticker"],
        name:                   parsed["name"],
        links:                  parsed["website"],
        logo_url:               parsed["logo"],
        description:            parsed["description"],
        kind:                   parsed["type"],       # e.g. "common stock", "etf"
        exchange_operating_mic: exchange_operating_mic
      )
    end
  end

  def fetch_security_price(symbol:, exchange_operating_mic:, date:)
    with_provider_response do
      response = client.get("#{base_url}/price/#{symbol}", params: { date: date.to_s })
      parsed = JSON.parse(response.body)

      SecurityConcept::Price.new(
        symbol:                 symbol,
        date:                   Date.parse(parsed["date"]),
        price:                  parsed["close"].to_f,
        currency:               parsed["currency"],
        exchange_operating_mic: exchange_operating_mic
      )
    end
  end

  def fetch_security_prices(symbol:, exchange_operating_mic:, start_date:, end_date:)
    with_provider_response do
      response = client.get("#{base_url}/prices/#{symbol}", params: {
        start: start_date.to_s,
        end: end_date.to_s
      })
      parsed = JSON.parse(response.body)

      parsed.map do |row|
        SecurityConcept::Price.new(
          symbol:                 symbol,
          date:                   Date.parse(row["date"]),
          price:                  row["close"].to_f,
          currency:               row["currency"],
          exchange_operating_mic: exchange_operating_mic
        )
      end
    end
  end

  # Optional: limit how far back the importer fetches history.
  # nil = unlimited. Free tiers often have limits.
  def max_history_days
    365
  end

  # Optional: health check for admin UI
  def healthy?
    with_provider_response do
      response = client.get("#{base_url}/status")
      JSON.parse(response.body)["status"] == "ok"
    end
  end

  # Optional: usage stats for admin UI
  def usage
    with_provider_response do
      Provider::UsageData.new(
        used: daily_request_count,
        limit: MAX_REQUESTS_PER_DAY,
        utilization: (daily_request_count.to_f / MAX_REQUESTS_PER_DAY * 100).round(1),
        plan: "Free"
      )
    end
  end

  private

    def base_url
      "https://api.yourprovider.com/v1"
    end

    def client
      @client ||= Faraday.new do |conn|
        conn.headers["Authorization"] = "Bearer #{@api_key}"
        conn.response :raise_error
      end
    end

    # Map provider's exchange names to ISO 10383 MIC codes.
    # This is critical — the app uses MIC codes everywhere.
    def map_exchange_to_mic(exchange_name)
      {
        "NASDAQ" => "XNAS",
        "NYSE"   => "XNYS",
        "LSE"    => "XLON",
        "XETRA"  => "XETR",
        "AMS"    => "XAMS"
        # Add all exchanges your provider returns
      }[exchange_name]
    end
end

Data Structures (defined in SecurityConcept)

All methods must return these exact types (wrapped in with_provider_response):

SecurityConcept::Security    = Data.define(:symbol, :name, :logo_url, :exchange_operating_mic, :country_code, :currency)
SecurityConcept::SecurityInfo = Data.define(:symbol, :name, :links, :logo_url, :description, :kind, :exchange_operating_mic)
SecurityConcept::Price       = Data.define(:symbol, :date, :price, :currency, :exchange_operating_mic)

Error Handling

All public methods must be wrapped in with_provider_response:

def some_method
  with_provider_response do
    # Your logic. Raise on errors.
    # The block return value becomes response.data
  end
end

Callers always receive a Provider::Response:

response = provider.fetch_security_price(...)
response.success?  # true/false
response.data      # the return value from the block
response.error     # Provider::Error instance (on failure)

Rate Limiting

If your provider has rate limits, include RateLimitable and use cache-based atomic counters:

include RateLimitable

MIN_REQUEST_INTERVAL = 1.5 # seconds between requests

MAX_REQUESTS_PER_DAY = 20

private

  def throttle_request
    super # enforces MIN_REQUEST_INTERVAL
    enforce_daily_limit!
  end

  def enforce_daily_limit!
    cache_key = "your_provider:daily:#{Date.current}"
    count = Rails.cache.increment(cache_key, 1, expires_in: 1.day, initial: 0)
    raise RateLimitError, "Daily limit reached" if count > MAX_REQUESTS_PER_DAY
  end

Exchange Mapping

Every provider has its own naming for exchanges. You must map them to ISO 10383 MIC codes. Define bidirectional maps:

# Provider exchange name → MIC code (for parsing search results)
PROVIDER_TO_MIC = { "NASDAQ" => "XNAS", ... }.freeze

# MIC code → Provider exchange name (for building API requests)
MIC_TO_PROVIDER = PROVIDER_TO_MIC.invert.freeze

See config/exchanges.yml for the full list of MIC codes and their display names.

Currency Handling

Some providers don't return currency in every response. Common pattern: cache currency from search results and reuse later:

def search_securities(symbol, **opts)
  with_provider_response do
    results = api_call(...)
    results.each do |r|
      Rails.cache.write("your_provider:currency:#{r[:ticker].upcase}", r[:currency], expires_in: 24.hours)
    end
    # ...
  end
end

def fetch_security_prices(symbol:, **)
  with_provider_response do
    # ...
    currency = Rails.cache.read("your_provider:currency:#{symbol.upcase}") || fallback_currency(exchange)
    # ...
  end
end

Step 2: Register in the Provider Registry

Edit app/models/provider/registry.rb:

Add to available_providers (around line 144):

def available_providers
  case concept
  when :exchange_rates
    %i[twelve_data yahoo_finance]
  when :securities
    %i[twelve_data yahoo_finance tiingo eodhd alpha_vantage mfapi your_provider]
  # ...
  end
end

Add the factory method (private section, around line 85):

def your_provider
  api_key = ENV["YOUR_PROVIDER_API_KEY"].presence || Setting.your_provider_api_key # pipelock:ignore
  return nil unless api_key.present?
  Provider::YourProvider.new(api_key)
end

If your provider needs no API key (like Yahoo Finance or MFAPI):

def your_provider
  Provider::YourProvider.new
end

Step 3: Add Settings

Edit app/models/setting.rb:

Add the API key field (around line 40):

field :your_provider_api_key, type: :string, default: ENV["YOUR_PROVIDER_API_KEY"]

Add to encrypted fields (in EncryptedSettingFields, around line 55):

ENCRYPTED_FIELDS = %i[
  twelve_data_api_key
  tiingo_api_key
  eodhd_api_key
  alpha_vantage_api_key
  your_provider_api_key    # ← add here
  openai_access_token
  external_assistant_token
]

Step 4: Add to the Settings UI

Hostings Controller

Edit app/controllers/settings/hostings_controller.rb:

In show — add a flag to control visibility:

@show_your_provider_settings = enabled_securities.include?("your_provider")

In update — handle the API key:

update_encrypted_setting(:your_provider_api_key)

Hostings View

Edit the settings view to add your provider's checkbox and API key field. Follow the existing pattern for Tiingo/EODHD (checkbox in the provider selection list, API key input shown when enabled).

Step 5: Add Translations

Edit config/locales/views/settings/hostings/en.yml:

Add provider name (under provider_selection.providers):

providers:
  twelve_data: "Twelve Data"
  yahoo_finance: "Yahoo Finance"
  tiingo: "Tiingo"
  eodhd: "EODHD"
  alpha_vantage: "Alpha Vantage"
  mfapi: "MFAPI.in"
  your_provider: "Your Provider"    # ← add here

Add hint text (under provider_selection):

your_provider_hint: "requires API key, N calls/day limit"

Add settings section (for the API key input):

your_provider_settings:
  title: "Your Provider"
  description: "Get your API key from https://yourprovider.com/dashboard"
  label: "API Key"
  env_configured_message: "The YOUR_PROVIDER_API_KEY environment variable is set."

Also add a display name in config/locales/en.yml under securities.providers:

securities:
  providers:
    your_provider: "Your Provider"

This is used in the combobox dropdown to show which provider each search result comes from.

Step 6: Test

Manual Testing

# In rails console:
provider = Provider::YourProvider.new("your_api_key")

# Search
response = provider.search_securities("AAPL")
response.success? # => true
response.data     # => [SecurityConcept::Security(...), ...]

# Price
response = provider.fetch_security_price(symbol: "AAPL", exchange_operating_mic: "XNAS", date: Date.current)
response.data.price    # => 150.25
response.data.currency # => "USD"

# Historical prices
response = provider.fetch_security_prices(symbol: "AAPL", exchange_operating_mic: "XNAS", start_date: 30.days.ago.to_date, end_date: Date.current)
response.data.size # => ~30

Enable and Search

# Enable the provider
Setting.securities_providers = "your_provider"

# Search via the app's multi-provider system
results = Security.search_provider("AAPL")
results.map { |s| [s.ticker, s.price_provider] }
# => [["AAPL", "your_provider"]]

# Create a security with your provider
security = Security::Resolver.new("AAPL", exchange_operating_mic: "XNAS", price_provider: "your_provider").resolve
security.price_provider # => "your_provider"

# Import prices
security.import_provider_prices(start_date: 30.days.ago.to_date, end_date: Date.current)
security.prices.count

How It All Connects

Search Flow

When a user types in the securities combobox:

1. SecuritiesController#index calls Security.search_provider(query) (app/models/security/provided.rb)
2. search_provider queries all enabled providers concurrently using Concurrent::Promises with an 8-second timeout per provider
3. Results are deduplicated (key: ticker|exchange|provider) and ranked by relevance
4. Each result's ComboboxOption#id is "TICKER|EXCHANGE|PROVIDER" (e.g., "AAPL|XNAS|your_provider")
5. When the user selects one, price_provider is stored on the Security record

Price Fetch Flow

When prices are needed:

1. Security#price_data_provider looks up the provider by security.price_provider
2. If the assigned provider is unavailable (disabled in settings), returns nil — the security is skipped, not silently switched
3. If no provider assigned, falls back to the first enabled provider
4. Security::Price::Importer calls fetch_security_prices and gap-fills missing dates using LOCF (last observation carried forward)
5. Prices are upserted in batches of 200

Provider Resolution Priority

security.price_provider present?
  ├── YES → Security.provider_for(price_provider)
  │         ├── Provider enabled & configured → use it
  │         └── Provider unavailable → return nil (skip security)
  └── NO  → Security.providers.first (first enabled provider)

Checklist

• Provider class at app/models/provider/your_provider.rb
  • Inherits from Provider
  • Includes SecurityConcept
  • Implements search_securities, fetch_security_info, fetch_security_price, fetch_security_prices
  • Returns correct Data.define types
  • All methods wrapped in with_provider_response
  • Exchange names mapped to MIC codes
  • Currency handling (cached or mapped)
  • Rate limiting if applicable
• Registry entry in app/models/provider/registry.rb
  • Added to available_providers for :securities
  • Private factory method with ENV + Setting fallback
• Setting field in app/models/setting.rb (if API key needed)
  • field :your_provider_api_key
  • Added to ENCRYPTED_FIELDS
• Settings UI in hostings controller/view
• Translations in config/locales/
  • Provider name in provider_selection.providers
  • Provider name in securities.providers (for combobox display)
  • Hint text and settings section
• Tested: search, single price, historical prices, info