Postmortem
We recently deployed an authentication related infrastructure change that got overwhelmed at market open due to high request rate on our api.
This was triggering several alerts as well as customer complaints about partial HTTP 5xx error responses in our API. Once we identified the issue, we rolled back the change.
Impact: In the first 17 minutes of the market open, we had approximately 10% global error rate in our API (meaning 90% of the requests were still going through without impact).
Posted Sep 05, 2025 - 10:17 EDT
Resolved
We have confirmed that we are stabilized
Posted Sep 05, 2025 - 09:55 EDT
Monitoring
We have identified the issue and have addressed it. We are now monitoring
Posted Sep 05, 2025 - 09:48 EDT
Investigating
We are seeing intermittent issues that is causing slowing in API requests. The team is actively investigating the issue
Posted Sep 05, 2025 - 09:46 EDT
This incident affected: Live Trading API (Account API, Orders API, Positions API, Assets API, Trade Update Streaming), Paper Trading API (Paper Account API, Paper Order API, Paper Positions API, Paper Assets API), Accounts (Transfers, Bank Links, New Account Onboarding, Turbo Tax Integration), Market Data Streaming API (wss://stream.data.alpaca.markets/v2/sip, wss://stream.data.alpaca.markets/v2/iex, wss://stream.data.alpaca.markets/v1beta3/crypto/us, wss://stream.data.alpaca.markets/v1beta1/news, wss://stream.data.alpaca.markets/v1beta1/otc, wss://stream.data.alpaca.markets/v1beta1/opra), Market Data Crypto REST API (https://data.alpaca.markets/v1beta3/crypto/us/bars, https://data.alpaca.markets/v1beta3/crypto/us/latest/bars, https://data.alpaca.markets/v1beta3/crypto/us/latest/orderbooks, https://data.alpaca.markets/v1beta3/crypto/us/latest/trades, https://data.alpaca.markets/v1beta3/crypto/us/latest/quotes, https://data.alpaca.markets/v1beta3/crypto/us/trades, https://data.alpaca.markets/v1beta3/us/crypto/quotes, https://data.alpaca.markets/v1beta3/crypto/us/snapshots), Market Data Stocks REST API (https://data.alpaca.markets/v2/stocks/auctions, https://data.alpaca.markets/v2/stocks/bars, https://data.alpaca.markets/v2/stocks/trades, https://data.alpaca.markets/v2/stocks/quotes, https://data.alpaca.markets/v2/stocks/snapshots, https://data.alpaca.markets/v2/stocks/trades/latest, https://data.alpaca.markets/v2/stocks/quotes/latest, https://data.alpaca.markets/v2/stocks/meta/conditions, https://data.alpaca.markets/v2/stocks/meta/exchanges), Market Data Options REST API (https://data.alpaca.markets/v1beta1/options/bars, https://data.alpaca.markets/v1beta1/options/quotes/latest, https://data.alpaca.markets/v1beta1/options/trades/latest, https://data.alpaca.markets/v1beta1/options/trades, https://data.alpaca.markets/v1beta1/options/snapshots), Broker API (broker.accounts.get, broker.accounts.account_id.transfers.get, broker.assets.get, broker.trading.accounts.account_id.account.get, broker.trading.accounts.account_id.orders.get, broker.accounts.account_id.documents.get, broker.journals.get, broker.ledgers.get, broker.accounts.activities.get, broker.accounts.account_id.recipient_banks.get, broker.events.accounts.status.get, broker.events.trades.get, broker.events.journals.status.get), Broker API (Sandbox) (broker.accounts.get, broker.accounts.account_id.transfers.get, broker.assets.get, broker.trading.accounts.account_id.account.get, broker.trading.accounts.account_id.orders.get, broker.accounts.account_id.documents.get, broker.journals.get, broker.ledgers.get, broker.accounts.activities.get, broker.accounts.account_id.recipient_banks.get, broker.events.accounts.status.get, broker.events.trades.get, broker.events.journals.status.get), and Dashboard, Broker Dashboard.