High DB connection usage creating intermittent API failure

Incident Report for Alpaca

Postmortem

High usage of database connections was identified around 6:00 PM ET. The team was able to identify an application process that was creating contention due to a lock wait, which persisted even after the request timed out. Both the application and the database were restarted to recover. The issue persisted intermittently but eventually resolved.

Posted Oct 08, 2025 - 10:01 EDT

Resolved

This incident has been resolved.

Posted Oct 08, 2025 - 08:57 EDT

Update

We have identified and resolved the issue that was affecting our systems. Since implementing the fix, all systems have been operating normally with no recurrence of the problem.
We have been closely monitoring our systems over the past hour, and all indicators show stable performance. Our monitoring infrastructure continues to track system health to detect any potential issues early. Our engineering team remains available to respond immediately if any concerns arise.

Posted Oct 07, 2025 - 21:45 EDT

Update

We are continuing to monitor for any further issues.

Posted Oct 07, 2025 - 21:37 EDT

Monitoring

We will continue to monitor the system and take appropriate the action.

Posted Oct 07, 2025 - 21:37 EDT

Update

System is working as expected. We are monitoring the performance

Posted Oct 07, 2025 - 21:28 EDT

Update

We are seeing spike in connections again. Teams are actively working on it

Posted Oct 07, 2025 - 21:07 EDT

Update

After restarting, the database connections are under control. We are monitoring the system.

Posted Oct 07, 2025 - 20:27 EDT

Update

Database is restarted and team is monitoring it

Posted Oct 07, 2025 - 19:54 EDT

Update

Team is still working on the recovery.

Posted Oct 07, 2025 - 19:42 EDT

Update

We are restarting the DB and API calls are expected to fail.

Posted Oct 07, 2025 - 19:26 EDT

Update

We are continuing to investigate this issue.

Posted Oct 07, 2025 - 19:22 EDT

Update

We are still working on identifying the underlying cause of DB connections

Posted Oct 07, 2025 - 19:20 EDT

Investigating

We are seeing high DB usage causing the API to fail. We are checking internally.

Posted Oct 07, 2025 - 19:05 EDT

This incident affected: Live Trading API (Account API, Orders API, Positions API, Assets API, Trade Update Streaming), Accounts (Transfers), and Dashboard.