A delisted symbol caused an error in the end of day consolidation of our ledger system. This caused that the bookings for the previous trading day could not be closed. However, the API servers were already assuming the current booking day is the current trading day (which is the case normally). This caused that any newly created journals got booked for the previous day in the ledger, while the api server was already assuming the previous day’s balance is closed and did not consider those events as intraday cash flow. The end result is that in the API, even after an executed journal, the customer balance was not reflecting the cash movement.

Follow up actions Alpaca is planning to implement to prevent such issues in the future:

• We are designing better escalation process to tackle EOD related issues sooner. This way, such issues may not be customer facing (as long as the remediation can happen before the next trading day’s BOD process).
• Implementing a safety check that ensures that the API servers are not syncing closing balances from the ledger until that trading day is still open for bookings.