Incorrect values in balances
Incident Report for Alpaca
Postmortem

We were experiencing issues in our trade posting process during EOD on 11/03 due to network timeouts. As a consequence, roughly half of the trades from yesterday were missing from our books before closing the day and got only booked on the next day (11/04) in our book keeping system. This led to inaccurate balances on some of the accounts in the timeframe of 2:15am - 4:30am ET, since the cash balance calculation was mistakenly ignoring the trades that were only booked for the next trading day.

To prevent such issues to happen in the future, we are planning the following actions:

  • we are increasing the network timeouts to be more resilient in case of similar situations
  • we are improving our monitoring to detect slow/late completion of the trade posting process

Client's balances and buying power may still be incorrect. In particular, buying power will not be correct for many users until batch runs overnight.

Posted Nov 04, 2022 - 11:46 EDT

Resolved
This incident has been resolved.
Posted Nov 04, 2022 - 11:43 EDT
Update
Post mortem

We were experiencing issues in our trade posting process during EOD on 11/03 due to network timeouts. As a consequence, roughly half of the trades from yesterday were missing from our books before closing the day and got only booked on the next day (11/04) in our book keeping system. This led to inaccurate balances on some of the accounts in the timeframe of 2:15am - 4:30am ET, since the cash balance calculation was mistakenly ignoring the trades that were only booked for the next trading day.

To prevent such issues to happen in the future, we are planning the following actions:
* we are increasing the network timeouts to be more resilient in case of similar situations
* we are improving our monitoring to detect slow/late completion of the trade posting process
Posted Nov 04, 2022 - 11:43 EDT
Update
We are continuing to monitor for any further issues.
Posted Nov 04, 2022 - 11:42 EDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Nov 04, 2022 - 08:57 EDT
Identified
We had a reconciliation issue this morning around 2:15 ET which caused balances showing incorrect values. We were able to restore balances around 4:20 ET with a quick fix, however, we are still working on the complete restoration
Posted Nov 04, 2022 - 04:31 EDT