The root cause of the issue has been identified and remediated.
Between 02:15:02 to 03:32:57 ET on January 30th 2024, non trade activity that occurred between January 26-30th, 2024 may have been incorrectly updated with erroneous values during our beginning of day (BOD) synchronization processes. The root cause was identified as ineffectual memory reassignment in an internal streaming endpoint between our ledger and other services. After identifying and fixing the root cause, all BOD processes were rerun to fix all erroneous non trade activity.
We are monitoring and identifying any other subsequent client initiated activity that may have occurred between 2:15AM to 4:00AM ET as a result.
We will share further updates once this monitoring phase has been completed. In the interim, please reach out to our customer success and support teams if you have any questions.
Posted Jan 30, 2024 - 10:11 EST
Investigating
We detected invalid account balances during ~2am ET 2024-01-30 and ~4am ET on accounts which had a journal activity between 2024-01-26 - 2024-01-29. The balances are showing correct now, but the investigations are still ongoing. Invalid balances might lead to account activity that should be prevented based on the correct cash balance - we are working to identify such cases.
Posted Jan 30, 2024 - 06:50 EST
This incident affected: Live Trading API (Account API) and Broker API (Sandbox) (broker.accounts.get, broker.trading.accounts.account_id.account.get, broker.accounts.activities.get).