A change introduced during the deployment of one of our services after (after market hours on Aug 23, 2022 at 20:50 EST) adversely impacted the cash balance calculations for a select accounts that had intra-day trading activity. Shortly after the deployment, our internal heuristics alerted our team of the anomalous behavior. After assessing the integrity of our remediation plan in a secondary environment, the release was rolled back and finally patched with a fix at 23:10 EST where all balance calculations and account state was corrected.
Root cause: A change was introduced to not include certain asset classes when calculating intra-day net cash flow related calculation. Part of this change included an optimization that lead to unintentionally omitting required attributes that are summarized in our intra-day and unsettled cash flow values upon account hydration in memory. When the in-memory state was rebuilt, the balance related fields didn't correctly attribute for intra-day activity.
Impact: * Certain accounts were able to place open orders (this happened outside of the extending trading session when markets were closed) in excess of their true buying power. * Certain accounts were able to place withdrawal requests in excess of their buying power.
Follow up: We were able to ascertain why this issue didn't surface in our various test suites and continuous deployment procedures. Secondly, multiple engineering teams will be prioritizing additional smoke and regression testing as part of our continuous deployment and release procedures across all our services.
Our teams have been in touch with impacted accounts and partners. Please reach out to our support team if you need any further assistance or clarity.