On the morning of Saturday, April 10th, as early as 9:30 AM Eastern time, some users started to see their account equity value flickering and eventually displaying inaccurate values. Around the same time, our engineering team started to look into the issue and realized that we were receiving real time market data updates via SIP feed with test data prices for NASDAQ listed stocks despite the market being closed at the time.
NASDAQ schedules test trade sessions on a pre-scheduled Saturday of each month, and the market updates are sent through the SIP as “unmarked data”. As we rolled out our new market data integration last month, this was our first time going through this process. The real-time account equity calculation is based on the last sale prices from this market data feed, and a large number of tickers, including AAPL and QQQ, had prices such as $1.1 as the last sale. Ideally, this data would be excluded from our data consumption and processing service but it was treated as if the test data was real trade data (as it was unmarked). As a result, a number of customer accounts temporarily displayed wrong market values on positions for these tickers, and the equity value, which is based on the position market value, temporarily displayed an inaccurate amount.
Our engineers got together as soon as we noticed the issue and went into the war room. The first few hours were spent to isolate and identify the problems. Then, we remediated data stored in several components in our backend system consuming the real time market data for various purposes. This took us another few hours. By around 7 PM Eastern time, we successfully purged any test data that was stored in our system and also confirmed the new data updates for the test session had stopped a few hours earlier.
We will implement a solution so that this event doesn’t occur again. As our new market data integration just went live last month, we realize we didn’t have enough cases to cover such events. We apologize for the surprises on Saturday morning and appreciate your patience. As always, your feedback helps us make our platform more robust as we continue to improve our API platform.