What happened

We were experiencing hardware failures in our underlying cloud infrastructure, which was mostly invisible due to planned redundancy in our architecture. However, the underlying job scheduler did not recover its internal state properly and when it recovered from the failure, it started to rerun one of the jobs that is originally scheduled before trading hours. This job is responsible for updating cash balances and equity data from the ledger system. Part of its processing, it resets these values to 0 in case of no records for a given trading day. Since the job was running unexpectedly during trading hours, it started to zero out accounts mistakenly.

We detected the discrepancy within a couple of minutes and interrupted the unexpected process, and started to recover the proper account data from the ledger system.

Remediation

Although these failures are unlikely to happen again, we are evaluating several options to make our services more resilient from such failures:

• Upcoming versions of Kubernetes (base infrastructure component in our system) will come with an improved support of job status tracking (to prevent such failure scenarios)
• We are evaluating a more robust workflow engine
• We are evaluating to introduce further failure prevention logic to avoid running such jobs during trading hours

We sincerely apologise for the inconvenience this issue may caused to you and we remain committed to continuously improve our services.