BOD process delayed

Incident Report for Alpaca

Postmortem

Service Status Update: Account Data Processing Delay (January 6, 2026)

What Happened

On the morning of January 6, 2026, our platform experienced a delay in the standard start-of-day data synchronisation process. This was caused by a temporary resource constraint within our processing environment, which prevented the internal jobs responsible for updating account balances and positions from initialising at their scheduled time.

Upon identifying the capacity issue, our engineering teams manually adjusted system resources to allow the processing to begin. However, the initial recovery attempt encountered a synchronisation timeout, requiring a secondary re-run of the update job to ensure complete data accuracy. The process was successfully completed, and all systems returned to a healthy state by 06:44 AM ET.

 

 Impact

The delay primarily affected the visibility of up-to-date account information for our partners and their end-users.

  • Data Accuracy: Users may have seen delayed updates to their account balances, holdings, and recent transaction history.
  • Transaction Processing: Some users may have encountered "insufficient balance" errors during fund transfer attempts, as the system was awaiting the finalised balance updates from the previous business day.
  • Data Integrity: We can confirm that no data was lost and all funds remain entirely secure. This was a processing delay only, and all account activities have since been reconciled correctly.

 

 Resolution

Our technical team took the following immediate actions to resolve the incident:

  • Capacity Expansion: At 04:10 AM ET, we manually increased the available system capacity to bypass the resource bottleneck and allow the pending tasks to execute.
  • Job Restoration: After the initial job timed out, we initiated a controlled re-run of the entire start-of-day sequence to ensure every account was accurately updated.
  • Final Verification: We conducted a full audit of account balances and positions to confirm that all data was synchronized and the service was fully restored by 06:44 AM ET. 

 Preventative Measures

We are committed to maintaining the highest levels of reliability for our partners. To prevent a recurrence, we are implementing the following improvements:

  • System Capacity Review: We are auditing our resource allocation policies to ensure that core financial processing tasks always have priority and cannot be delayed by auxiliary or non-critical system activities.
  • Resilience & Monitoring: We are enhancing our automated alerting for "pending" internal tasks to identify and resolve resource constraints before they impact the start of the business day.

Process Optimization: Our engineering team is optimizing the timeout thresholds and retry logic for long-running data jobs to ensure more resilient performance during peak processing windows.

Posted Jan 07, 2026 - 11:56 EST

Resolved

Since we are no longer seeing issue ,Marking this resolved.
Posted Jan 06, 2026 - 06:59 EST

Update

The BOD job has now completed successfully, and all account data (balances, positions, activities ) should be up to date.
Posted Jan 06, 2026 - 06:50 EST

Update

We are continuing to monitor for any further issues.
Posted Jan 06, 2026 - 06:31 EST

Update

All balances have now been successfully and accurately updated.
Posted Jan 06, 2026 - 06:28 EST

Update

We are still monitoring the job ,will keep you posted here.
Posted Jan 06, 2026 - 06:25 EST

Update

The rerun has been progressing without any errors since it was initiated. Our team is actively monitoring the job, and we'll promptly share any further updates as they become available.
Posted Jan 06, 2026 - 05:48 EST

Update

Job is running without any issue as of now, we are actively monitoring and will keep posting status here.
Posted Jan 06, 2026 - 05:25 EST

Monitoring

B.O.D. Job failed due to timeout , Job is rerunning . We will keep you posted for updates.
Posted Jan 06, 2026 - 05:11 EST

Identified

We have identified the issue with resources on the node and it has been rectified. BOD process is running. ETA is 90 Mins by now.
Posted Jan 06, 2026 - 04:18 EST

Update

BOD process is running: The resource constraint issues have been resolved, and the Beginning-of-Day (BOD) process is now running.
Estimated time: The estimated completion time is 2 hours from now
Posted Jan 06, 2026 - 04:17 EST

Investigating

The BOD process delay is caused by a GoBroker BOD pod being stuck in a Pending state due to resource constraints.
This is not a system outage. The engineering team is actively investigating the underlying resource and node scheduling issue.
Posted Jan 06, 2026 - 03:56 EST
This incident affected: Dashboard and Accounts (Buying Power).