2026-08-22
Incident Report — 22 August 2026
Status: Resolved
All affected data has been repaired and verified.
If you persisted responses from userCompletedTrades or userCompletedTradesByTime fetched between 22 August 15:00 UTC and 24 August 08:00 UTC, please refetch that data from our API.
No other action is required.
Summary
At 13:57 UTC on 22 August, our production non-validator node began processing blocks at approximately 6 blocks per second, compared with the approximately 14 blocks per second the chain maintained during that window. This caused our ingestion pipeline to fall significantly behind.
The underlying cause was degraded network performance at our hosting provider, Vultr, in its Tokyo region. During the incident, our production host experienced approximately 50% packet loss on the affected network path.
Vultr's network conditions recovered at approximately 23:00 UTC on 22 August, just over nine hours after the initial onset.
We restored service earlier, at 15:30 UTC, by moving traffic to a secondary environment in another datacenter.
All affected data has since been repaired and verified, and at 08:00 UTC on 24 August we moved traffic back to the fully restored production environment.
Impact
The incident affected our data ingestion pipeline and, as a result, the freshness and availability of data served through our API and WebSocket.
13:57–14:30 UTC — Data remained available but was severely delayed, with delays measured in minutes rather than milliseconds.
14:30–15:30 UTC — The API and WebSocket were effectively unavailable.
15:30 UTC onward — Service was restored by failing over to a secondary environment in another datacenter. During this period, average data latency was approximately 100 ms higher because of the secondary environment's peering setup.
During the failover window —
userCompletedTradesanduserCompletedTradesByTimecould return incomplete historical results for data from 26 June onward. This was the only known case where an API request could return an incomplete result rather than a delayed or unavailable one.08:00 UTC, 24 August — Traffic was moved back to the fully restored production environment.
All incomplete historical data has since been repaired and verified.
Action required for persisted trade data
If you persisted results from userCompletedTrades or userCompletedTradesByTime fetched between 22 August 15:00 UTC and 24 August 08:00 UTC, please refetch those results from our API. Results covering any period from 26 June onward may have been incomplete.
We use 15:00 UTC as a conservative start time for this window, preceding the completed failover at 15:30 UTC.
No other action is required.
Root cause
The immediate cause of the incident was severe packet loss on the network path to our production infrastructure at Vultr's Tokyo region.
TCP-mode traceroutes run in both directions, minutes apart, consistently showed loss and elevated latency once traffic entered Vultr's internal network. The measurement below is the inbound path to our production host:
Intermediate-hop traceroute results can be misleading because routers may deprioritize probe responses, so we did not use intermediate-hop loss alone to identify the fault. The significant result was the 43.3% packet loss measured at the destination, together with severe end-to-end throughput degradation.
A 300 MB download over the affected path ran at approximately 41 KB/s, compared with a normal line rate measured in tens of MB/s. After the provider's network recovered, the destination returned to 0% loss and 2.1 ms average latency, while some intermediate hops continued to report elevated probe latency.
As an additional control, we provisioned a fresh host at a different provider with an identical configuration. That host synchronized from empty in approximately six minutes.
Taken together, these measurements isolated the problem to the network path within Vultr's infrastructure rather than the node software or its configuration.
Network conditions at Vultr recovered at approximately 23:00 UTC on 22 August.
Timeline
All times are UTC.
13:57, 22 August — Block ingestion rate drops from approximately 14 blocks/s to 6 blocks/s. There is no crash, application error, or restart.
14:03 — Alerting fires on delayed data.
14:03–15:30 — We investigate the node slowdown, restart the non-validator, test alternative peer connections, diagnose the network path, and prepare the secondary environment for failover.
14:30 — API and WebSocket availability deteriorates to the point of being effectively unavailable.
15:30 — Failover to the secondary datacenter completes and service is restored.
23:00, 22 August — Packet loss on the Vultr network path clears.
23–24 August — Affected data is repaired and hash-verified.
08:00, 24 August — Traffic is returned to the fully restored production environment.
What went wrong on our side
Although the triggering event was a network failure at our hosting provider, the duration of the customer impact was longer than it should have been.
We took too long to identify the failure as a network issue
We have previously experienced non-validator nodes falling behind for reasons unrelated to network connectivity. Because of that history, our initial response was to treat this as another node-level synchronization issue.
We restarted the non-validator and attempted to connect it to a different peer before concluding that the underlying problem was the network path itself.
Those actions did not resolve the issue and delayed our decision to fail over.
We took too long to fail over
Once the network issue became apparent, we should have moved traffic to another datacenter immediately and continued diagnosing the underlying problem after service had been restored.
Instead, too much of the diagnosis happened before the failover decision.
We are changing this operational priority: restore service first when an incident can be isolated to a single provider or datacenter, then continue root-cause analysis.
The failover environment had drifted from production
Our secondary environment had changes ahead of the production environment.
Before we could safely fail over, those changes had to be rolled back to match the production state. This added unnecessary time to the recovery process.
A failover environment should be ready to take production traffic immediately, but ours was not.
Changes already made
Fail over first when a provider-specific failure is confirmed. Once an incident is isolated to a single hosting provider or datacenter, we move traffic to a healthy environment immediately and continue diagnosis afterward. This is now our standing operational priority.
Underway
Network-level observability
We are adding continuous monitoring and alerting for per-peer and per-hop packet loss and latency, displayed alongside application and ingestion metrics. This should make network failures distinguishable from software or synchronization failures much earlier in an incident.
Keeping the failover environment aligned with production
We are changing how our secondary environment is managed so that it remains able to receive production traffic without manual rollback or reconciliation first. Today this is a policy rather than an enforced mechanism, and closing that gap is part of this work.
Multi-node, multi-datacenter ingestion
We are changing the ingestion architecture to ingest data from multiple nodes distributed across multiple datacenters and multiple hosting providers.
The goal is to remove the network path to any single node, datacenter, or provider as a single point of failure for the ingestion pipeline.
Planned
Multi-datacenter API and WebSocket serving
We plan to serve the API and WebSocket from more than one datacenter behind health-checked routing.
If a host, datacenter, or network path becomes unhealthy, traffic should automatically move away from it rather than waiting for a manual switchover.
Combined with multi-node ingestion, this architecture is designed to prevent a single degraded provider or network path from stopping the entire pipeline.
Questions
If anything in your own data still looks wrong, send us the endpoint and affected time range and we will check it against source data.
— Hydromancer team
Last updated