> For the complete documentation index, see [llms.txt](https://docs.hydromancer.xyz/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.hydromancer.xyz/readme/incidents/2026-08-22.md).

# 2026-08-22

Incident Report — 22 August 2026

{% hint style="success" %}
**Status: Resolved**

All affected data has been repaired and verified.

If you persisted responses from `userCompletedTrades` or `userCompletedTradesByTime` fetched between **22 August 15:00 UTC and 24 August 08:00 UTC**, please refetch that data from the Hydromancer API.

**No other action is required.**
{% endhint %}

### Summary

At **13:57 UTC on 22 August**, the Hydromancer production non-validator node began processing blocks at approximately **6 blocks per second**, compared with the approximately **14 blocks per second** the chain maintained during that window. This caused the Hydromancer ingestion pipeline to fall significantly behind.

The underlying cause was degraded network performance at Hydromancer's hosting provider, Vultr, in its Tokyo region. During the incident, the Hydromancer production host experienced approximately **50% packet loss on the affected network path**.

Vultr's network conditions recovered at approximately **23:00 UTC on 22 August**, just over nine hours after the initial onset.

Hydromancer restored service earlier, at **15:30 UTC**, by moving traffic to a secondary environment in another datacenter.

All affected data has since been repaired and verified, and at **08:00 UTC on 24 August** Hydromancer moved traffic back to the fully restored production environment.

### Impact

The incident affected the Hydromancer data ingestion pipeline and, as a result, the freshness and availability of data served through the Hydromancer API and WebSocket.

* **13:57–14:30 UTC** — Data remained available but was severely delayed, with delays measured in minutes rather than milliseconds.
* **14:30–15:30 UTC** — The API and WebSocket were effectively unavailable.
* **15:30 UTC onward** — Service was restored by failing over to a secondary environment in another datacenter. During this period, average data latency was approximately **100 ms higher** because of the secondary environment's peering setup.
* **During the failover window** — `userCompletedTrades` and `userCompletedTradesByTime` could return incomplete historical results for data from **26 June onward**. This was the only known case where an API request could return an incomplete result rather than a delayed or unavailable one.
* **08:00 UTC, 24 August** — Traffic was moved back to the fully restored production environment.

All incomplete historical data has since been repaired and verified.

{% hint style="warning" %}

#### Action required for persisted trade data

If you persisted results from `userCompletedTrades` or `userCompletedTradesByTime` fetched between **22 August 15:00 UTC and 24 August 08:00 UTC**, please refetch those results from the Hydromancer API. Results covering any period from **26 June onward** may have been incomplete.

Hydromancer uses **15:00 UTC** as a conservative start time for this window, preceding the completed failover at 15:30 UTC.

No other action is required.
{% endhint %}

### Root cause

The immediate cause of the incident was severe packet loss on the network path to Hydromancer production infrastructure at Vultr's Tokyo region.

TCP-mode traceroutes run in both directions, minutes apart, consistently showed loss and elevated latency once traffic entered Vultr's internal network. The measurement below is the inbound path to the Hydromancer production host:

```
HOP                     Loss%   Snt    Avg   Best   Wrst
3. <peer-network>         0.0%    30   38.8    0.3  1033.
4. <transit>              0.0%    30  154.0    1.2  2688.
5. <provider-internal>   56.7%    30    6.1    1.2    39.3   <- Vultr internal
6. <our-host>            43.3%    30  122.8    0.6  1055.    <- destination
```

Intermediate-hop traceroute results can be misleading because routers may deprioritize probe responses, so Hydromancer did not use intermediate-hop loss alone to identify the fault. The significant result was the **43.3% packet loss measured at the destination**, together with severe end-to-end throughput degradation.

A **300 MB download** over the affected path ran at approximately **41 KB/s**, compared with a normal line rate measured in tens of MB/s. After the provider's network recovered, the destination returned to **0% loss and 2.1 ms average latency**, while some intermediate hops continued to report elevated probe latency.

As an additional control, Hydromancer provisioned a fresh host at a different provider with an identical configuration. That host synchronized from empty in approximately **six minutes**.

Taken together, these measurements isolated the problem to the network path within Vultr's infrastructure rather than the node software or its configuration.

Network conditions at Vultr recovered at approximately **23:00 UTC on 22 August**.

### Timeline

All times are UTC.

* **13:57, 22 August** — Block ingestion rate drops from approximately 14 blocks/s to 6 blocks/s. There is no crash, application error, or restart.
* **14:03** — Alerting fires on delayed data.
* **14:03–15:30** — Hydromancer investigates the node slowdown, restarts the non-validator, tests alternative peer connections, diagnoses the network path, and prepares the secondary environment for failover.
* **14:30** — API and WebSocket availability deteriorates to the point of being effectively unavailable.
* **15:30** — Failover to the secondary datacenter completes and service is restored.
* **23:00, 22 August** — Packet loss on the Vultr network path clears.
* **23–24 August** — Affected data is repaired and hash-verified.
* **08:00, 24 August** — Traffic is returned to the fully restored production environment.

### What went wrong on our side

Although the triggering event was a network failure at our hosting provider, the duration of the customer impact was longer than it should have been.

#### Hydromancer took too long to identify the failure as a network issue

Hydromancer had previously experienced non-validator nodes falling behind for reasons unrelated to network connectivity. Because of that history, the initial response treated this as another node-level synchronization issue.

Hydromancer restarted the non-validator and attempted to connect it to a different peer before concluding that the underlying problem was the network path itself.

Those actions did not resolve the issue and delayed our decision to fail over.

#### Hydromancer took too long to fail over

Once the network issue became apparent, Hydromancer should have moved traffic to another datacenter immediately and continued diagnosing the underlying problem after service had been restored.

Instead, too much of the diagnosis happened before the failover decision.

Hydromancer is changing this operational priority: **restore service first when an incident can be isolated to a single provider or datacenter, then continue root-cause analysis.**

#### The failover environment had drifted from production

The Hydromancer secondary environment had changes ahead of the production environment.

Before Hydromancer could safely fail over, those changes had to be rolled back to match the production state. This added unnecessary time to the recovery process.

A failover environment should be ready to take production traffic immediately, but Hydromancer's was not.

### Changes already made

* **Fail over first when a provider-specific failure is confirmed.** Once an incident is isolated to a single hosting provider or datacenter, Hydromancer moves traffic to a healthy environment immediately and continues diagnosis afterward. This is now Hydromancer's standing operational priority.

### Underway

#### Network-level observability

Hydromancer is adding continuous monitoring and alerting for per-peer and per-hop packet loss and latency, displayed alongside application and ingestion metrics. This should make network failures distinguishable from software or synchronization failures much earlier in an incident.

#### Keeping the failover environment aligned with production

Hydromancer is changing how its secondary environment is managed so that it remains able to receive production traffic without manual rollback or reconciliation first. Today this is a policy rather than an enforced mechanism, and closing that gap is part of this work.

#### Multi-node, multi-datacenter ingestion

Hydromancer is changing the ingestion architecture to ingest data from multiple nodes distributed across **multiple datacenters and multiple hosting providers**.

The goal is to remove the network path to any single node, datacenter, or provider as a single point of failure for the ingestion pipeline.

### Planned

#### Multi-datacenter API and WebSocket serving

Hydromancer plans to serve the API and WebSocket from more than one datacenter behind health-checked routing.

If a host, datacenter, or network path becomes unhealthy, traffic should automatically move away from it rather than waiting for a manual switchover.

Combined with multi-node ingestion, this architecture is designed to prevent a single degraded provider or network path from stopping the entire pipeline.

### Questions

If anything in your own data still looks wrong, send Hydromancer the **endpoint and affected time range**. Hydromancer will check it against source data.

— **Hydromancer team**