Live Message Chunking

Opt-in message chunking for reliable delivery of large batched events

Chunking is opt-in and backward compatible. Existing clients require zero changes. Enable chunking for improved connection stability during high-volume periods.

Why Chunking?

During high-volume trading periods, a single block can contain thousands of fills and other batched events. Sending these as a single live WebSocket message can cause connection drops due to frame size limits and TCP backpressure.

Chunking splits large live batches into smaller pieces, eliminating disconnects while preserving all data.

Enabling Chunking

Add liveFormat=chunked-v1 to your connection URL:

wss://api.hydromancer.xyz/ws?token=your-api-key&liveFormat=chunked-v1

The server confirms the format in the connected message:

{
  "type": "connected",
  "clientId": "abc-123",
  "sessionId": "def-456",
  "liveFormat": "chunked-v1"
}

liveFormat is not persisted on the session. You must include liveFormat=chunked-v1 in the URL on every connection, including reconnects. If omitted, the connection falls back to legacy (unchunked) format. You can detect this by checking whether the connected or reconnected message includes the liveFormat field.

Chunked Message Format

When chunking is enabled, batched messages include three additional fields:

{
  "type": "allFills",
  "seq": 42,
  "cursor": "924885600:1710541070123:5",
  "fills": [...],
  "chunk": 2,
  "totalChunks": 4,
  "batchId": 48291
}

Field	Type	Description
chunk	integer	1-indexed chunk number within the batch
totalChunks	integer	Total number of chunks for this batch
batchId	integer	Unique identifier for the batch — use this as the reassembly key

Chunking for live messages is currently count-based only. The server splits a batch when it exceeds the configured per-message item limit. It does not inspect serialized JSON byte size when deciding where to split.

When messages are not chunked

If a batch fits within a single message (the common case), you still receive the chunk fields:

{
  "chunk": 1,
  "totalChunks": 1,
  "batchId": 12345
}

This means your client can use a single code path for all messages.

Affected Subscription Types

Chunking applies to all batched subscription types:

Subscription	Data field
allFills, userFills, builderFills, liquidationFills	fills
userOrderUpdates, builderOrderUpdates	updates
allCompletedTrades, userCompletedTrades, builderCompletedTrades	trades
builderLiquidations, allBuilderLiquidations	liquidations
allTwapStatusUpdates	updates
allUserNonFundingLedgerEvents, userNonFundingLedgerEvents	events
setOracleUpdates	updates
allLeverageUpdates, userLeverageUpdates	updates
allIsolatedMarginUpdates, userIsolatedMarginUpdates	updates

| allCandles | data |

Non-batched subscriptions (l2Book, l2BookDiff, bbo, candles, fundingRates, activeAssetCtx) are unaffected.

Sequence Numbers and Cursors

Each chunk is a standalone message in the sequence stream:

• seq increments per chunk (not per batch). A 4-chunk batch consumes 4 sequence numbers.

• cursor on each replay-supported channel chunk reflects the last item in that chunk. setOracleUpdates is the exception: it always sends "0" and does not support replay.

This means:

• Gap detection works the same as without chunking — a missing seq means a missing message.

• On reconnect, replay-supported channels resume from the cursor you provide, not the batch start.

For the most accurate resume point, always reconnect with the last cursor you successfully processed. If you omit cursor and rely on server-stored session state, the fallback resume point is less precise because the session only stores block:timestamp, not the full item-level cursor.

Sequence numbers are monotonically increasing but not necessarily contiguous across batches. Concurrent block processing means another batch's chunks can interleave between chunks of your batch. Use batchId to group chunks, not sequence number contiguity.

Client Integration

For item-by-item consumers (most common)

If you process fills, orders, or trades individually (not requiring full-block atomicity), chunking is transparent. Simply enable it and process each message as before — the items arrive in order, just in smaller batches.

ws.on('message', (data) => {
    const msg = JSON.parse(data);
    if (msg.type === 'allFills') {
        // Process fills — works identically with or without chunking
        for (const fill of msg.fills) {
            processFill(fill);
        }
    }
});

For block-atomic consumers

If you need to reassemble complete blocks before processing, use batchId and chunk/totalChunks:

const pendingBatches = new Map();

ws.on('message', (data) => {
    const msg = JSON.parse(data);
    if (msg.type === 'allFills' && msg.batchId !== undefined) {
        if (!pendingBatches.has(msg.batchId)) {
            pendingBatches.set(msg.batchId, {
                totalChunks: msg.totalChunks,
                chunks: new Map()
            });
        }

        const batch = pendingBatches.get(msg.batchId);
        batch.chunks.set(msg.chunk, msg.fills);

        if (batch.chunks.size === batch.totalChunks) {
            // All chunks received — reassemble in order
            const allFills = [];
            for (let i = 1; i <= batch.totalChunks; i++) {
                allFills.push(...batch.chunks.get(i));
            }
            processBlock(allFills);
            pendingBatches.delete(msg.batchId);
        }
    }
});

Incomplete batch handling: If you disconnect mid-batch, some chunks may be missing. On replay-supported channels, discard any incomplete batchId and rely on replay from your saved cursor. setOracleUpdates does not support replay, so reconnecting resumes live delivery only.

Backward Compatibility

• Legacy clients (no liveFormat parameter): zero wire change. Messages never include chunk, totalChunks, or batchId fields.

• Unknown liveFormat values (e.g., liveFormat=foo): fall back to legacy format silently.

• Reconnect without liveFormat: falls back to legacy even if the previous connection used chunked-v1. The connected/reconnected message will not include liveFormat, allowing clients to detect the fallback.

Reconnection with Chunking

When reconnecting with a session:

1. Include liveFormat=chunked-v1 in the URL (it is not persisted)

2. Optionally include cursor for precise resume (recommended)

3. The reconnected message confirms the format:

{
  "type": "reconnected",
  "clientId": "abc-123",
  "sessionId": "def-456",
  "liveFormat": "chunked-v1"
}

Replay messages use the existing chunked replay format (with chunk and totalChunks fields). Live messages after replay use the chunked-v1 format with batchId.

Configuration

The server splits live batches at 1000 items by default. This limit is configurable server-side and may be adjusted based on monitoring. Clients should not depend on a specific chunk size.