# Live Message Chunking {% hint style="info" %} Chunking is **opt-in** and **backward compatible**. Existing clients require zero changes. Enable chunking for improved connection stability during high-volume periods. {% endhint %} ## Why Chunking? During high-volume trading periods, a single block can contain thousands of fills and other batched events. Sending these as a single live WebSocket message can cause connection drops due to frame size limits and TCP backpressure. Chunking splits large live batches into smaller pieces, eliminating disconnects while preserving all data. ## Enabling Chunking Add `liveFormat=chunked-v1` to your connection URL: ``` wss://api.hydromancer.xyz/ws?token=your-api-key&liveFormat=chunked-v1 ``` The server confirms the format in the `connected` message: ```json { "type": "connected", "clientId": "abc-123", "sessionId": "def-456", "liveFormat": "chunked-v1" } ``` {% hint style="warning" %} **`liveFormat` is not persisted on the session.** You must include `liveFormat=chunked-v1` in the URL on every connection, including reconnects. If omitted, the connection falls back to legacy (unchunked) format. You can detect this by checking whether the `connected` or `reconnected` message includes the `liveFormat` field. {% endhint %} ## Chunked Message Format When chunking is enabled, batched messages include three additional fields: ```json { "type": "allFills", "seq": 42, "cursor": "924885600:1710541070123:5", "fills": [...], "chunk": 2, "totalChunks": 4, "batchId": 48291 } ``` | Field | Type | Description | | ------------- | ------- | ---------------------------------------------------------------- | | `chunk` | integer | 1-indexed chunk number within the batch | | `totalChunks` | integer | Total number of chunks for this batch | | `batchId` | integer | Unique identifier for the batch — use this as the reassembly key | Chunking for live messages is currently **count-based only**. The server splits a batch when it exceeds the configured per-message item limit. It does **not** inspect serialized JSON byte size when deciding where to split. ### When messages are not chunked If a batch fits within a single message (the common case), you still receive the chunk fields: ```json { "chunk": 1, "totalChunks": 1, "batchId": 12345 } ``` This means your client can use a single code path for all messages. ## Affected Subscription Types Chunking applies to all batched subscription types: | Subscription | Data field | | --------------------------------------------------------------------- | -------------- | | `allFills`, `userFills`, `builderFills`, `liquidationFills` | `fills` | | `userOrderUpdates`, `builderOrderUpdates` | `updates` | | `allCompletedTrades`, `userCompletedTrades`, `builderCompletedTrades` | `trades` | | `builderLiquidations`, `allBuilderLiquidations` | `liquidations` | | `allTwapStatusUpdates` | `updates` | | `allUserNonFundingLedgerEvents`, `userNonFundingLedgerEvents` | `events` | | `setOracleUpdates` | `updates` | | `allLeverageUpdates`, `userLeverageUpdates` | `updates` | | `allIsolatedMarginUpdates`, `userIsolatedMarginUpdates` | `updates` | \| `allCandles` | `data` | Non-batched subscriptions (`l2Book`, `l2BookDiff`, `bbo`, `candles`, `fundingRates`, `activeAssetCtx`) are unaffected. ## Sequence Numbers and Cursors Each chunk is a standalone message in the sequence stream: * **`seq`** increments per chunk (not per batch). A 4-chunk batch consumes 4 sequence numbers. * **`cursor`** on each replay-supported channel chunk reflects the last item in that chunk. `setOracleUpdates` is the exception: it always sends `"0"` and does not support replay. This means: * Gap detection works the same as without chunking — a missing `seq` means a missing message. * On reconnect, replay-supported channels resume from the cursor you provide, not the batch start. {% hint style="info" %} For the most accurate resume point, always reconnect with the last `cursor` you successfully processed. If you omit `cursor` and rely on server-stored session state, the fallback resume point is less precise because the session only stores `block:timestamp`, not the full item-level cursor. {% endhint %} {% hint style="info" %} Sequence numbers are **monotonically increasing** but **not necessarily contiguous** across batches. Concurrent block processing means another batch's chunks can interleave between chunks of your batch. Use `batchId` to group chunks, not sequence number contiguity. {% endhint %} ## Client Integration ### For item-by-item consumers (most common) If you process fills, orders, or trades individually (not requiring full-block atomicity), chunking is transparent. Simply enable it and process each message as before — the items arrive in order, just in smaller batches. ```javascript ws.on('message', (data) => { const msg = JSON.parse(data); if (msg.type === 'allFills') { // Process fills — works identically with or without chunking for (const fill of msg.fills) { processFill(fill); } } }); ``` ### For block-atomic consumers If you need to reassemble complete blocks before processing, use `batchId` and `chunk`/`totalChunks`: ```javascript const pendingBatches = new Map(); ws.on('message', (data) => { const msg = JSON.parse(data); if (msg.type === 'allFills' && msg.batchId !== undefined) { if (!pendingBatches.has(msg.batchId)) { pendingBatches.set(msg.batchId, { totalChunks: msg.totalChunks, chunks: new Map() }); } const batch = pendingBatches.get(msg.batchId); batch.chunks.set(msg.chunk, msg.fills); if (batch.chunks.size === batch.totalChunks) { // All chunks received — reassemble in order const allFills = []; for (let i = 1; i <= batch.totalChunks; i++) { allFills.push(...batch.chunks.get(i)); } processBlock(allFills); pendingBatches.delete(msg.batchId); } } }); ``` {% hint style="warning" %} **Incomplete batch handling:** If you disconnect mid-batch, some chunks may be missing. On replay-supported channels, discard any incomplete `batchId` and rely on replay from your saved cursor. `setOracleUpdates` does not support replay, so reconnecting resumes live delivery only. {% endhint %} ## Backward Compatibility * **Legacy clients** (no `liveFormat` parameter): zero wire change. Messages never include `chunk`, `totalChunks`, or `batchId` fields. * **Unknown `liveFormat` values** (e.g., `liveFormat=foo`): fall back to legacy format silently. * **Reconnect without `liveFormat`**: falls back to legacy even if the previous connection used `chunked-v1`. The `connected`/`reconnected` message will not include `liveFormat`, allowing clients to detect the fallback. ## Reconnection with Chunking When reconnecting with a session: 1. Include `liveFormat=chunked-v1` in the URL (it is not persisted) 2. Optionally include `cursor` for precise resume (recommended) 3. The `reconnected` message confirms the format: ```json { "type": "reconnected", "clientId": "abc-123", "sessionId": "def-456", "liveFormat": "chunked-v1" } ``` Replay messages use the existing chunked replay format (with `chunk` and `totalChunks` fields). Live messages after replay use the `chunked-v1` format with `batchId`. ## Configuration The server splits live batches at **1000 items** by default. This limit is configurable server-side and may be adjusted based on monitoring. Clients should not depend on a specific chunk size. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.hydromancer.xyz/readme/websocket/live-message-chunking.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.