HIP-1193 - Records to block streams cutover #1193
Conversation
Mark-Swirlds
changed the title
|
Updated HIP w/ PR number and proper DCO. |
This HIP defines the requirements and implementation details for transitioning the Hedera network from record and event streams to block streams. This transition is a critical step toward enhancing blockchain compatibility, improving data integrity, and enabling future network optimizations such as state proofs. The proposal outlines the coordinated changes required across consensus nodes, mirror nodes, and supporting infrastructure to ensure a clean cutover without service disruption. It specifies new storage paths, file formats, and mechanisms to handle the transition state, including a marker file approach to indicate the final record stream entry before blocks begin. Signed-off-by: Mark Blackman <mark@hashgraph.com>
Mark-Swirlds
force-pushed
the
hip_blockstreams_cutover
branch
from
285854b to
b6a950e
Compare
| The new block stream files will follow a revised path structure that includes network, shard, realm, and node ID components: | ||
|
|
||
| ``` | ||
| block/{network}-{YYYY-MM-DDTHH:mm}/{realm}/{shard}/{nodeID}/0000000000000000000000000000000000000.blk.gz |
There was a problem hiding this comment.
I suggest the following:
-
Include some information in the file name as well as using a subfolder structure to store the files. This will make it easier to support multiple folder structures in the future, including a completely flat structure, while maintaining coherence and avoiding collisions.
-
Use an ID (i.e. chainID = 295) instead of a label (i.e. 'mainnet') to identify the network.
-
Use the block date further down the folder structure.
-
Move the node ID as the first dynamic folder
-
Remove leading zeros in the block number.
In particular, I propose the following structure:
blocks/{node ID}/{network ID}/{realm ID}/{shard ID}/YYYYMMDD/{network ID}_{realm ID}_{shard ID}_{YYYYMMDD}_{block number}.blk.zstd
For example:
blocks/10/295/0/0/20250901/295_0_0_20250901_83406349.blk.zstd
There was a problem hiding this comment.
It should be noted here that this file structure is still very much temporary.
The long-term approach is that Block Nodes will store blocks, and access via buckets will end. This temporary structure is intended to require minimal changes to existing clients during this interim period (so that clients can focus more resources on switching fully to block nodes as the data source).
Block Nodes (at least the Hiero implementation) use a more detailed and complex structure for local storage that makes automated access much easier; but that's an internal detail. Users of block streams will use the API to request individual blocks or a stream of blocks (with several associated options).
There was a problem hiding this comment.
One small suggestion: we can remove the extra characters in the date/time format to make it a little shorter and more obviously intended for automation:
| block/{network}-{YYYY-MM-DDTHH:mm}/{realm}/{shard}/{nodeID}/0000000000000000000000000000000000000.blk.gz | |
| block/{network}-{YYYYMMDD_HHmm}/{realm}/{shard}/{nodeID}/0000000000000000000000000000000000000.blk.gz |
| - The shard and realm values will be included in the directory structure | ||
| - The nodeID will identify the consensus node that produced the block | ||
| - Block files will use a 36-digit zero-padded block number format | ||
| - Files will use gzip compression and the `.blk.gz` extension |
There was a problem hiding this comment.
We mentioned above we are going to use zstd (.blk.zstd), rather than gzip.
There was a problem hiding this comment.
Good catch.
The bucket files will still be gzip because that's what the existing uploader and consensus node support.
Block Nodes may use ZStandard internally, but that is an internal implementation detail that might change arbitrarily. The use of ZStandard should not be included in this HIP.
| Consensus nodes will implement the new path structure, for example: | ||
|
|
||
| ``` | ||
| block/mainnet/0/0/10/000000000000000000000000000083406349.blk.gz |
There was a problem hiding this comment.
I suggest changes to this here.
As a side note, in case we don't want to proceed with the suggested changes, this example is missing the date/time folder.
There was a problem hiding this comment.
The date and time are optional and used for resettable networks (which mainnet is not).
| - The network value will identify the network (mainnet, testnet, etc.) | ||
| - The shard and realm values will be included in the directory structure | ||
| - The nodeID will identify the consensus node that produced the block | ||
| - Block files will use a 36-digit zero-padded block number format |
There was a problem hiding this comment.
What are the advantages of a 36-digit zero-padded number instead of the simple plain number?
There was a problem hiding this comment.
The 36-digit length is an implementation choice in consensus node (it cannot be more than 19 non-zero digits, in actuality, and should be 9 or less).
The reason to choose a fixed-length value is that sorting works better and it is much easier for the software to pre-calculate reliably so that we do not need to list directories (which is abnormally expensive in cloud buckets).
|
|
||
| - Stop producing record and event streams at the designated cutover point | ||
| - Generate marker files to signal the end of record streams | ||
| - Begin producing block streams with proper running hash continuity |
There was a problem hiding this comment.
As it's not technically a running hash on both sides (it's more complex than that) perhaps describe a from-to pair?
| - Begin producing block streams with proper running hash continuity | |
| - Begin producing block streams with proper continuity from record stream hash to block hash. |
| 7. As a consumer of mirror node data and associated metrics, I want the transition to be seamless with no impact on how I consume mirror node data. | ||
|
|
||
| ## Specification | ||
| i### 1\. Cutover Point Definition |
There was a problem hiding this comment.
typo
| i### 1\. Cutover Point Definition | |
| ### 1\. Cutover Point Definition |
| - The first block will contain zero transactions to ensure proper formatting without impacting network usage | ||
| - Block continuity will be maintained by carrying forward: | ||
| - The correct block number (incremented by one from the last record) | ||
| - The running hash values \- cryptographic continuity of the Hedera blockchain. |
There was a problem hiding this comment.
Hash description might be a little clearer as follows:
| - The running hash values \- cryptographic continuity of the Hedera blockchain. | |
| - The last running hash value — cryptographic continuity of the Hedera blockchain. |
|
|
||
| Consensus nodes will: | ||
|
|
||
| - Begin construction of the first block with zero transactions |
There was a problem hiding this comment.
Small suggestion
| - Begin construction of the first block with zero transactions | |
| - Construct the first block with zero transactions |
| - Apply the correct running hash and block number from the final record | ||
| - Upload to new block path structure for uploaders | ||
|
|
||
| NOTE: It is required all event stream format changes are completed prior to the cutover |
There was a problem hiding this comment.
Clarity suggestion
| NOTE: It is required all event stream format changes are completed prior to the cutover | |
| NOTE: All event stream format changes must be completed and active prior to the cutover |
| The new block stream files will follow a revised path structure that includes network, shard, realm, and node ID components: | ||
|
|
||
| ``` | ||
| block/{network}-{YYYY-MM-DDTHH:mm}/{realm}/{shard}/{nodeID}/0000000000000000000000000000000000000.blk.gz |
There was a problem hiding this comment.
One small suggestion: we can remove the extra characters in the date/time format to make it a little shorter and more obviously intended for automation:
| block/{network}-{YYYY-MM-DDTHH:mm}/{realm}/{shard}/{nodeID}/0000000000000000000000000000000000000.blk.gz | |
| block/{network}-{YYYYMMDD_HHmm}/{realm}/{shard}/{nodeID}/0000000000000000000000000000000000000.blk.gz |
| - Block number-based naming for sequential processing | ||
| - ISO timestamp format of last network reset (optional field for resettable networks) | ||
|
|
||
| Block files will use the `.blk.zstd` extension and Zstandard (Zstd) compression to optimize storage while maintaining reasonable processing performance. |
There was a problem hiding this comment.
Shouldn't refer to ZStandard here, as the transitional files will still be gzip.
| Block files will use the `.blk.zstd` extension and Zstandard (Zstd) compression to optimize storage while maintaining reasonable processing performance. | |
| Block files will use the `.blk.gz` extension and GZip compression to maintain better compatibility with current record stream processing. |
| - Support for sharding | ||
| - Node identification to identify source of block | ||
| - Block number-based naming for sequential processing | ||
| - ISO timestamp format of last network reset (optional field for resettable networks) |
There was a problem hiding this comment.
Minor clarification
| - ISO timestamp format of last network reset (optional field for resettable networks) | |
| - ISO 8601 date and time format values of last network reset (optional field for resettable networks) |
| Consensus nodes will implement the new path structure, for example: | ||
|
|
||
| ``` | ||
| block/mainnet/0/0/10/000000000000000000000000000083406349.blk.gz |
There was a problem hiding this comment.
The date and time are optional and used for resettable networks (which mainnet is not).
|
|
||
| Timestamp to Block Number Mapping: A solution is needed for mirror nodes to map between timestamps and block numbers, particularly for startup and historical data access. | ||
| ## References | ||
| A collection of URLs used as references throughout the HIP. |
There was a problem hiding this comment.
Might want to remove this boilerplate text.
Neurone
changed the title
|
|
||
| 2. As a mirror node operator, I want to understand the new block file format and paths so that I can adjust my infrastructure to process block streams efficiently. | ||
|
|
||
| 3. As a developer building on Hedera, I want to understand how the transition affects data availability and processing so that my applications continue to function correctly. |
There was a problem hiding this comment.
does this user story take into account developers or projects that currently ingest data via S3 and record files, rather than using the mirror nodes?
If so, they will likely need to rework their architecture and it would be helpful to provide guidance or migration resources to support the transition.
There was a problem hiding this comment.
Thank you for the comment Nadine. Happy to adjust the user story and document the impact appropriately. Do you know of devs or projects directly ingesting the files? If yes can you share their use case and why they are using records directly.
Description:
Related issue(s):
Fixes #
Notes for reviewer:
Checklist