HIP-1056: Block Streams

[rbair23]

Abstract

This HIP introduces a new output data format for consensus nodes, called Block Streams, that replaces the existing
event streams, record streams, state files, and signature files with one single stream.
Each block within the block stream is a self-contained entity, including every event and all transactions that were
part of the block, the state changes as a result of those transactions, and a
network signature (using a BLS threshold signature scheme we call TSS) that proves the block was signed by a
threshold of the network.

By including state changes, downstream services can seamlessly maintain and verify state alongside consensus nodes. This
enhancement fosters greater transparency and trust within the Hedera network. Any downstream service can independently
rebuild and verify the state of consensus nodes at the time of the block, verified by the TSS network signature. Using
this state, they can provide additional services such as state proofs, state snapshots, and more.

With the event information within blocks, downstream services can be reconstructed and the hashgraph algorithm replayed,
permitting anyone to verify the correct execution of the consensus algorithm and removes the need for the extra event
stream. In doing so Hedera users gain comprehensive visibility into network activities through an easily consumable
format that can be delivered with low latency.

A key design criteria is for the block stream to be easily consumed by any programming language, and with minimal
complexity or dependencies. For example, state data can be utilized for basic queries without having to reconstruct a
merkle tree.

Block streams are an upgrade to the existing RecordStream V6. Block streams restructure and aggregate the multiple
record types in record streams, including EventStream, RecordStream, and Hedera state data to produce a single
unified stream of items.

The key enhancements offered by block streams include:

• Unified Data Stream: Block stream consolidates event streams, record streams, sidecars, and signature files into a
  single cohesive data stream.
• State Change Data: Each block will include state change data for the given round of consensus.
• Verifiable Proof: Each block will be independently verifiable, containing a full proof of transactions, network
  consensus and state changes.
• Comprehensive Protobuf specification By defining blocks in protobuf, the inputs, outputs, and state changes of
  consensus nodes are clearly specified, paving the way for future implementations and greater client diversity.

With the adoption of block streams, data output by Hedera consensus nodes will be consolidated into a single, verifiable
chronicle of the network, thereby strengthening the integrity and transparency of the Hedera ecosystem.

[netlify]

✅ Deploy Preview for hedera-hips ready!

Name	Link
🔨 Latest commit	5e49ad5
🔍 Latest deploy log	https://app.netlify.com/sites/hedera-hips/deploys/67e3ff093b6b270008ecdd91
😎 Deploy Preview	https://deploy-preview-1056--hedera-hips.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[jsync-swirlds]

Just a few typos and wording suggestions.

[poulok]

Got to Design for Verifiability. Will finish the rest when I can.

[derektriley]

Should this HIP at all mention/reference the behaviors around consensus node to block node communication? Will this information be covered in the block node HIP?

[poulok]

Got further, but still not done.

[jsync-swirlds]

Should this HIP at all mention/reference the behaviors around consensus node to block node communication? Will this information be covered in the block node HIP?

This is currently documented in the block node codebase, and should be included, to the extent appropriate, in the forthcoming "Block Node" HIP. That HIP will cover all of the Block Node APIs, not just the publishBlockStream API used by consensus nodes.

[Nana-EC]

Hi everyone,

I hope this message finds you in good spirits. We've definitely made progress on this PR, so great job! However, I've noticed there are still about 280 discussions that need to be resolved.

Given the importance of this PR, it's crucial that we address these concerns and move forward. I encourage everyone to revisit these discussions and extract the necessary actions to resolve them.

I'd like to extend my thanks and appreciation to all of you for your relentless effort and high-quality insights. Together, we're making our project better. Let's continue resolving these discussions so we can push towards our goal. Remember, every contribution makes a difference and sets us one step closer to the finish line.

Have a productive day!

Best regards, Michael

@jasperpotts @edwin-greene @IvanKavaldzhiev @jsync-swirlds @Nana-EC @poulok @tinker-michaelj @derektriley @tinker-michaelj @xin-hedera @steven-sheehy @jasperpotts

Addressing a lot of these in the next 1-2 weeks.
Started with #1173

All conversations appear to be resolved. If they are not, we will continue the discussion on this page after the merge