Block Node #1081

Nana-EC · 2024-11-17T20:27:26Z

Description:
Abstract
This HIP introduces and proposes the adoption of a new node type that is designed to parse the block stream, store
latest state and serve up additional network APIs such as state proofs and block retrieval that support decentralized
querying and verification of state.

A block node will primarily sit in between a consensus node and a mirror node to serve as a decentralized source of
verifiable network state and transaction information. As an emergent feature a block node will introduce a new node
economy through independent community operations. Anyone will be able to independently run a block node or block
stream parsing product that provides value added services to users e.g. aggregated data insights, data availability.

Block Nodes will have a rich API, allowing consumers to receive a subset of the stream, or the full stream. Block Nodes
can innovate on payment -- they can be commercial and have connectivity rules / rate limits / SLAs for customers.
They can require payment for serving blocks, or they can serve blocks for free.

Related issue(s):

Expands on #1055

Notes for reviewer:

Checklist

Documented (Code comments, README, etc.)
Tested (unit, integration, etc.)

netlify · 2024-11-17T20:27:41Z

✅ Deploy Preview for hedera-hips ready!

Name	Link
🔨 Latest commit	`cc62608`
🔍 Latest deploy log	https://app.netlify.com/sites/hedera-hips/deploys/67d9853ca362470008a31ca3
😎 Deploy Preview	https://deploy-preview-1081--hedera-hips.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

jsync-swirlds

About half way through the hip document so far.

HIP/hip-1081.md

jsync-swirlds

Finally reached the end of the document.

HIP/hip-1081.md

mgarbs · 2025-02-03T18:00:03Z

Hello everyone,

I hope this message finds you well!

I am reaching out just to give a gentle reminder regarding the open PR titled "Block Node". Currently, we still have 42 unresolved discussions which need your valuable input. It'd be really helpful if we could collaborate and get these discussions resolved to make further progress on this PR.

I appreciate your time, commitment, and collaboration thus far. Your thoughts and perspectives continue to be essential in making sure we're moving in the right direction.

Thank you again for your contributions to this project. Looking forward to concluding these discussions and achieving our overall goal together!

Best regards,
Michael
@jsync-swirlds
@Nana-EC

Nana-EC · 2025-02-22T04:10:55Z

Hello everyone,

I hope this message finds you well!

I am reaching out just to give a gentle reminder regarding the open PR titled "Block Node". Currently, we still have 42 unresolved discussions which need your valuable input. It'd be really helpful if we could collaborate and get these discussions resolved to make further progress on this PR.

I appreciate your time, commitment, and collaboration thus far. Your thoughts and perspectives continue to be essential in making sure we're moving in the right direction.

Thank you again for your contributions to this project. Looking forward to concluding these discussions and achieving our overall goal together!

Best regards, Michael @jsync-swirlds @Nana-EC

Thanks @mgarbs
A few commits are coming through to address the comments and fill in some of the empty sections

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

…nd service definitions Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Signed-off-by: Joseph Sinclair <121976561+jsync-swirlds@users.noreply.github.com>

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

* Added skip block message to block service `PublishStreamResponse` * Added resend block message to block service `PublishStreamResponse` * Renamed end of stream response to `end_of_stream` for clarity. * Added persitence failed response * Removed (unnecessary) item acknowledgement Signed-off-by: Joseph Sinclair <121976561+jsync-swirlds@users.noreply.github.com>

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

…section Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

…atest Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

HIP/hip-1081.md

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Nana-EC · 2025-05-22T02:01:23Z

Abstract, Motivation and Rationale and User Story sections i.e. everything above Specification have been updated in readiness for further socialization on it's way to TSC Approval.
Specification section and below will see an update to reflect updates from design and early development of Block Nodes.

assets/hip-1081/protobuf/block_stream_publish_service.proto

… for non-technical folks. Thanks mark@swirldslabs.com Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

jsync-swirlds

Not done, but sending what's there so far.

HIP/hip-1081.md

@jasperpotts

…s from @jasperpotts @rbair and @atul-hedera Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

jsync-swirlds

Still reading through, but have a few suggestions, corrections, and clarifications.

HIP/hip-1081.md

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Nana-EC · 2025-05-29T03:31:50Z

Need to circle back for another pass to update the protocol and response code descriptions with the latest from BN repo

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

georgi-l95 · 2025-05-29T08:19:32Z

HIP/hip-1081.md

+| **Verified Block**                | A verified block is a block for which a Block Proof is received and for which the TSS signature of the network ledger ID is valid.  |
+| **Full History Block Node**       | A Block Node that maintains block history from network streaming start.  |
+| **Archive Block Node**            | A Block Node that maintains block history but does not expose consumer API services.  |
+| **Rolling History Block Node**    | A Block Node that maintains blocks but prunes them after a configured period of time.  |


Light Node, maybe ? Similar to what other networks are calling nodes with similar behaviour.

Using names common in other networks, such as "Light Node", implies a lot of things not stated and not accurate about block nodes because the name is common in other networks (which don't have the hashgraph algorithm).
It's very hard to connect block nodes to node types in other networks without creating confusion, as they really don't properly match the setup used anywhere else.

I proposed Light Block Node as naming, because the other two already somewhat match the naming of nodes from other environments, like geth (for example). And it might be easier for folks from other networks, if we are consistent with the naming that they are familiar with.

However I agree with you, and in the future we might add additional template, which more closely match Light Nodes behaviour. So it's fine either way 👍

HIP/hip-1081.md

jsync-swirlds · 2025-05-29T22:55:47Z

HIP/hip-1081.md

+12. As a Block Node Operator, I want to offer filtered block streams to external clients, so that consumers only
+receive the data relevant to their use cases, reducing bandwidth and processing overhead


Might want to remove this and defer to a future "Fitlering" HIP.

I'm not sure, filtering got bumped up so we need to define the API soon but the user story may be more timely.
Let's confirm though. If indeed a different HIP it can be removed

HIP/hip-1081.md

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

HIP/hip-1081.md

* Address some review comments. * Updated sequence diagrams with latest publisher protocol expectations. * Fixed some grammar and spelling. * Added a use case for snapshot format compatibility. Signed-off-by: Joseph S <121976561+jsync-swirlds@users.noreply.github.com>

Nana-EC self-assigned this Nov 17, 2024

Nana-EC mentioned this pull request Dec 2, 2024

HIP-1056: Block Streams #1056

Merged

jsync-swirlds reviewed Dec 4, 2024

View reviewed changes

jsync-swirlds reviewed Dec 10, 2024

View reviewed changes

jsync-swirlds force-pushed the block-node branch 5 times, most recently from b358e97 to 04e11af Compare March 1, 2025 01:07

Nana-EC and others added 9 commits March 13, 2025 14:08

Add initial block node HIP draft doc

897dc40

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Expanded on rational and spec section tasks. Next up missing protos a…

518313b

…nd service definitions Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Assigning HIP 1081 and renaming file to hip-1081.md

ffc39c7

Signed-off-by: Joseph Sinclair <121976561+jsync-swirlds@users.noreply.github.com>

Cleaned up and added protobuf assets

b1c00a7

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Fix spelling errors

4ec87c0

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Add communication protocol descriptions

3b48ce1

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Cleaned up and expanded based on comments, thanks jsync-swirlds

90baa7a

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Add API response codes

bb028b3

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

jsync-swirlds force-pushed the block-node branch from 04e11af to b8f0062 Compare March 13, 2025 21:21

mgarbs and others added 8 commits March 18, 2025 10:37

Merge branch 'main' into block-node

cc62608

Update Hedera reference and cleaned up a bit

819da7b

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Cleaned up some user stories

f2b3a35

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Cleaned up layout, added layout image and updated proto

c9f1682

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Revised wordings and tries to fix tone for items above Specification …

12c7ef1

…section Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Convert png to svg

8ecf437

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Further improve wording of section above spec and update proto with l…

2fe1957

…atest Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Fix working group YAML rendering

0afa246

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

jsync-swirlds reviewed May 15, 2025

View reviewed changes

HIP/hip-1081.md Outdated Show resolved Hide resolved

jsync-swirlds reviewed May 15, 2025

View reviewed changes

HIP/hip-1081.md Outdated Show resolved Hide resolved

jsync-swirlds reviewed May 15, 2025

View reviewed changes

HIP/hip-1081.md Show resolved Hide resolved

Nana-EC added 2 commits May 21, 2025 16:08

Addressed items from jsync-swirlds

d3df7a3

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Added refinements of defintions suggested by mark@swirldslabs.com

60666d5

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

jsync-swirlds reviewed May 22, 2025

View reviewed changes

assets/hip-1081/protobuf/block_stream_publish_service.proto Outdated Show resolved Hide resolved

Nana-EC added 2 commits May 22, 2025 23:05

Significant restructuring of Specification section to make it easier…

0125c32

… for non-technical folks. Thanks mark@swirldslabs.com Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Update to latest proto

86e9d17

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

jsync-swirlds reviewed May 23, 2025

View reviewed changes

HIP/hip-1081.md Outdated Show resolved Hide resolved

HIP/hip-1081.md Outdated Show resolved Hide resolved

HIP/hip-1081.md Outdated Show resolved Hide resolved

HIP/hip-1081.md Outdated Show resolved Hide resolved

Added improvements around Abstract and Motivation based on suggestion…

72b1c9d

…s from @jasperpotts @rbair and @atul-hedera Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

jsync-swirlds reviewed May 28, 2025

View reviewed changes

Nana-EC added 2 commits May 28, 2025 19:03

Set last call

8c25e70

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Cleaned up some code and addressed some of jsync-swirlds comments

2cf4692

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Made a few improvements based on atul-hedera feedback

f6ccfc4

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

georgi-l95 reviewed May 29, 2025

View reviewed changes

jsync-swirlds reviewed May 29, 2025

View reviewed changes

HIP/hip-1081.md Outdated Show resolved Hide resolved

jsync-swirlds reviewed May 29, 2025

View reviewed changes

HIP/hip-1081.md Show resolved Hide resolved

jsync-swirlds reviewed May 29, 2025

View reviewed changes

HIP/hip-1081.md Outdated Show resolved Hide resolved

jsync-swirlds reviewed May 29, 2025

View reviewed changes

HIP/hip-1081.md Show resolved Hide resolved

Nana-EC added 5 commits May 30, 2025 00:49

another pass and address feedback form jsync and-swirlds atul-hedera

2aaab26

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Added some more user stories based on suggestion fromatul-hedera

1708f00

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Further touch ups based on questions from atul-hedera

28fa6ec

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Reorganized for easier reading flow

e951f34

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Cleaned up grammar a bit more

125e67f

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

jsync-swirlds reviewed Jun 2, 2025

View reviewed changes

HIP/hip-1081.md Outdated Show resolved Hide resolved

HIP/hip-1081.md Outdated Show resolved Hide resolved

HIP/hip-1081.md Outdated Show resolved Hide resolved

		12. As a Block Node Operator, I want to offer filtered block streams to external clients, so that consumers only
		receive the data relevant to their use cases, reducing bandwidth and processing overhead

Block Node #1081

Are you sure you want to change the base?

Block Node #1081

Uh oh!

Conversation

Nana-EC commented Nov 17, 2024

Uh oh!

netlify bot commented Nov 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for hedera-hips ready!

Uh oh!

jsync-swirlds left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsync-swirlds left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mgarbs commented Feb 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Nana-EC commented Feb 22, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Nana-EC commented May 22, 2025

Uh oh!

Uh oh!

jsync-swirlds left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jsync-swirlds left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Nana-EC commented May 29, 2025

Uh oh!

georgi-l95 May 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsync-swirlds May 29, 2025

Choose a reason for hiding this comment

Uh oh!

georgi-l95 May 29, 2025

Choose a reason for hiding this comment

Uh oh!

netlify bot commented Nov 17, 2024 •

edited

Loading

mgarbs commented Feb 3, 2025 •

edited

Loading

georgi-l95 May 29, 2025 •

edited

Loading