Add communication protocol descriptions

Signed-off-by: Nana Essilfie-Conduah <nana@swirldslabs.com>

Author: Nana-EC

HIP/hip-1081.md

@@ -64,9 +64,9 @@ performance streaming, use of protobufs (continued commitment to simple and clea
 flexibility with microservice pluggable options.
 
 As such consumers will utilize gRPC clients to communicate.
-This matches the communication protocol with the CN but differs from other web3 nodes which often utilize JSON RPC APIs over HTTP.
-In the future it may be valuable to explore and add additional protocol such as WS and HTTP (REST API, graphQL) based on community needs.
-
+This matches the communication protocol with the CN but differs from other web3 nodes which often utilize JSON RPC APIs
+over HTTP. In the future it may be valuable to explore and add additional protocol such as WS and HTTP (REST API,
+graphQL) based on community needs.
 
 
 <aside>
@@ -94,6 +94,16 @@ The block node will support semantic version values and will make it's current v
 
 ## User stories
 
+### Terms
+- Block Node: A software system intended to store and process a Block Stream. The API for a Block Node is defined in
+    HIP 1056, among others.
+- Block Number: A monotonically increasing number assigned by consensus to each block produced by the network.
+- Publisher: An entity publishing blocks to a Block Node via the `publishBlockStream` API. This is typically a
+    Consensus Node or another Block Node.
+- Subscriber: An entity that subscribes to a verified or unverified Block Stream from a Block Node.
+- Verified Block: A verified block is a block for which a Block Proof is received and for which the TSS signature of
+    the network ledger ID is valid.
+
 ### Personas
 
 - Council consensus node operators
@@ -131,8 +141,8 @@ minimum possible latency.
 possible latency.
 15. As a custom dApp I want to receive the block stream starting at any historical block and continue forward from that
 point indefinitely.
-16. As a custom dApp I want an API to request a snapshot of the state, as of an arbitrary historical block, from a block
-node.
+16. As a custom dApp I want an API to request a snapshot of the state, as of an arbitrary historical block, from a
+block node.
 17. As a custom dApp I want an API to request a state proof for any item in state, as of an arbitrary historical block,
 from a block node.
 18. As a custom dApp I want an API to request a state proof for any item in state, as of the most recent block, from a
@@ -150,8 +160,8 @@ determine, in software, the best choice(s) of block node(s) to use.
 
 ## Specification
 
-The block node at its simplest will be another node server that offers APIs to consumers to obtain needed network information and insights.
-Clean extendable schemas and easy to consume APIs are important to reach this goal
+The block node at its simplest will be another node server that offers APIs to consumers to obtain needed network
+information and insights. Clean extendable schemas and easy to consume APIs are important to reach this goal
 
 ### Block Stream Schema
 
@@ -168,8 +178,10 @@ in the [hedera-protobuf repository](https://github.com/hashgraph/hedera-protobuf
 </aside>
 
 ### Services
-To achieve the above the block node will offer multiple pluggable modular services which can be enabled by node operators.
-Note: protobuf summaries are noted in this HIP doc. The full protobuf specification can be found under
+To achieve the above the block node will offer multiple pluggable modular services which can be enabled by node
+operators.
+
+> Note: protobuf summaries are noted in this HIP doc. The full protobuf specification can be found under
 [block node protobuf](./../assets/hip-1081/protobuf)
 
 #### Block Node Service
@@ -192,23 +204,11 @@ service BlockNodeService {
 
 #### Block Stream Service
 
-One of the primary goals of the Block Node will be to expose a public streaming endpoint of block information obtained
-from Consensus Node outputs.
+One of the primary goals of the Block Node will be to expose a public streaming endpoint of block information
+obtained from Consensus Node outputs.
 
-There will be 2 types of stream services offered
-
-- Unverified block stream for low latency
-- Verified block stream for high confidence
-
-Depending on a user, consuming clients can optimize for requesting Block Items as fast as possible for their own
-optional verification or rely on confirmed blocks as confirmed by a Block Node following the Block Proof algorithm.
-
-Both streams may also offer filtering capabilities based on user requests and Block Node functionality support.
-Filtering support is expanded upon in the Block Stream HIP 1056 and is charectorized buy the replacement of block items
-with hash values whiles maintian the ability to carry out block and state proofs on the block information.
-
-Initially filtering will be at the Block Item level (e.g. a Mirror Node may want only Transaction input and outputs)
-but in the future block nodes could stream blockItems out based on the matching of state or transaction matches.
+The block node will support 2 modes of data flow connections from external clients: publishBlockStream (push) &
+subscribeBlockStream (pull)
 
 ```protobuf
 /**
@@ -227,8 +227,100 @@ service BlockStreamService {
 }
 ```
 
-Notably, a block node may consume a block stream from multiple sources e.g. multiple CNs.
-This approach can be used to increase the reliability and availability of network data
+#### publishBlockStream: Push mode
+
+For multiple reasons including security and ease of data rate controls the consensus node will push block stream data
+to a block node. As such a block node must be ready and available to accept connections that push the block stream to
+the node as blocks are agreed upon by the network. In this mode the Consensus Node or other block stream clients act as
+the Producer feeding the block node data.
+
+A summarized handshake of communication between a publisher and BN is as follows
+- A publisher will send a Block Header item on connection
+- A block node will accept the stream if it is the next block or the node has no cache of previous blocks. 
+    - If the block is less than the last known block, the block node will respond with a `DuplicateBlock` response
+    including details of the last known block. In this case the Block Node itself may explore a backfill path from
+    another block node to resolve gaps in blocks where applicable.
+    - If the block is greater than the last known block, the block node will respond with a `Behind` response including
+    details of the last known block. A publisher must send either an earlier block or an `EndOfStream` in response. If
+    an an `EndOfStream` is sent by the publisher it must include its earliest known block to signal to the Block Node
+    how large of a catchup range it must resolve. In this case the Block Node itself may explore a backfill path from
+    another block node to resolve gaps in blocks where applicable.
+- During streaming errors may occur from either side.
+    - If a Publisher detects an error it must send an `EndStream` response in the next BlockItem. A Block Node will
+    drop unverified blocks from that Publisher.
+    - If a Block Node detects an error it will send an `EndStream` response in the next BlockItem to all publishers.
+    A publisher must respond with a a replay of the given block in sessions and will apply exponential
+    backoff if the error repeats.
+
+<aside>
+🚨 **Open Task:** Add sequence diagram to illustrate handshake
+</aside>
+
+<aside>
+🚨 **Open Task:** Add sequence diagram to illustrate error handling modes
+</aside>
+
+Additional response codes and their details are noted below
+
+| Response code         | Description   |
+| --------------------- | ------------- |
+| STREAM_END_UNKNOWN    | This status indicates the server software failed to set a status, and SHALL be considered
+a software defect. |
+
+<aside>
+🚨 **Open Task:** Add all known response codes
+</aside>
+
+> When the producer is a consensus node it is important to expose error cases that suggest a blocks node is falling
+behind or failing to store blocks when expected.
+
+> Notably, a block node may consume a block stream from multiple sources e.g. multiple CNs.
+This approach can be used to increase the reliability and availability of network data.
+
+#### subscribeBlockStream: Pull mode
+In addition to ingesting the block stream, teh block will also support the export of the block stream data in a similar
+manner to the consensus node. 
+
+There will be 2 types of stream services offered
+
+- Unverified block stream for low latency
+- Verified block stream for high confidence
+
+Depending on a user, consuming clients can optimize for requesting Block Items as fast as possible for their own
+optional verification or rely on confirmed blocks as confirmed by a Block Node following the Block Proof algorithm.
+
+Both streams may also offer filtering capabilities based on user requests and Block Node functionality support.
+Filtering support is expanded upon in the Block Stream HIP 1056 and is characterized by the replacement of block items
+with hash values whiles maintain the ability to carry out block and state proofs on the block information.
+
+Initially filtering will be at the Block Item level (e.g. a Mirror Node may want only Transaction input and outputs)
+but in the future block nodes could stream blockItems out based on the matching of state or transaction matches.
+
+An overview of communication between a Block Node and a subscriber is as follows
+- A subscriber will send a desired block range and desired for verified or unverified streaming on connection
+- A block node will accept the stream if the desired block range is supported. 
+    - If the block is less than the last available block, the block node will respond with a
+    `READ_STREAM_INVALID_START_BLOCK_NUMBER` response.
+    - If the block is greater than the last available block, the block node will respond with a
+    `READ_STREAM_INVALID_END_BLOCK_NUMBER` response.
+- During streaming errors may occur.
+    - If a Block Node detects an error it will send an `EndStream` response in the next BlockItem to all subscribers. 
+    A subscriber may choose to retry or connect to a different block node.
+
+<aside>
+🚨 **Open Task:** Complete communication overview
+</aside>
+
+Additional response codes and their details are noted below
+
+| Response code         | Description   |
+| --------------------- | ------------- |
+| READ_STREAM_UNKNOWN   | This status indicates the server software failed to set a status and SHALL be considered a
+software defect. |
+
+<aside>
+🚨 **Open Task:** Add all known response codes
+</aside>
 
 #### Block Service
 
@@ -248,6 +340,10 @@ service BlockAccessService {
 }
 ```
 
+<aside>
+🚨 **Open Task:** Add all known response codes for API
+</aside>
+
 #### State Service
 
 Live state, snapshot, changes and single entity state
@@ -267,13 +363,19 @@ service StateService {
 
 <aside>
 🚨 **Open Task:** Flesh out 3 different state related query endpoints
-1. State snapshot - covering latest state and historical state. All cases represent verified state at the end of a block.
-2. State changes e.g. what state changes occurred in block x, this is a subset of the block query and avoid the transmission of extra block information beyond state
+1. State snapshot - covering latest state and historical state. All cases represent verified state at the end of a
+    block.
+2. State changes e.g. what state changes occurred in block x, this is a subset of the block query and avoid the
+    transmission of extra block information beyond state
   - Add new endpoint to proto
 3. Single entity state query e.g. return account 0.0.x 
   - Add new endpoint to proto
 </aside>
 
+<aside>
+🚨 **Open Task:** Add all known response codes for API
+</aside>
+
 #### Reconnect Service
 
 Today the network consensus node act as teachers to each other when a new node comes online and needs to ge the latest
@@ -294,8 +396,8 @@ consensus node or block node coming online and looking for latest network detail
 #### Proof Service
 
 Consensus nodes provided proof of state APIs has long been desired, however, this would put additional work on the
-consensus node that could be better provided elsewhere. By maintaining live state and streaming block stream information
-a block node can finally fill this gap and provide users with proofs including that of state.
+consensus node that could be better provided elsewhere. By maintaining live state and streaming block stream
+information a block node can finally fill this gap and provide users with proofs including that of state.
 
 <aside>
 🚨 **Open Task:** Flesh out response components
@@ -330,9 +432,9 @@ Block nodes will perform significant work by consuming the block stream, verifyi
 multiple API services to further users. To block node operators it is thus important to offer capabilities to cover
 the cost of work and encourage a vibrant economy.
 
-To achieve this the Block Node will adopt a charge card comparable model with crypto transfers of hbar required prior to the
-consumption of APIs. The block node will initially manage an hbar ledger on node to track the remaining hbar balance
-that an account ID holds with the block node.
+To achieve this the Block Node will adopt a charge card comparable model with crypto transfers of hbar required prior
+to the consumption of APIs. The block node will initially manage an hbar ledger on node to track the remaining hbar
+balance that an account ID holds with the block node.
 
 To achieve this it is required that a block node have it's own account Id that users can transfer hbar to.
 
@@ -431,4 +533,5 @@ A collections of URLs used as references through the HIP.
 
 ## Copyright/license
 
-This document is licensed under the Apache License, Version 2.0 -- see [LICENSE](../LICENSE) or (https://www.apache.org/licenses/LICENSE-2.0)
+This document is licensed under the Apache License, Version 2.0 -- see [LICENSE](../LICENSE) or
+(https://www.apache.org/licenses/LICENSE-2.0)