Atomic transaction chains

[mgarbs]

Opening up the discussion around atomic transaction chains on Hedera.

From @se7enarianelabs:

"Example:
TokenUnfreezeTransaction -> TransferTransaction -> TokenFreezeTransaction
The whole transaction chain should be executed atomically.
This would allow the creation of more complex flows that must occur in sequence, without using smart contracts, and listening to mirror nodes."

[paulmadsenhbar]

Would be useful for enabling soul bound tokens, eg chain together a CryptoTransfer & TokenFreeze txs.

[rocketmay]

Some consideration will need to be made about ensuring that transaction chains cannot fail halfway.

Ideally you would not start the chain without checking multiple factors to predict whether or not the chain will succeed. I'm not sure how reasonable this is but such functionality seems critical.

Transactions cannot be reversed in many cases, and a chain which fails part way could be catastrophic if there is a lot of value being transferred in multiple steps.

[se7enarianelabs]

I think the transaction chain should support all ACID properties, so If one transaction fails the whole chain should be rolled back.

[exploreriii]

I think this is essential! Being able to group multiple transactions together will make it easier to interact with the network.
It would be great if I could:
i) buy and associate NFTs in one go
ii) Post to 4orum in one go
iii) create accounts with my email in one go

[se7enarianelabs]

---

hip: <HIP number (this is determined by the HIP editor)>
title: Atomic transactions chain
author: Piotr Swierzy piotr.swierzy@arianelabs.com aka @se7enarianelabs
type: Standards Track
category: Service
needs-council-approval: Yes
status: Idea
created: 2022-07-25
discussions-to: #531

Abstract

This HIP defines a mechanism to execute atomic transaction chains such that a series of transactions(HAPI calls) depending on each other can be rolled into one transaction that passes the ACID test (atomicity, consistency, isolation, and durability)

Motivation

The existing implementation of transactions in the Hedera network does not allow multiple different HAPI transactions to be called in one single network transaction that would have all the ACID properties.
This makes it impossible to create more complicated flows without using smart contracts (which do not support all the HAPI transactions at this point) and listening to the mirror node to check the status of the previous transaction.
This way we can also achieve an abstraction away from smart contracts

Rationale

Atomic transactions chain is from the protobuf point of view just a list of SignedTransactions that's, why we based our new protobuf message on it,
we only have to change the bodyBytes field to be repeated, that way we can store there multiple transaction bodies,
SignatureMap sigMap can stay unchanged because it's defined as repeated SignaturePair sigPair.

User stories

As a Hedera token service user, I want to be able to unfreeze an account, send an NFT, and freeze it again in one ACID transaction, that way I can achieve an account-bound NFT(nontransferable NFT) collection, without using the hedera smart contract service.
We can use nontransferable NFTs e.g. as someone’s achievements, the creation of digital references, etc.

As a Hedera token service user, I want to be able to unfreeze an account, send an FT and freeze it again in one ACID transaction, that way I can achieve an account-bound FT(nontransferable FT) collection, without using the hedera smart contract service.
We can use nontransferable FT e.g. as game points, reputation points, etc.

As a Hedera token service user, I want to be able to wipe a token, mint a token, and transfer a token in one ACID transaction.
That way a backend developer could create complicated flows without the need to use smart contracts and the need to handle partially successful flows in the backend,
e.g. the wipe transaction was successful but the mint transaction was not.

As a Hedera service user, I want to be able to create atomic transaction chains, that use multiple Hedera services e. g.
I want to be able to wipe a token, mint a token, transfer a token, and submit a topic message in one ACID transaction.
That way a backend developer could create complicated flows without the need to use smart contracts and the need to handle partially successful flows in the backend,
e.g. the wipe transaction was successful but the mint transaction was not.

Specification

We could create a new protobuf message based on SignedTransaction message:

message AtomicTransactionsChainTransaction {
    /**
    * TransactionBodies serialized into bytes, which must be signed
    */
    repeated bytes bodyBytes = 1;

    /**
     * The signatures on the body with the new format, to authorize the transactions
     */
    SignatureMap sigMap = 2;
}

Backwards Compatibility

Security Implications

How to Teach This

Reference Implementation

Rejected Ideas

We rejected the idea to add the support of conditional branching to the atomic transaction chains,
because we don't think it's useful compared to the increase of complications in implementing the HIP.

Open Issues

References

Copyright/license

This document is licensed under the Apache License, Version 2.0 -- see LICENSE or (https://www.apache.org/licenses/LICENSE-2.0)

[mgarbs]

Hedera has many unique mechanics that could create many opportunities if they could be called up one after another in a single transaction that would have all the ACID properties.

Looking at Abstracts in other hips and trying to mirror that..., what do you think about the following for an Abstract?

"This HIP defines a mechanism to execute atomic transaction chains such that a series of transactions depending on each other can be rolled into one transaction that passes the ACID test (atomicity, consistency, isolation, and durability)"

Something like this

[se7enarianelabs]

Yes you're right, that's better

[rbair23]

I like the idea behind this. I have been thinking of something similar, which I was calling "BatchTransaction". All transactions are atomic, and in the name I wanted to capture the notion of a batch of transactions all being submitted together. Not sure if I like "Batch" better than "Chain", but "BatchTransaction" is short and keeps to our convention of ending transaction message names with Transaction. What do you think?

[se7enarianelabs]

Yes, I agree that BatchTransaction is better because it is:

short and keeps to our convention of ending transaction message names with Transaction

but maybe we could call it something like AtomicBatchTransaction, to underline the atomic property?

[rbair23]

I don't think it needs Atomic in the name since all transactions are Atomic. It seems redundant and calls into question whether other transactions are atomic. What do you think?

[se7enarianelabs]

Yes, I agree

[rbair23]

FWIW I'm reconsidering it today -- in a conversation I heard it stated that "BatchTransaction" was confusing because it wasn't clearly if each transaction within it was all-or-nothing. So maybe AtomicBatchTransaction is better, because it clearly says everything within the batch is done atomically.

[rbair23]

Another challenge to think through is there is a 6K limit to the size of any gRPC request. This limit can be raised, but only with careful considering and not very far (i.e. it will never be in the megabytes). There are a few reasons for this.

First, we need to protect against a DOS attack. If a legitimate gRPC request can be 1MB, then a bad gRPC request can also be 1MB and we won't know the difference between the two until we have gone through the effort of decoding the protobuf. It might even be valid protobuf up until the very last byte and then turns out to be bogus. An attacker can attempt a DDOS attack by sending many smaller requests, but they will be interleaved with legitimate requests, whereas if they can send fewer massive requests, then they can squeeze out legitimate requests more easily.

Second, these transactions are batched up into events and gossiped. Big events mean higher latency in gossip, and if we support transactions (and by extension events) that are too big, then our finality and e2e latency times may go past our SLA.

Keys make up a significant portion of the size of any given transaction. If each tx in the batch has its own sigmap, and if the sigmap is the same for many transactions, then we will hit the size limit very quickly -- maybe only a few transactions per batch. Also the payer is repeated, and each "inner" transaction has other metadata that is repeated. Is it all necessary?

Which brings me to another point, we need to think about what the maximum number of transactions per batch will be. This is a difficult number to come to objectively, unless we work backwards from whatever our tx size limit should be and if we want each tx in the batch to be independent or not.

[se7enarianelabs]

Also the payer is repeated, and each "inner" transaction has other metadata that is repeated. Is it all necessary?

I think the end specification should be developed by the engineering team, we just should make it as flexible as possible to not limit the community.

Which brings me to another point, we need to think about what the maximum number of transactions per batch will be. This is a difficult number to come to objectively, unless we work backwards from whatever our tx size limit should be and if we want each tx in the batch to be independent or not.

I don't think we should set a fixed maximum number of transactions, I believe the transaction size limit(gRPC) is enough.

[bguiz]

I agree with this. I think that the protobuf should be serialised with num transactions and num signatures as its first couple of fields, so as to enable failing fast during deserialisation (in the scenario where these numbers are above a certain limit).

I'd suggest that these numbers are specified in a configuration file (rather than hardcoded), in such a manner that changing them would constitute a consensus change, and therefore require a HIP to do so. And this current HIP should specify the initial values for these numbers.

Which brings me to another point, we need to think about what the maximum number of transactions per batch will be.

[rbair23]

There is another challenge which was mentioned by @rocketmay, which is thinking through the failure scenario. We have three components of the fees: node, network, and service. The node fee is the fee collected by the node that processed your tx request and submitted it for consensus. The network fee is the fee paid to all nodes for having done the work of consensus. And the service fee is the fee paid to all nodes for handling your transaction. To prevent DOS attacks where an attacker causes the network to do work that it is not compensated for, we need to include in the HIP language for how to calculate the service fee in the event that the batch of transactions fail. And they can fail for several reasons.

The transactions in the batch may fail because they are internally inconsistent. They may have the wrong signatures, or invalid protobuf bytes, or other problems with their own structure or internal state.

They may also fail because at the time we get to handling those transactions the throttle limits for one or more of them has been exceeded. If one or more of the inner transactions are smart contract calls, they may fail by running out of gas, which we cannot determine ahead of time.

They may also fail because by the time they execute, the working state on the node make them invalid. For example, at the time the batch transaction was submitted some account had plenty of hbars in it to perform the transfers and work denoted in the batch transactions, but by the time we actually get to processing those transactions, the account no longer has enough hbars.

In all these cases, the nodes are going to do a lot of work only to find out that partway through the list of transactions it encounters a failure. In this case, we need to determine how to charge the user. Do we charge them full price for transactions that we didn't actually commit to state, or do we charge them a smaller price for the prep-work we had to do? For example, if a token mint is $1, and the batch has 20 token mints followed by a crypto transfer, and all the token mints are done (but not committed to state) and the crypto transfer fails, then do we charge $20 for those token mints that we're going to rollback or do we charge some smaller nominal fee for having prepared to do the token mints, but not having actually done them? I think we should do the latter, because maybe the crypto transfer failed due to a throttle or something else completely outside the hands of the person who submitted the batch transaction and they'd be irate if they lost $20 because a throttle was exceeded. Yet we have to charge something equivalent to the work the nodes actually did or it is an attack vector.

[se7enarianelabs]

How does this work in a smart contract, in a smart contract we can have the same scenario that the network did a lot of work but then something failed and everything is rolled-back?
Maybe we could handle the BatchTransactions failed cost the same way? That we include some sort of gas for the prepwork?

[rbair23]

I was talking with Atul the other day and he convinced me that each "internal" transaction within the batch should be fully signed individually. This is critical because if we used the keys on the batch itself to authorize everything within the batch, then somebody might be able to fool me into signing a batch for the sake of one of the inner transactions and inadvertently use my key on a second batch. So each inner transaction has to be signed individually.

So let each inner transaction be complete. To prevent replay (where I have a batch and you have a batch and you submit an inner transaction, and I find it on the mirror node and include it in my batch and send it again), we need to have a tie from the inner transaction back to the batch it is part of. So we should update the transactionID specification to support a reference to the outer batch node, which will tie those transactions strongly to the batch.

Now, in terms of paying for things, we have a difference from smart contracts. With smart contracts, the contract call transaction pays for all gas, including for things that are not successfully completed. I think we have two options:

• The payer of each individual transaction pays for it, even if the change is rolled back due to a failure of a subsequent transaction.
• The payer of the batch pays for everything, we don't actually require or use the payer for each inner transaction.

[rbair23]

Oh, I forgot. I think we should always have each inner transaction have its own payer, and if I have transaction A, B, and C, and A succeeds and B fails, then:

• Payer of A will be charged (normal success charge)
• Payer of B will be charged (normal failure charge)
• Payer of C will not be charged
• Batch payer will pay node + network fee

This would allow a few different business models. Maybe the guy paying for the Batch is different from the other transactions, and each of those inner transactions take on the risk if the entire batch doesn't succeed. Or, the batch payer could also be the payer on each inner transaction, and the batch payer takes on the risk. It becomes a business decision, while at a technical level we leave it open to users to decide how to do it.

[rbair23]

Another thing we should be really explicit about in the HIP is what we are not going to do, and decide up front how this feature would grow and take shape over time. Right now, it is proposed as a simple list of transactions (yay!). But somebody, somewhere, is going to say "I'd like to have this set of transactions succeed together or fail together, and this other set to execute only if the first failed for some reason". In other words, somebody is going to propose arrays of transactions with different semantics, like "run the first batch of transactions that actually succeeds and then stop processing the other batches". We should be clear that this feature is limited in nature, it will never be as flexible as a smart contract, and shouldn't be. It is a simple mechanism to enable some additional use cases but will never be capable of doing everything a smart contract can do.

[se7enarianelabs]

Yes, I agree with you, the end goal of this feature should be a simple list of transactions. This HIP is not created to replace smart contracts in more complex flows.

[bguiz]

This is an interesting use case, and I agree that it should be for a separate HIP, as it would be significantly more complex.

Perhaps BatchRecursiveTransaction where the protobuf would be a list of either BatchRecursiveTransaction or BatchTransaction.

This should specify a total number of transactions as well as a max depth of transactions in order to enable fail fast during deserialisation (and processing), as a counter-DOS measure ... similar to my comment here

But somebody, somewhere, is going to say "I'd like to have this set of transactions succeed together or fail together, and this other set to execute only if the first failed for some reason".

[rbair23]

I broke all that feedback into separate comments so it would be easy to have threads on each individual area. Given all the different things we need to think through, we should also ask ourselves whether this is a feature we want to add, or whether what we really need are more system contracts that can be called through a smart contract and let that be the mechanism for this kind of thing. It is far more powerful, so it will scale from simple use cases to very complex ones, whereas this feature will not scale in such a way. To be honest, I'm not sure.

[se7enarianelabs]

Smart contracts are more powerful and trustless, but they are also far more challenging to develop and maintain(update/change) than SDK HAPI calls.
I see in Hedera basically two camps:

1. The smart contract enthusiasts, just want a precompile for every HAPI call.
2. The "native" camp, they do not want to use smart contracts to develop their applications, they use the SDK, etc.

In my opinion, AtomicBatchTransaction, would be a great addition to the latter camp, That way we can just embed them into the web2 backend and create complex flows that way, and if we do not need the trustless property, then why should we have to take the "burden" of smart contract, and have some hybrid of web2 backend codebase and smart contracts?

[rbair23]

I agree, I like AtomicBatchTransaction in concept. As long as we keep it a simple atomic list of transactions, work out charging and other little details, it seems totally reasonable and opens up some other use cases.

[bguiz]

I'm aware of multicall, but that would only work within HSCS (and perhaps any other hedera services that expose their interface into HSCS via a system contract such as HTS).

That would be a more generic solution than the one in this HIP-551.
... but it would only cover actions that are currently possible when the EthereumTransaction HAPI is the vehicle.

Say you wanna interact with HFS directly, or interact with parts of HTS that aren't exposed via system contracts,
then you would stillneed this HIP-551.

[bugbytesinc]

Folks, this is still not ready. The protobuf is unworkable. SignedTransaction is the wrong archetype for BatchTransaction. It has no info for the batch payer, transaction id etc. If I read the spec correctly, you need a transaction for the batch itself.

If you don't want to hurt the service devs and the sdk devs; you need to create an AtomicTransactionListBody message that has a repeated bytes property named transactionBodyBytes that contains the bytes of your SignedTransactions one for each transaction. Then you need to add this AtomicTransactionListBody as one of the oneof data properties on the TransactionBody itself. Now you get all the transaction ids, payers timeouts etc etc. all lined up and accounted for properly.

[nickpoorman]

@bugbytesinc We can and are happy to propose an amendment to the HIP. Often, we find the implementation details change or are required to change as developers begin working on implementing a HIP. While I think it's certainly better that we iron out all the details in the initial HIP before going into "Last Call," the goal was to get this moving along since the PR has been open since Aug 2022. Now that it's in last call, we can create a new PR to amend the implementation details of the HIP.

[nickpoorman]

@bugbytesinc Please feel free to submit a PR to amend the original document with the protobuf changes you are requesting.

[kenthejr]

So, how does this affect the rate limit on HAPI? If I submit the max allowed create account transactions, what does that do to the rest of the network?

[nickpoorman]

@kenthejr The thought is that any transaction within a batch must fall within any rate limits already imposed on individual transactions. When Alice submits a batch transaction that includes individual tx1 and tx2, they are atomically applied as all-or-nothing, which means if tx1 succeeds because the limits have not been exhausted, then tx2 could fail if the limits are exhausted. At that point, the entire transaction would fail. However, if tx1 and tx2 individually within the batch do not cause the limits to be exhausted, then the batch would succeed as a whole.

[topshef]

Just bumping this excellent hip and looking forward to it coming to fruition, is there any eta?

btw I can imagine a dynamic market in layer 2 services building on this, eg forecasting transaction execution status, tracking reputation of signers etc

[jaycoolh]

Hey @topshef fyi this is coming out in next release

[topshef]

thanks a lot this is great news

had trouble finding link on mobile so posting here for ref
https://hips.hedera.com/hip/hip-551