Transfer-v2: initial proposal

[piegamesde]

Motivation

The current file transfer implementations has limitations that cannot be resolved in a fully backwards-compatible way. Thus, I present the idea of a second generation transfer protocol that allows for modern features like partial/resumed transfer and better directory sending.

Proposal

The sender sends an offer of files with their sizes and time stamps. If the user acknowledges, the receiver responds with an accept. That message may contain byte offsets for individual files. The sender must send the requested files, starting from that offset. This is how retransfers can be implemented. The sender sends all files packed into a tarball, optionally compressed. Tar has the advantage over zip that files can be streamed from and into the archive without having to know its compressed size in advance.

Backwards compatibility

The "application version" key sent is currently not really used by the protocol, but it states that unknown values should be ignored. Thus, we can define new keys in there without breaking anything. If those keys are not present, defaults that imply "v1" must be used.

Thus, the existing APPID can continue to be used. Applications that support the new protocol can recognize peers that don't and fall back to v1.

Future proofing

Future proofing is only done for the versions message. The versions message specifies an open list of "feature flags". Two applications exchange their feature flags and then take the union, thus they only use supported features.

Future proofing further messages is not needed.

Unresolved questions

This proposal does not take the dilation API into account, as the latter has not been written down yet. It thus builds upon the relay protocol instead. It would make sense to coordinate efforts between the APIs so that they match their features and fit well together.

[meejah]

Interesting!

My first thoughts around this went towards: minimal negotiation, set up a direct peer connection and then do file-transfer offer/acks inline. That is, before setting up the peer connection, decide who is the "sender" ("leader"? "initiator"?) and go from there .. with offers etc inline over the peer connection.

This would also allow more open-ended UX from the endpoints. For example, once the connection is set up the participants may want to add more files or transfer files in both directions, etc. My reasoning here is that the "hard thing" is setting up the connection in the first place.

"Dilation" is I guess related here and could complicate the thinking: lets say that we have Dilation already. That means that the UX on the endpoints could indeed easily do what I suggest above (i.e. back-and-forth transfers, adding more files, etc) by leveraging the Dilation protocol to re-open the peer connection for every file. This would actually simplify the file-protocol since you'd only ever have to do a single transfer over any given connection (so there's just "offer" and "ack" with the latter possibly hinting "I already have 1234 bytes" or whatever too and then bulk-transfer of bytes).

[piegamesde]

The feature of sending text messages (without a transit connection), on the other hand, got removed." <-- what do you mean here?

If you want to send simple text messages, either use the v1 protocol or wrap them as a "paste" file.

[piegamesde]

I have been toying with the idea of using the rendezvous connection as little as possible, and do everything over the relay. This gives us:

• Less load on the rendezvous server, which is the single point of failure and the only thing we can't easily load balance.
• Less metadata leak if the peers managed to get a local connection running. In this case, absolutely no traffic will leave their network.

For this to work, we have to:

• Move the transit connection establishing to be the first step
• Encode our file data within normal messages. The easiest way would be to switch to a binary JSON equivalent, where we can simply add the payload as bytes.

And then we get for free:

• By simply adding a transaction ID to all messages, we can easily have multiple distinct transfers going on concurrently, and from both sides.

[meejah]

Encode our file data within normal messages. The easiest way would be to switch to a binary JSON equivalent, where we can simply add the payload as bytes.

I don't see why we need a binary alternative to JSON here; after all, "encoded text" is just bytes.

[piegamesde]

Yes encoded text is just bytes, but for JSON we'd need the opposite direction: encode bytes (of the file) as text (JSON string). Which would require base64 or something which comes with processing and file size overhead. msgpack has a really similar API to JSON, except that it can encode arbitrary binary payloads without overhead (and also has size optimizations for other types of fields).

[meejah]

Yes encoded text is just bytes, but for JSON we'd need the opposite direction: encode bytes (of the file) as text (JSON string). Which would require base64 or something which comes with processing and file size overhead. msgpack has a really similar API to JSON, except that it can encode arbitrary binary payloads without overhead (and also has size optimizations for other types of fields).

This sounds like mixing two problems together: message framing, and message encoding. I am familiar with msgpack (but would also consider CBOR here, for binary message encoding).

We could decide that "control" messages are encoded in JSON (or whatever) while "data" messages are just bytes. That is, the framing part might need a way to decide what sort of message it just got. It also needs to decide which (logical) channel the message applies to anyway (and needs to handle encryption) so it'll already need some structure on top of the "payload" bytes (whether those payload bytes are UTF8-encoded JSON, msgpack, CBOR, etc). So I think what you're suggesting is to use msgpack to do both the framing and message-encoding (forcing all messages to be msgpack)..? But how will that handle encryption?

I'll also note here that all these problems are already considered and addressed by Dilation; making another standard for multiplexing data will need to re-consider them all. I'm not convinced this is any less work than simply using Dilation and using the control-channel as intended (that is, for control messages, for which we would have to specify an encoding .. e.g. CBOR, msgpack, JSON, etc). Sub-channels can carry anything, so e.g. "one subchannel per transfer" means all the messages on that subchannel can simply be the bytes for that transfer (since the metadata was already communicated in the offer via the control subchannel).

[piegamesde]

The data flow is unchanged: You have some application layer message (let it be JSON, or whatever). From the Transit (or Dilation) layer POV, it is just another opaque chunk of bytes. It will dutifully encrypt them and add a header for framing (just the length for Transit, a few control bytes more for Dilation). All of this is completely independent to how the application layer encodes their messages.

We could decide that "control" messages are encoded in JSON (or whatever) while "data" messages are just bytes. That is, the framing part might need a way to decide what sort of message it just got.

This sounds like more effort. If we use a data format that efficiently supports binary values, we can simply mix and match contents as desired. Applications would just send messages and don't have to deal with "oh this is a control message now we have to use base64 for this bit". And they could just tag some metadata next to the payload without any effort. For example, the messages to the rendezvous server are JSON but they contain binary payloads (the encrypted application messages, which are JSON again themselves but the rendezvous server never sees this anyways). If the protocol had gone with a different encoding, there would be no need for base64, and the implementation would not be any more complex.

Re: file format choice—I don't have any strong personal preferences for a format, as long as messages are self-describing and support binary payloads. Also, there should be off the shelf implementations for various languages. I did a survey back when I decided on msgpack and it checked most of the boxes. The details of the decision can still be found somewhere in the backlog of #magic-wormhole. (I should dig it out some time because this topic comes up again every once in a while)

[meejah]

Applications already do simply send "messages" ... so are you now talking about changing the mailbox protocol too?

(Since "transfer v2" is an application protocol, lets be clear when discussing other magic-wormhole protocols like the actual Dilation protocol or the actual Mailbox protocol).

If you're in fact proposing to switch the mailbox protocol to use a different encoding that's orthogonal to any transfer-v2 proposal. Same with any Dilation changes.

Above I'm talking from "file-transfer, an application-level protocol" perspective. It sounds like you're proposing an alternative Transit protocol which multiplexes messages, so I was also trying to compare that approach to simply using Dilation (which also does that) -- where the application is deciding e.g. the formatting of all control-channel messages (and what's in any other subchannels). Of course, it doesn't control anything higher-up into the Dilation protocol nor the Mailbox protocol.

[piegamesde]

Applications already do simply send "messages" ... so are you now talking about changing the mailbox protocol too?

No, this was just an example.

It sounds like you're proposing an alternative Transit protocol which multiplexes messages

In the past I had indeed advocated for this. Since we are using Dilation, it makes no sense to have yet another level of multiplexing, so I removed it from the proposal.

[ewanas]

Totally agree with msgpack!