Skip to content

Conversation

@jstuczyn
Copy link
Contributor

@jstuczyn jstuczyn commented Nov 3, 2025

this is a stepping stone towards the the naive versions NET-451 and NET-450 as part of NET-341(https://nymtech.atlassian.net/browse/NET-341
and supersedes #5986.

this PR contains all of the ongoing work that can be merged into develop to reduce code rot whilst additional work continues on a separate branch.

to fully understand the context behind the changes, it is crucial to familiarise yourself with the following document: https://nymtech.atlassian.net/wiki/spaces/KB/pages/263880725/Implementation#

as long as nothing gets put on the url controlled by UPGRADE_MODE_ATTESTATION_URL (by default "https://nym.com/upgrade-mode/attestation.json" on mainnet), there should be no changes in behaviour in our clients or nodes.

for reference for @simonwicky: this PR does NOT contain changes to BandwidthTicketProvider trait. this is still ongoing work on another branch

Changelog

the following changes are by no means exhaustive, because, due to the fault of my own, I have severely underestimated the scope of the work and haven't been keeping track of all the relevant updates and had to attempt to produce the list on a later date.

Env Changes

Added

  • UPGRADE_MODE_ATTESTATION_URL to control url used for polling attestation endpoint
  • UPGRADE_MODE_ATTESTER_ED25519_BS58_PUBKEY to control expected key used for signing the attestation - @benedettadavico @tommyv1987: please visit our vault in order to retrieve private keys for sandbox and canary

Interface Changes

Added

  • lints.workspace = true section to Cargo.toml files of additional crates and fixed associated warnings
  • nym-credentials-interface: BandwidthCredential to act as a wrapper for different types of credentials. currently it includes the old CredentialSpendingData (i.e. zk-nym ticketbooks) and JWT tokens issued during upgrade mode. in the future it is expected to get expanded to include emergency mode credentials issued by signer quorum

Changed

  • multiple struct naming typos, e.g. RegistredData => RegisteredData (including all conversion impls) or RegistredAndFree => RegisteredAndFree

Nym Node Changes

  • there is now additional upgrade-mode related shared state between:
    • Wireguard Metadata Endpoint
    • WebSocket Listener
    • Authenticator Service Provider
    • Attestation Watcher
      to control upgrade-mode related features
  • during upgrade mode client bandwidth (either wireguard or mixnet) is not metered anymore (base value is returned for legacy support - please refer to the confluence docs for details)

Authenticator Service Provider

Added

  • UpgradeModeStatus trait to expose information on the current state of the upgrade mode:
pub enum CurrentUpgradeModeStatus {
    Enabled,
    Disabled,
    // everything pre-v6
    Unknown,
}

it is implemented for all authenticator responses, i.e. PendingRegistration, Registered, RemainingBandwidth, TopUpBandwidth

  • BandwidthClaim struct that wraps around BandwidthCredential and includes explicit bandwidth kind (i.e. mixnet/wg entry/exit). note: conversion from CredentialSpendingData can fail if it has invalid embedded ticket data
  • finalise_registration method to existing PendingRegistrationResponse trait to remove code repetition and simplify version-matching logic
  • v6 version of authenticator messages
    • now includes CheckUpgradeMode to explicitly request upgrade mode state check
    • each response type includes additional upgrade_mode_enabled: bool field
    • FinalMessage uses BandwidthClaim instead of CredentialSpendingData for the credential field. this is reflected in the trait changes
    • for semver compatibility reasons, currently set as being released alongside mozzarella (1.22.0) - IF THE PR TAKES ADDITIONAL RELEASE CYCLE TO GET MERGED THIS HAS TO BE UPDATED

Changed

  • credential() method on the FinalMessage trait now returns Option<BandwidthClaim> as opposed to Option<CredentialSpendingData> to allow the upgrade mode flow where user would not have a valid zk-nym
  • new v6 authenticator version marked as "latest"
  • ClientMessage bytes() method now returns SerialisedRequest instead of the tuple (Vec<u8>, u64) to be more explicit about its content. the internal data stayed unchanged

AuthenticatorClient

Changed

  • query_bandwidth now returns Result<AvailableBandwidthClientResponse> which is a wrapper around bandwidth and upgrade mode information
  • top_up now returns Result<TopUpClientResponse> which is a wrapper around bandwidth and upgrade mode information

Gateway Mixnet WebSocket

Added

  • send_upgrade_mode_jwt on the client to allow sending upgrade mode token to the gateway in order to prompt it to perform state refresh
  • protocol version is now being negotiated during registration handshake - there is no new data being sent; it simply isn't being ignored anymore

Changed

  • bumped up the default CURRENT_PROTOCOL_VERSION from 5 to 6 ('upgrade mode')
  • ServerResponse::Authenticate contains upgrade_mode flag to indicate whether gateway thinks the system is undergoing the upgrade
  • ServerResponse::Register contains upgrade_mode flag to indicate whether gateway thinks the system is undergoing the upgrade
  • ServerResponse::Bandwidth contains upgrade_mode flag to indicate whether gateway thinks the system is undergoing the upgrade
  • ServerResponse::Send contains upgrade_mode flag to indicate whether gateway thinks the system is undergoing the upgrade

Fixed

  • client and gateway will now attempt to use the highest known protocol version rather than being incorrectly capped at v4

Wireguard Metadata Endpoint

Added

  • macros for implementing common conversions between different request/response types that are identical between versions (just for different types)
  • v2 version of metadata endpoint messages that include support for the upgrade mode

Changed

  • new v2 version marked as "latest"
  • RequestData::TopUpBandwidth now uses BandwidthCredential rather than `CredentialSpendingData
  • removed empty unit struct from RequestData::AvailableBandwidth
  • ResponseData::AvailableBandwidth now has explicit field indicating remaining bandwidth alongside upgrade mode information
  • ResponseData::TopUpBandwidth now has explicit field indicating available bandwidth alongside upgrade mode information

Testing changes

  • traits needed for testing are now directly implemented inside the shared crate under "testing" feature, e.g. Construct for Response now exists in common/wireguard-private-metadata/shared/src/models/v1/interface.rs instead of duplicating the type and reimplementig the trait in common/wireguard-private-metadata/tests/src/v1/interface.rs
  • introduced mock_connect_info to be able to write future tests against clients originating from different ip addresses
  • explicitly exposed versioned AppState per each test server
  • starting with v2 introduced MockPeerControllerV2 to allow registration of expected responses given particular request for more fine-grained test control
  • starting with v2 introduced ServerTest that wraps around different test features and provides helpful interface for interacting with the state

This change is Reviewable

@vercel
Copy link

vercel bot commented Nov 3, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Preview Comments Updated (UTC)
nym-explorer-v2 Error Error Nov 14, 2025 0:50am
nym-node-status Ready Ready Preview Comment Nov 14, 2025 0:50am
1 Skipped Deployment
Project Deployment Preview Comments Updated (UTC)
docs-nextra Ignored Ignored Preview Nov 14, 2025 0:50am

@jstuczyn jstuczyn changed the title Feature/additional upgrade mode feat: merge intermediate upgrade mode changes Nov 3, 2025
Base automatically changed from feature/credential-proxy-jwt to develop November 3, 2025 16:42
@jstuczyn jstuczyn force-pushed the feature/additional-upgrade-mode branch from 7a01618 to 7785472 Compare November 3, 2025 16:45
@jstuczyn jstuczyn force-pushed the feature/additional-upgrade-mode branch from 7785472 to 8329327 Compare November 3, 2025 17:04
@jstuczyn jstuczyn marked this pull request as ready for review November 4, 2025 14:38
@jstuczyn jstuczyn requested review from neacsu and simonwicky November 4, 2025 15:02
Copy link
Contributor

@neacsu neacsu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@neacsu reviewed 70 of 158 files at r1, all commit messages.
Reviewable status: 70 of 158 files reviewed, all discussions resolved (waiting on @simonwicky)

jstuczyn added a commit that referenced this pull request Nov 7, 2025
… easily resolve merge conflicts during rebasing

added additional v2 query for metadata endpoint for requesting upgrade mode recheck

added additional message to v6 authenticator to request explicit upgrade mode recheck

clippy

test fixes due to updated keys

updated assertion for upgrading v1 top up request to v2

compare attester public key against the expected value within the credential proxy

use pre-generated attestation public keys within nym-nodes

remove version deprecation

bugfix: default bandwidth response for authenticator

expose upgrade mode information in authenticator responses

adding tests for new v2 server

passing upgrade mode information in metadata endpoint

v2 wireguard private metadata

bugfix: make sure to immediately poll for attestation after spawning task

fix gateway probe and remove code duplication for finalizing registration

squashing before rebasing

post rebasing fixes

AuthenticatorVersion helpers

additional nits

allow unwraps in mocks

fixed linux build

clippy

integrating upgrade mode into authenticator

fixed build after adding wrappers to response types

conditionally updating peer handle bandwidth

cleanup

negotiate initial protocol during registration

change auth to use highest protocol

handler for JWT message

dont meter client bandwidth in upgrade mode

handling recheck requests

sending information about upgrade_mode on client messages

gateway watching for upgrade mode attestation

wip: gateways to disable bandwidth metering on upgrade mode
@jstuczyn jstuczyn force-pushed the feature/additional-upgrade-mode branch from 35ba5de to 1b159c8 Compare November 7, 2025 16:19
jstuczyn added a commit that referenced this pull request Nov 13, 2025
… easily resolve merge conflicts during rebasing

added additional v2 query for metadata endpoint for requesting upgrade mode recheck

added additional message to v6 authenticator to request explicit upgrade mode recheck

clippy

test fixes due to updated keys

updated assertion for upgrading v1 top up request to v2

compare attester public key against the expected value within the credential proxy

use pre-generated attestation public keys within nym-nodes

remove version deprecation

bugfix: default bandwidth response for authenticator

expose upgrade mode information in authenticator responses

adding tests for new v2 server

passing upgrade mode information in metadata endpoint

v2 wireguard private metadata

bugfix: make sure to immediately poll for attestation after spawning task

fix gateway probe and remove code duplication for finalizing registration

squashing before rebasing

post rebasing fixes

AuthenticatorVersion helpers

additional nits

allow unwraps in mocks

fixed linux build

clippy

integrating upgrade mode into authenticator

fixed build after adding wrappers to response types

conditionally updating peer handle bandwidth

cleanup

negotiate initial protocol during registration

change auth to use highest protocol

handler for JWT message

dont meter client bandwidth in upgrade mode

handling recheck requests

sending information about upgrade_mode on client messages

gateway watching for upgrade mode attestation

wip: gateways to disable bandwidth metering on upgrade mode
@jstuczyn jstuczyn force-pushed the feature/additional-upgrade-mode branch from 70abac2 to 85856d7 Compare November 13, 2025 10:03
jstuczyn added a commit that referenced this pull request Nov 13, 2025
… easily resolve merge conflicts during rebasing

added additional v2 query for metadata endpoint for requesting upgrade mode recheck

added additional message to v6 authenticator to request explicit upgrade mode recheck

clippy

test fixes due to updated keys

updated assertion for upgrading v1 top up request to v2

compare attester public key against the expected value within the credential proxy

use pre-generated attestation public keys within nym-nodes

remove version deprecation

bugfix: default bandwidth response for authenticator

expose upgrade mode information in authenticator responses

adding tests for new v2 server

passing upgrade mode information in metadata endpoint

v2 wireguard private metadata

bugfix: make sure to immediately poll for attestation after spawning task

fix gateway probe and remove code duplication for finalizing registration

squashing before rebasing

post rebasing fixes

AuthenticatorVersion helpers

additional nits

allow unwraps in mocks

fixed linux build

clippy

integrating upgrade mode into authenticator

fixed build after adding wrappers to response types

conditionally updating peer handle bandwidth

cleanup

negotiate initial protocol during registration

change auth to use highest protocol

handler for JWT message

dont meter client bandwidth in upgrade mode

handling recheck requests

sending information about upgrade_mode on client messages

gateway watching for upgrade mode attestation

wip: gateways to disable bandwidth metering on upgrade mode
@jstuczyn jstuczyn force-pushed the feature/additional-upgrade-mode branch from 85856d7 to ec7f499 Compare November 13, 2025 10:10
jstuczyn added a commit that referenced this pull request Nov 13, 2025
this includes bringing over changes introduced in #6174
jstuczyn added a commit that referenced this pull request Nov 14, 2025
this includes bringing over changes introduced in #6174
… easily resolve merge conflicts during rebasing

added additional v2 query for metadata endpoint for requesting upgrade mode recheck

added additional message to v6 authenticator to request explicit upgrade mode recheck

clippy

test fixes due to updated keys

updated assertion for upgrading v1 top up request to v2

compare attester public key against the expected value within the credential proxy

use pre-generated attestation public keys within nym-nodes

remove version deprecation

bugfix: default bandwidth response for authenticator

expose upgrade mode information in authenticator responses

adding tests for new v2 server

passing upgrade mode information in metadata endpoint

v2 wireguard private metadata

bugfix: make sure to immediately poll for attestation after spawning task

fix gateway probe and remove code duplication for finalizing registration

squashing before rebasing

post rebasing fixes

AuthenticatorVersion helpers

additional nits

allow unwraps in mocks

fixed linux build

clippy

integrating upgrade mode into authenticator

fixed build after adding wrappers to response types

conditionally updating peer handle bandwidth

cleanup

negotiate initial protocol during registration

change auth to use highest protocol

handler for JWT message

dont meter client bandwidth in upgrade mode

handling recheck requests

sending information about upgrade_mode on client messages

gateway watching for upgrade mode attestation

wip: gateways to disable bandwidth metering on upgrade mode
@jstuczyn jstuczyn merged commit 6b2bb30 into develop Nov 14, 2025
17 of 21 checks passed
@jstuczyn jstuczyn deleted the feature/additional-upgrade-mode branch November 14, 2025 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants