How to get Federation working behind Cloudflare CDN and reverse proxy on local network

[naTariks]

Playbook Configuration:

My vars.yml file looks like this:

---
# The bare domain name which represents your Matrix identity.
# Matrix user IDs for your server will be of the form (`@alice:example.com`).
#
# Note: this playbook does not touch the server referenced here.
# Installation happens on another server ("matrix.example.com", see `matrix_server_fqn_matrix`).
#
# If you've deployed using the wrong domain, you'll have to run the Uninstalling step,
# because you can't change the Domain after deployment.
matrix_domain: nathanrichard.dev

# The Matrix homeserver software to install.
# See:
#  - `roles/custom/matrix-base/defaults/main.yml` for valid options
#  - the `docs/configuring-playbook-IMPLEMENTATION_NAME.md` documentation page, if one is available for your implementation choice
#
# By default, we use Synapse, because it's the only full-featured Matrix server at the moment.
#
# Note that the homeserver implementation of a server will not be able to be changed without data loss.
matrix_homeserver_implementation: synapse

# A secret used as a base, for generating various other secrets.
# You can put any string here, but generating a strong one is preferred. You can create one with a command like `pwgen -s 64 1`.
matrix_homeserver_generic_secret_key: '<SECRET>'

# By default, the playbook manages its own Traefik (https://doc.traefik.io/traefik/) reverse-proxy server.
# It will retrieve SSL certificates for you on-demand and forward requests to all other components.
# For alternatives, see `docs/configuring-playbook-own-webserver.md`.
matrix_playbook_reverse_proxy_type: playbook-managed-traefik

# Ensure that public urls use https
matrix_playbook_ssl_enabled: true

# Disable the web-secure (port 443) endpoint, which also disables SSL certificate retrieval.
# This has the side-effect of also automatically disabling TLS for the matrix-federation entrypoint
# (by toggling `matrix_federation_traefik_entrypoint_tls`).
traefik_config_entrypoint_web_secure_enabled: false

# If your reverse-proxy runs on another machine, consider using `0.0.0.0:81`, just `81` or `SOME_IP_ADDRESS_OF_THIS_MACHINE:81`
traefik_container_web_host_bind_port: '0.0.0.0:81'

# We bind to `127.0.0.1` by default (see above), so trusting `X-Forwarded-*` headers from
# a reverse-proxy running on the local machine is safe enough.
# If you're publishing the port (`traefik_container_web_host_bind_port` above) to a public network interface:
# - remove the `traefik_config_entrypoint_web_forwardedHeaders_insecure` variable definition below
# - uncomment and adjust the `traefik_config_entrypoint_web_forwardedHeaders_trustedIPs` line below
traefik_config_entrypoint_web_forwardedHeaders_insecure: true
# traefik_config_entrypoint_web_forwardedHeaders_trustedIPs: ['IP-ADDRESS-OF-YOUR-REVERSE-PROXY']

# Expose the federation entrypoint on a custom port (other than port 8448, which is normally used publicly).
#
# We bind to `127.0.0.1` by default (see above), so trusting `X-Forwarded-*` headers from
# a reverse-proxy running on the local machine is safe enough.
#
# If your reverse-proxy runs on another machine, consider:
# - using `0.0.0.0:8449`, just `8449` or `SOME_IP_ADDRESS_OF_THIS_MACHINE:8449` below
# - adjusting `matrix_playbook_public_matrix_federation_api_traefik_entrypoint_config_custom` (below) - removing `insecure: true` and enabling/configuring `trustedIPs`
matrix_playbook_public_matrix_federation_api_traefik_entrypoint_host_bind_port: '0.0.0.0:8449'

# Disable HTTP/3 for the federation entrypoint.
# If you'd like HTTP/3, consider configuring it for your other reverse-proxy.
#
# Disabling this also sets `matrix_playbook_public_matrix_federation_api_traefik_entrypoint_host_bind_port_udp` to an empty value.
# If you'd like to keep HTTP/3 enabled here (for whatever reason), you may wish to explicitly
# set `matrix_playbook_public_matrix_federation_api_traefik_entrypoint_host_bind_port_udp` to something like '127.0.0.1:8449'.
matrix_playbook_public_matrix_federation_api_traefik_entrypoint_config_http3_enabled: false

# Depending on the value of `matrix_playbook_public_matrix_federation_api_traefik_entrypoint_host_bind_port` above,
# this may need to be reconfigured. See the comments above.
matrix_playbook_public_matrix_federation_api_traefik_entrypoint_config_custom:
  forwardedHeaders:
    trustedIPs: ['192.168.68.63']
    # insecure: true

# Controls whether container networks will be created with IPv6 support.
#
# If you also have IPv6 support on your server/network and AAAA DNS records pointing to the server,
# enabling this will effectively give you full public IPv6 connectivity (powered by NAT66).
#
# We recommend leaving this enabled even if you don't currently have IPv6 connectivity on your server/network.
# This way, once you eventually get IPv6 connectivity, you won't have to change anything (besides DNS records).
#
# Flipping this setting later on requires manual work (stopping services, deleting and recreating all container networks).
#
# In the future, this setting will likely default to `true`, so if you really want IPv6 disabled, explicitly set this to `false`.
#
# People managing Docker themselves and running an older Docker version will need additional configuration.
#
# Learn more in `docs/configuring-ipv6.md`.
devture_systemd_docker_base_ipv6_enabled: true

# A Postgres password to use for the superuser Postgres user (called `matrix` by default).
#
# The playbook creates additional Postgres users and databases (one for each enabled service) using this superuser account.
#
# Changing this value subsequently requires manual work.
# The value used here must be shorter than 100 characters.
postgres_connection_password: '<SECRET>'

# By default, we configure coturn's external IP address using the value specified for `ansible_host` in your `inventory/hosts` file.
# If this value is an external IP address, you can skip this section.
#
# If `ansible_host` is not the server's external IP address, you have 2 choices:
# 1. Uncomment the line below, to allow IP address auto-detection to happen (more on this below)
# 2. Uncomment and adjust the line below to specify an IP address manually
#
# By default, auto-detection will be attempted using the `https://ifconfig.co/json` API.
# Default values for this are specified in `matrix_coturn_turn_external_ip_address_auto_detection_*` variables in the coturn role
# (see `roles/custom/matrix-coturn/defaults/main.yml`).
#
# If your server has multiple IP addresses, you may define them in another variable which allows a list of addresses.
# Example: `matrix_coturn_turn_external_ip_addresses: ['1.2.3.4', '4.5.6.7']`
#
matrix_coturn_turn_external_ip_address: ''

# Optimisation for Raspberry Pi
# set CPU architecture
matrix_architecture: arm64

# Disabling this will prevent email-notifications and other such things from working.
exim_relay_enabled: false

# This makes Synapse not keep track of who is online/offline.
#
# Keeping track of this and announcing such online-status in federated rooms with
# hundreds of servers inside is insanely heavy (https://github.com/matrix-org/synapse/issues/3971).
#
# If your server does not federate with hundreds of others, enabling this doesn't hurt much.
matrix_synapse_presence_enabled: false

matrix_playbook_docker_installation_enabled: false

# Element Client
matrix_client_element_default_theme: 'dark'
matrix_client_element_themes_enabled: true

# Enable the Element Call frontend UI to allow standalone use of Element Call.
# Enabling this also auto-enables the Matrix RTC stack.
matrix_element_call_enabled: true

# install synapse-admin
# allows oversight of users, generation of such and access tokens
# Also allow registration, but with access token, which can be generated on synapse-admin
matrix_synapse_admin_enabled: true
matrix_synapse_enable_registration: true
matrix_synapse_registration_requires_token: true

# config for maturix bridges
matrix_admin: "<SECRET>:nathanrichard.dev"

matrix_bridges_encryption_enabled: true
matrix_bridges_encryption_default: true

matrix_appservice_double_puppet_enabled: true

# signal bridge
matrix_mautrix_signal_enabled: true

# whatsapp bridge
matrix_mautrix_whatsapp_enabled: true

# open to federation
matrix_server_fqn_matrix_federation: "federation.{{ matrix_domain }}"
matrix_synapse_http_listener_resource_name: ["client", "federation"]

matrix_federation_public_port: 443
matrix_synapse_federation_port_enabled: false

matrix_synapse_tls_federation_listener_enabled: false

Matrix Server:

• OS: DietPi v9.12.1 (Debian GNU/Linux 12 (bookworm) aarch64)
• Architecture: Raspberry Pi 5 16g arm64

Problem description:

System Architecture: I'm running two Raspberry Pi in my home network to which my base url (nathanrichard.dev) points. As a DNS I use Cloudflare and it's proxies. In my network, the one Raspberry Pi runs a caddy2 reverse Proxy, from which I reference the other Raspberry Pi running the Synapse Server.

To make it work I followed the guide of Fronting the integrated webserver with another reverse-proxy (changes in vars.yml) and redirect the traffic with rules in caddy 2 (see below).
As the Cloudflare Proxy doesn't allow port 8448 I also followed the guide on changing the federation port from 8448 to 443.

The Server runs fine, I can write messages etc, but I can't figure out how to get federation working. In the .well-known server file the address now is federation.nathanrichard.dev:443.
This URL is handled by caddy2 and should redirect it to the synapse server with port 8449.

Unfortunately, when I run the Matrix Federation Tester the response is an Error 502 (Bad Gateway)

I assume that with the change of the port (8448 -> 443) I also have to redirect it to another address on the synapse server, but I'm so lost, as I invested a lot of time in trying different things and still couldn't figure it out.

Do I have to change something from the configuration or reference another port in the reverse Proxy?

Additional context
Caddyfile:

{
  acme_dns cloudflare {env.CF_API_TOKEN}
}

www.nathanrichard.dev {
        redir https://nathanrichard.dev{uri}
}

nathanrichard.dev {
        reverse_proxy /.well-known/matrix/* https://matrix.nathanrichard.dev {
                header_up Host {http.reverse_proxy.upstream.hostport}
        }
}

matrix.nathanrichard.dev, element.nathanrichard.dev, call.element.nathanrichard.dev {
        reverse_proxy 192.168.68.94:81 {
               header_up X-Forwarded-Port {http.request.port}
               header_up X-Forwarded-TlsProto {tls_protocol}
               header_up X-Forwarded-TlsCipher {tls_cipher}
               header_up X-Forwarded-HttpsProto {proto}
        }
}

federation.nathanrichard.dev {
        reverse_proxy 192.168.68.94:8449 {
               header_up X-Forwarded-Port {http.request.port}
               header_up X-Forwarded-TlsProto {tls_protocol}
               header_up X-Forwarded-TlsCipher {tls_cipher}
               header_up X-Forwarded-HttpsProto {proto}
        }
}