docs: remove ephemeral auth on Kafka API (#1210)

Co-authored-by: Paulo Borges <paulohtb@hotmail.com>
Co-authored-by: Joyce Fee <102751339+Feediver1@users.noreply.github.com>
Co-authored-by: Paulo Borges <paulohtb@Paulos-MacBook-Pro.local>

Author: 4 people

modules/get-started/pages/release-notes/redpanda.adoc

@@ -1,6 +1,7 @@
 = What's New in Redpanda
 :description: Summary of new features and updates in this Redpanda release.
 :page-aliases: get-started:whats-new-233.adoc, get-started:whats-new-241.adoc, get-started:whats-new.adoc
+:page-whats-new:
 
 This topic includes new content added in version {page-component-version}. For a complete list of all product updates, see the https://github.com/redpanda-data/redpanda/releases/[Redpanda release notes^]. See also:
 
@@ -18,6 +19,21 @@ The Redpanda Admin API now includes new health probes to help you ensure safe br
 * xref:manage:cluster-maintenance/rolling-restart.adoc[]
 * xref:upgrade:rolling-upgrade.adoc[]
 
+== HTTP Proxy authentication changes
+
+[IMPORTANT]
+====
+include::shared:partial$http-proxy-ephemeral-credentials-breaking-change.adoc[]
+====
+
+HTTP Proxy previously used automatically-generated ephemeral credentials to authenticate with the Kafka API when the HTTP Proxy listeners specified `authentication_method: none`. To improve security and simplify the authentication model, ephemeral credentials are no longer available.
+
+If you need to maintain the current HTTP Proxy functionality while transitioning to authenticated clients, configure the following HTTP Proxy client properties in your `redpanda.yaml` configuration:
+
+- xref:reference:properties/broker-properties.adoc#scram_username[`scram_username`]: Username for SASL/SCRAM authentication
+- xref:reference:properties/broker-properties.adoc#scram_password[`scram_password`]: Password for SASL/SCRAM authentication  
+- xref:reference:properties/broker-properties.adoc#sasl_mechanism[`sasl_mechanism`]: SASL mechanism (typically `SCRAM-SHA-256` or `SCRAM-SHA-512`)
+
 == Redpanda Console v3.0.0
 
 The Redpanda Console v3.0.0 release includes the following updates:

modules/manage/pages/security/authentication.adoc

@@ -4,6 +4,7 @@
 :page-aliases: security:authentication.adoc
 :page-toclevels: 3
 :page-categories: Management, Security
+:page-http-proxy-auth:
 
 include::manage:partial$authentication.adoc[]
 

modules/manage/partials/authentication.adoc

@@ -655,12 +655,17 @@ endif::[]
 [[schema-and-http-to-redpanda]]
 ===== Configure Schema Registry and HTTP Proxy to connect to Redpanda with SASL
 
-Schema Registry and HTTP Proxy connect to Redpanda over the Kafka API. For the Kafka username and password, Redpanda uses ephemeral credentials internal to the cluster. Ephemeral credentials are regular SCRAM credentials, but they're only stored in memory and are lost when a broker restarts. When the Schema Registry or HTTP Proxy start up, they broadcast an ephemeral credential to other brokers over the internal RPC. If authentication fails to a particular broker, new ephemeral credentials are sent to that broker, and the service reconnects.
+Schema Registry and HTTP Proxy connect to Redpanda over the Kafka API.
+
+[IMPORTANT]
+====
+include::shared:partial$http-proxy-ephemeral-credentials-breaking-change.adoc[]
+====
 
 Schema Registry and HTTP Proxy support only the SASL/SCRAM mechanism.
 
 ifndef::env-kubernetes[]
-You can override the ephemeral credentials and manually configure Schema Registry and HTTP Proxy to connect to Redpanda with SASL. This approach is particularly useful when you want to apply specific access control through ACLs for the Schema Registry and HTTP Proxy users.
+Starting in Redpanda 25.2, you must manually configure HTTP Proxy to connect to Redpanda with SASL when HTTP Proxy API listeners use `authentication_method: none`. You can also manually configure Schema Registry to connect to Redpanda with SASL. This approach is particularly useful when you want to apply specific access control through ACLs for the Schema Registry and HTTP Proxy users.
 
 . xref:manage:security/authorization/index.adoc[Create appropriate ACLs] for the Schema Registry and HTTP Proxy users to define and restrict their access rights within the Redpanda cluster.
 
@@ -698,7 +703,22 @@ schema_registry_client:
   sasl_mechanism: SCRAM-SHA-256
 ----
 +
-For HTTP Proxy:
+For HTTP Proxy (required when `authentication_method: none` starting in Redpanda 25.2):
++
+[,yaml]
+----
+pandaproxy_client:
+  brokers:
+    - address: 127.0.0.1
+      port: 9092
+  scram_username: <username>
+  scram_password: <password>
+  sasl_mechanism: SCRAM-SHA-256
+----
++
+When HTTP Proxy API listeners use `authentication_method: none`, the HTTP Proxy client uses these credentials to authenticate with the Kafka API. The user specified in `scram_username` must have appropriate permissions to access the required Kafka resources.
++
+If TLS is enabled for the Kafka API, additional configuration is required:
 +
 [,yaml]
 ----
@@ -1757,6 +1777,11 @@ For all available endpoints, see xref:api:ROOT:pandaproxy-schema-registry.adoc[]
 
 To disable authentication for a listener, set `authentication_method` to `none`:
 
+[IMPORTANT]
+====
+include::shared:partial$http-proxy-ephemeral-credentials-breaking-change.adoc[]
+====
+
 ifdef::env-kubernetes[]
 [tabs]
 ======

modules/reference/pages/properties/broker-properties.adoc

@@ -778,7 +778,9 @@ Delay (in milliseconds) for initial retry backoff.
 
 === sasl_mechanism
 
-The SASL mechanism to use when connecting.
+The SASL mechanism to use when the HTTP Proxy client connects to the Kafka API. These credentials are used when the HTTP Proxy API listener has `authentication_method: none` but the cluster requires authenticated access to the Kafka API.
+
+include::shared:partial$http-proxy-ephemeral-credentials-breaking-change.adoc[]
 
 *Visibility:* `user`
 
@@ -790,7 +792,9 @@ The SASL mechanism to use when connecting.
 
 === scram_password
 
-Password to use for SCRAM authentication mechanisms.
+Password to use for SCRAM authentication mechanisms when the HTTP Proxy client connects to the Kafka API. This property is required when the HTTP Proxy API listener has `authentication_method: none` but the cluster requires authenticated access to the Kafka API.
+
+include::shared:partial$http-proxy-ephemeral-credentials-breaking-change.adoc[]
 
 *Visibility:* `user`
 
@@ -802,7 +806,9 @@ Password to use for SCRAM authentication mechanisms.
 
 === scram_username
 
-Username to use for SCRAM authentication mechanisms.
+Username to use for SCRAM authentication mechanisms when the HTTP Proxy client connects to the Kafka API. This property is required when the HTTP Proxy API listener has `authentication_method: none` but the cluster requires authenticated access to the Kafka API.
+
+include::shared:partial$http-proxy-ephemeral-credentials-breaking-change.adoc[]
 
 *Visibility:* `user`
 

modules/shared/partials/http-proxy-ephemeral-credentials-breaking-change-short.adoc

@@ -0,0 +1 @@
+Starting in Redpanda 25.2, ephemeral credentials for HTTP Proxy are removed. You must configure explicit SASL credentials if your HTTP Proxy listeners use `authentication_method: none`.

modules/shared/partials/http-proxy-ephemeral-credentials-breaking-change.adoc

@@ -0,0 +1,11 @@
+*Breaking change in Redpanda 25.2:* Ephemeral credentials for HTTP Proxy are removed. If your HTTP Proxy API listeners use `authentication_method: none`, you must configure explicit SASL credentials (xref:reference:properties/broker-properties.adoc#scram_username[`scram_username`], xref:reference:properties/broker-properties.adoc#scram_password[`scram_password`], and xref:reference:properties/broker-properties.adoc#sasl_mechanism[`sasl_mechanism`]) for HTTP Proxy to authenticate with the Kafka API. Without these credentials, HTTP Proxy will attempt unauthenticated connections to the Kafka API. 
+
+Redpanda Data recommends enabling authentication on both HTTP Proxy and the Kafka API.
+
+ifndef::page-http-proxy-auth[]
+For configuration instructions, see xref:manage:security/authentication.adoc#schema-and-http-to-redpanda[Configure HTTP Proxy to connect to Redpanda with SASL].
+endif::[]
+
+ifndef::page-whats-new[]
+For details about this breaking change, see xref:get-started:release-notes/redpanda.adoc#http-proxy-authentication-changes[What's new].
+endif::[]

modules/upgrade/partials/incompat-changes.adoc

@@ -1,5 +1,7 @@
 === Review incompatible changes
 
+include::shared:partial$http-proxy-ephemeral-credentials-breaking-change.adoc[]
+
 * Redpanda Console v3.0.0 introduces breaking changes. If you are using Redpanda Console v2.x, xref:migrate:console-v3.adoc[review the migration guide] to address breaking changes before upgrading Redpanda Console.
 
 * Starting in version 24.2, when managing configuration properties using the AlterConfigs API directly, Redpanda resets all unspecified values to the default values. This aligns more closely with the behavior in Apache Kafka. There is no change if you're managing your configuration with tools like `rpk`, Redpanda Console, Kubernetes, Helm, or Terraform.