diff --git a/docs/_snippets/_users-and-roles-common.md b/docs/_snippets/_users-and-roles-common.md index 964f9a8b40d..0d2d3fbdf00 100644 --- a/docs/_snippets/_users-and-roles-common.md +++ b/docs/_snippets/_users-and-roles-common.md @@ -120,49 +120,51 @@ With this set of examples: Roles are used to define groups of users for certain privileges instead of managing each user separately. -1. Create a role to restrict users of this role to only see `column1` in database `db1` and `table1`: + + +##### Create a role to restrict users of this role to only see `column1` in database `db1` and `table1`: {#create-column-role} ```sql CREATE ROLE column1_users; ``` -2. Set privileges to allow view on `column1` +##### Set privileges to allow view on `column1` {#set-column-privileges} ```sql GRANT SELECT(id, column1) ON db1.table1 TO column1_users; ``` -3. Add the `column_user` user to the `column1_users` role +##### Add the `column_user` user to the `column1_users` role {#add-column-user-to-role} ```sql GRANT column1_users TO column_user; ``` -4. Create a role to restrict users of this role to only see selected rows, in this case, only rows containing `A` in `column1` +##### Create a role to restrict users of this role to only see selected rows, in this case, only rows containing `A` in `column1` {#create-row-role} ```sql CREATE ROLE A_rows_users; ``` -5. Add the `row_user` to the `A_rows_users` role +##### Add the `row_user` to the `A_rows_users` role {#add-row-user-to-role} ```sql GRANT A_rows_users TO row_user; ``` -6. Create a policy to allow view on only where `column1` has the values of `A` +##### Create a policy to allow view on only where `column1` has the values of `A` {#create-row-policy} ```sql CREATE ROW POLICY A_row_filter ON db1.table1 FOR SELECT USING column1 = 'A' TO A_rows_users; ``` -7. Set privileges to the database and table +##### Set privileges to the database and table {#set-db-table-privileges} ```sql GRANT SELECT(id, column1, column2) ON db1.table1 TO A_rows_users; ``` -8. grant explicit permissions for other roles to still have access to all rows +##### Grant explicit permissions for other roles to still have access to all rows {#grant-other-roles-access} ```sql CREATE ROW POLICY allow_other_users_filter @@ -173,17 +175,21 @@ Roles are used to define groups of users for certain privileges instead of manag When attaching a policy to a table, the system will apply that policy, and only those users and roles defined will be able to do operations on the table, all others will be denied any operations. In order to not have the restrictive row policy applied to other users, another policy must be defined to allow other users and roles to have regular or other types of access. ::: + + ## Verification {#verification} ### Testing role privileges with column restricted user {#testing-role-privileges-with-column-restricted-user} -1. Log into the clickhouse client using the `clickhouse_admin` user + + +##### Log into the clickhouse client using the `clickhouse_admin` user {#login-admin-user} ```bash clickhouse-client --user clickhouse_admin --password password ``` -2. Verify access to database, table and all rows with the admin user. +##### Verify access to database, table and all rows with the admin user. {#verify-admin-access} ```sql SELECT * @@ -201,13 +207,13 @@ Roles are used to define groups of users for certain privileges instead of manag └────┴─────────┴─────────┘ ``` -3. Log into the ClickHouse client using the `column_user` user +##### Log into the ClickHouse client using the `column_user` user {#login-column-user} ```bash clickhouse-client --user column_user --password password ``` -4. Test `SELECT` using all columns +##### Test `SELECT` using all columns {#test-select-all-columns} ```sql SELECT * @@ -230,7 +236,7 @@ Roles are used to define groups of users for certain privileges instead of manag Access is denied since all columns were specified and the user only has access to `id` and `column1` ::: -5. Verify `SELECT` query with only columns specified and allowed: +##### Verify `SELECT` query with only columns specified and allowed: {#verify-allowed-columns} ```sql SELECT @@ -250,15 +256,19 @@ Roles are used to define groups of users for certain privileges instead of manag └────┴─────────┘ ``` + + ### Testing role privileges with row restricted user {#testing-role-privileges-with-row-restricted-user} -1. Log into the ClickHouse client using `row_user` + + +##### Log into the ClickHouse client using `row_user` {#login-row-user} ```bash clickhouse-client --user row_user --password password ``` -2. View rows available +##### View rows available {#view-available-rows} ```sql SELECT * @@ -278,37 +288,41 @@ Roles are used to define groups of users for certain privileges instead of manag Verify that only the above two rows are returned, rows with the value `B` in `column1` should be excluded. ::: + + ## Modifying users and roles {#modifying-users-and-roles} Users can be assigned multiple roles for a combination of privileges needed. When using multiple roles, the system will combine the roles to determine privileges, the net effect will be that the role permissions will be cumulative. For example, if one `role1` allows for only select on `column1` and `role2` allows for select on `column1` and `column2` then the user will have access to both columns. -1. Using the admin account, create new user to restrict by both row and column with default roles + + +##### Using the admin account, create new user to restrict by both row and column with default roles {#create-restricted-user} ```sql CREATE USER row_and_column_user IDENTIFIED BY 'password' DEFAULT ROLE A_rows_users; ``` -2. Remove prior privileges for `A_rows_users` role +##### Remove prior privileges for `A_rows_users` role {#remove-prior-privileges} ```sql REVOKE SELECT(id, column1, column2) ON db1.table1 FROM A_rows_users; ``` -3. Allow `A_row_users` role to only select from `column1` +##### Allow `A_row_users` role to only select from `column1` {#allow-column1-select} ```sql GRANT SELECT(id, column1) ON db1.table1 TO A_rows_users; ``` -4. Log into the ClickHouse client using `row_and_column_user` +##### Log into the ClickHouse client using `row_and_column_user` {#login-restricted-user} ```bash clickhouse-client --user row_and_column_user --password password; ``` -5. Test with all columns: +##### Test with all columns: {#test-all-columns-restricted} ```sql SELECT * @@ -327,7 +341,7 @@ For example, if one `role1` allows for only select on `column1` and `role2` allo SELECT(id, column1, column2) ON db1.table1. (ACCESS_DENIED) ``` -6. Test with limited allowed columns: +##### Test with limited allowed columns: {#test-limited-columns} ```sql SELECT @@ -344,6 +358,7 @@ For example, if one `role1` allows for only select on `column1` and `role2` allo │ 2 │ A │ └────┴─────────┘ ``` + ## Troubleshooting {#troubleshooting} diff --git a/docs/cloud/_snippets/_security_table_of_contents.md b/docs/cloud/_snippets/_security_table_of_contents.md index 9ff837bb8a9..29b4f512c7b 100644 --- a/docs/cloud/_snippets/_security_table_of_contents.md +++ b/docs/cloud/_snippets/_security_table_of_contents.md @@ -1,8 +1,11 @@ -| Page | Description | -|---------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------| -| [Shared Responsibility Model](/cloud/security/shared-responsibility-model) | Understand how security responsibilities are divided between ClickHouse Cloud and your organization for different service types. | -| [Cloud Access Management](/cloud/security/cloud-access-management) | Manage user access with authentication, single sign-on (SSO), role-based permissions, and team invitations. | -| [Connectivity](/cloud/security/connectivity) | Configure secure network access including IP allow-lists, private networking, S3 data access, and Cloud IP address management. | -| [Enhanced Encryption](/cloud/security/cmek) | Learn about default AES 256 encryption and how to enable Transparent Data Encryption (TDE) for additional data protection at rest. | -| [Audit Logging](/cloud/security/audit-logging) | Set up and use audit logging to track and monitor activities in your ClickHouse Cloud environment. | -| [Privacy and Compliance](/cloud/security/privacy-compliance-overview) | Review security certifications, compliance standards, and learn how to manage your personal information and data rights. | \ No newline at end of file +| Page | Description | +|------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [ClickHouse Cloud Security Features](/cloud/security) | Details the security options and best practices available for ClickHouse organization and service protection. | +| [Cloud access management guides](/cloud/security/cloud_access_management) | This section contains step-by-step guides for managing access in ClickHouse Cloud. | +| [Setting IP filters](/cloud/security/setting-ip-filters) | A guide on how to create or modify an IP access list. | +| [Private networking](/cloud/security/connectivity/private-networking) | ClickHouse Cloud provides the ability to connect your services to your cloud virtual network. Refer to these guides for set up steps for your provider | +| [Data masking](/cloud/guides/data-masking) | Learn how you can mask data in ClickHouse. | +| [Data encryption](/cloud/security/cmek) | Learn how to enable Transparent Data Encryption as well as Customer Managed Encryption Keys. | +| [Audit logging](/cloud/security/audit_logging) | Guides on how to access and review audited events in the ClickHouse Cloud console, as well as sample logs and queries customers can use in developing their BYOC security program | +| [HIPAA onboarding](/cloud/security/compliance/hipaa-onboarding) | This page describes the process for enabling deployment of HIPAA compliant services in ClickHouse Cloud. | +| [PCI onboarding](/cloud/security/compliance/pci-onboarding) | This page describes the process for enabling deployment of PCI compliant services in ClickHouse Cloud. | diff --git a/docs/cloud/features/01_cloud_tiers.md b/docs/cloud/features/01_cloud_tiers.md index 782d9d70cb1..e0548da8326 100644 --- a/docs/cloud/features/01_cloud_tiers.md +++ b/docs/cloud/features/01_cloud_tiers.md @@ -173,7 +173,7 @@ You can upgrade to the Scale or Enterprise tier to scale their services. Designed for workloads requiring enhanced SLAs (2+ replica deployments), scalability, and advanced security. - Offers support for features such as: - - [Private networking support](/cloud/security/private-link-overview). + - [Private networking support](/cloud/security/connectivity/private-networking). - [Compute-compute separation](../reference/warehouses#what-is-compute-compute-separation). - [Flexible scaling](/manage/scaling) options (scale up/down, in/out). - [Configurable backups](/cloud/manage/backups/configurable-backups) diff --git a/docs/cloud/features/04_infrastructure/deployment-options.md b/docs/cloud/features/04_infrastructure/deployment-options.md new file mode 100644 index 00000000000..5be6398b835 --- /dev/null +++ b/docs/cloud/features/04_infrastructure/deployment-options.md @@ -0,0 +1,39 @@ +--- +title: 'Deployment Options' +slug: /infrastructure/deployment-options +description: 'Deployment options available for ClickHouse customers' +keywords: ['bring yor own cloud', 'byoc', 'private', 'government', 'self-deployed'] +doc_type: 'reference' +--- + +# ClickHouse Deployment Options + +ClickHouse provides a range of deployment options to cater to diverse customer requirements, offering varying degrees of control, compliance, and operational overhead. +This document outlines the distinct deployment types available, enabling users to select the optimal solution that aligns with their specific architectural preferences, regulatory obligations, and resource management strategies. + +## ClickHouse Cloud {#clickhouse-cloud} + +ClickHouse Cloud is a fully managed, cloud-native service that delivers the power and speed of ClickHouse without the operational complexities of self-management. +This option is ideal for users who prioritize rapid deployment, scalability, and minimal administrative overhead. +ClickHouse Cloud handles all aspects of infrastructure provisioning, scaling, maintenance, and updates, allowing users to focus entirely on data analysis and application development. +It offers consumption-based pricing, and automatic scaling, ensuring reliable and cost-effective performance for analytical workloads. It is available across AWS, GCP and Azure, with direct marketplace billing options. + +Learn more about [ClickHouse Cloud](/getting-started/quick-start/cloud). + +## Bring Your Own Cloud {#byoc} + +ClickHouse Bring Your Own Cloud (BYOC) allows organizations to deploy and manage ClickHouse within their own cloud environment while leveraging a managed service layer. This option bridges the gap between the fully managed experience of ClickHouse Cloud and the complete control of self-managed deployments. With ClickHouse BYOC, users retain control over their data, infrastructure, and security policies, meeting specific compliance and regulatory requirements, while offloading operational tasks like patching, monitoring, and scaling to the ClickHouse. This model offers the flexibility of a private cloud deployment with the benefits of a managed service, making it suitable for large-scale deployments at enterprises with stringent security, governance, and data residency needs. + +Learn more about [Bring Your Own Cloud](/cloud/reference/byoc). + +## ClickHouse Private {#clickhouse-private} + +ClickHouse Private is a self-deployed version of ClickHouse, leveraging the same proprietary technology that powers ClickHouse Cloud. This option delivers the highest degree of control, making it ideal for organizations with stringent compliance, networking, and security requirements, as well as for teams that possess the operational expertise to manage their own infrastructure. It benefits from regular updates and upgrades that are thoroughly tested in the ClickHouse Cloud environment, a feature-rich roadmap, and is backed by our expert support team. + +Learn more about [ClickHouse Private](/cloud/infrastructure/clickhouse-private). + +## ClickHouse Government {#clickhouse-government} + +ClickHouse Government is a self-deployed version of ClickHouse designed to meet the unique and rigorous demands of government agencies and public sector organizations that need isolated and accredited environments. This deployment option provides a highly secure, compliant, and isolated environment, focusing on FIPS 140-3 compliance utilizing OpenSSL, additional system hardening, and vulnerability management. It leverages the robust capabilities of ClickHouse Cloud while integrating specialized features and configurations to address the specific operational and security requirements of governmental entities. With ClickHouse Government, agencies can achieve high-performance analytics on sensitive data within a controlled and accredited infrastructure, backed by expert support tailored to public sector needs. + +Learn more about [ClickHouse Government](/cloud/infrastructure/clickhouse-government). \ No newline at end of file diff --git a/docs/cloud/features/05_admin_features/api/openapi.md b/docs/cloud/features/05_admin_features/api/openapi.md index 8ae99dcc940..c423e8ebdce 100644 --- a/docs/cloud/features/05_admin_features/api/openapi.md +++ b/docs/cloud/features/05_admin_features/api/openapi.md @@ -32,7 +32,7 @@ This document covers the ClickHouse Cloud API. For database API endpoints, pleas 3. To create an API key, specify the key name, permissions for the key, and expiration time, then click `Generate API Key`.
:::note -Permissions align with ClickHouse Cloud [predefined roles](/cloud/security/cloud-access-management/overview#console-users-and-roles). The developer role has read-only permissions for assigned services and the admin role has full read and write permissions. +Permissions align with ClickHouse Cloud [predefined roles](/cloud/security/console-roles). The developer role has read-only permissions for assigned services and the admin role has full read and write permissions. ::: Create API key form diff --git a/docs/cloud/features/06_security.md b/docs/cloud/features/06_security.md new file mode 100644 index 00000000000..627221243f8 --- /dev/null +++ b/docs/cloud/features/06_security.md @@ -0,0 +1,147 @@ +--- +sidebar_label: 'Security' +slug: /cloud/security +title: 'Security' +description: 'Learn more about securing ClickHouse Cloud and BYOC' +doc_type: 'reference' +--- + +# ClickHouse Cloud Security + +This document details the security options and best practices available for ClickHouse organization and service protection. +ClickHouse is dedicated to providing secure analytical database solutions; therefore, safeguarding data and service integrity is a priority. +The information herein covers various methods designed to assist users in securing their ClickHouse environments. + +## Cloud Console Authentication {#cloud-console-auth} + +### Password Authentication {#password-auth} + +ClickHouse Cloud console passwords are configured to NIST 800-63B standards with a minimum of 12 characters and 3 of 4 complexity requirements: upper case characters, lower case characters, numbers and/or special characters. + +Learn more about [password authentication](/cloud/security/manage-my-account#email-and-password). + +### Social Single Sign-On (SSO) {#social-sso} + +ClickHouse Cloud supports Google or Microsoft social authentication for single sign-on (SSO). + +Learn more about [social SSO](/cloud/security/manage-my-account#social-sso). + +### Multi-Factor Authentication {#mfa} + +Users using email and password or social SSO may also configure multi-factor authentication utilizing an authenticator app such as Authy or Google Authenticator. + +Learn more about [multi-factor authentication](/cloud/security/manage-my-account/#mfa). + +### Security Assertion Markup Language (SAML) Authentication {#saml-auth} + +Enterprise customers may configure SAML authentication. + +Learn more about [SAML authentication](/cloud/security/saml-setup). + +### API Authentication {#api-auth} + +Customers may configure API keys for use with OpenAPI, Terraform and Query API endpoints. + +Learn more about [API authentication](/cloud/manage/openapi). + +## Database Authentication {#database-auth} + +### Database Password Authentication {#db-password-auth} + +ClickHouse database user passwords are configured to NIST 800-63B standards with a minimum of 12 characters and complexity requirements: upper case characters, lower case characters, numbers and/or special characters. + +Learn more about [database password authentication](/cloud/security/manage-database-users#database-user-id--password). + +### Secure Shell (SSH) Database Authentication {#ssh-auth} + +ClickHouse database users may be configured to use SSH authentication. + +Learn more about [SSH authentication](/cloud/security/manage-database-users#database-ssh). + +## Access Control {#access-control} + +### Console Role-Based Access Control (RBAC) {#console-rbac} + +ClickHouse Cloud supports role assignment for organization, service and database permissions. Database permissions using this method are supported in SQL console only. + +Learn more about [console RBAC](/cloud/security/console-roles). + +### Database User Grants {#database-user-grants} + +ClickHouse databases support granular permission management and role-based access via user grants. + +Learn more about [database user grants](/cloud/security/manage-database-users#database-permissions). + +## Network Security {#network-security} + +### IP Filters {#ip-filters} + +Configure IP filters to limit inbound connections to your ClickHouse service. + +Learn more about [IP filters](/cloud/security/setting-ip-filters). + +### Private Connectivity {#private-connectivity} + +Connect to your ClickHouse clusters from AWS, GCP or Azure using private connectivity. + +Learn more about [private connectivity](/cloud/security/connectivity/private-networking). + +## Encryption {#encryption} + +### Storage Level Encryption {#storage-encryption} + +ClickHouse Cloud encrypts data at rest by default using cloud provider-managed AES 256 keys. + +Learn more about [storage encryption](/cloud/security/cmek#storage-encryption). + +### Transparent Data Encryption {#tde} + +In addition to storage encryption, ClickHouse Cloud Enterprise customers may enable database level transparent data encryption for additional protection. + +Learn more about [transparent data encryption](/cloud/security/cmek#transparent-data-encryption-tde). + +### Customer Managed Encryption Keys {#cmek} + +ClickHouse Cloud Enterprise customers may use their own key for database level encryption. + +Learn more about [customer managed encryption keys](/cloud/security/cmek#customer-managed-encryption-keys-cmek). + +## Auditing and Logging {#auditing-logging} + +### Console Audit Log {#console-audit-log} + +Activities within the console are logged. Logs are available for review and export. + +Learn more about [console audit logs](/cloud/security/audit-logging/console-audit-log). + +### Database Audit Logs {#database-audit-logs} + +Activities within the database are logged. Logs are available for review and export. + +Learn more about [database audit logs](/cloud/security/audit-logging/database-audit-log). + +### BYOC Security Playbook {#byoc-security-playbook} + +Sample detection queries for security teams managing ClickHouse BYOC instances. + +Learn more about the [BYOC security playbook](/cloud/security/audit-logging/byoc-security-playbook). + +## Compliance {#compliance} + +### Security and Compliance Reports {#compliance-reports} + +ClickHouse maintains a strong security and compliance program. Check back periodically for new third party audit reports. + +Learn more about [security and compliance reports](/cloud/security/compliance-overview). + +### HIPAA Compliant Services {#hipaa-compliance} + +ClickHouse Cloud Enterprise customers may deploy services housing protected health information (PHI) to HIPAA compliant regions after signing a Business Associate Agreement (BAA). + +Learn more about [HIPAA compliance](/cloud/security/compliance/hipaa-onboarding). + +### PCI Compliant Services {#pci-compliance} + +ClickHouse Cloud Enterprise customers may deploy services housing credit card information to PCI compliant regions. + +Learn more about [PCI compliance](/cloud/security/compliance/pci-onboarding). \ No newline at end of file diff --git a/docs/cloud/features/06_security/01_shared-responsibility-model.md b/docs/cloud/features/06_security/01_shared-responsibility-model.md deleted file mode 100644 index c9ed24a4bd3..00000000000 --- a/docs/cloud/features/06_security/01_shared-responsibility-model.md +++ /dev/null @@ -1,108 +0,0 @@ ---- -sidebar_label: 'Shared responsibility model' -slug: /cloud/security/shared-responsibility-model -title: 'Shared responsibility model' -description: 'Learn more about the security model of ClickHouse Cloud' -doc_type: 'reference' ---- - -## Service types {#service-types} - -ClickHouse Cloud offers three service types: Basic, Scale and Enterprise. For more information, review our [Service Types](/cloud/manage/cloud-tiers) page. - -## Cloud architecture {#cloud-architecture} - -The Cloud architecture consists of the control plane and the data plane. The control plane is responsible for organization creation, user management within the control plane, service management, API key management, and billing. The data plane runs tooling for orchestration and management, and houses customer services. For more information, review our [ClickHouse Cloud Architecture](/cloud/reference/architecture) diagram. - -## BYOC architecture {#byoc-architecture} - -Bring your own cloud (BYOC) enables customers to run the data plane in their own cloud account. For more information, review our [(BYOC) Bring Your Own Cloud](/cloud/reference/byoc) page. - -## ClickHouse Cloud shared responsibility model {#clickhouse-cloud-shared-responsibility-model} -The model below generally addresses ClickHouse responsibilities and shows responsibilities that should be addressed by customers of ClickHouse Cloud and ClickHouse BYOC, respectively. For more information on our PCI shared responsibility model, please download a copy of the overview available in our [Trust Center](https://trust.clickhouse.com). - -| Control | ClickHouse | Cloud Customer | BYOC Customer | -|-----------------------------------------------------------------------|--------------------|---------------------|---------------------| -| Maintain separation of environments | :white_check_mark: | | :white_check_mark: | -| Manage network settings | :white_check_mark: | :white_check_mark: | :white_check_mark: | -| Securely manage access to ClickHouse systems | :white_check_mark: | | | -| Securely manage organizational users in control plane and databases | | :white_check_mark: | :white_check_mark: | -| User management and audit | :white_check_mark: | :white_check_mark: | :white_check_mark: | -| Encrypt data in transit and at rest | :white_check_mark: | | | -| Securely handle customer managed encryption keys | | :white_check_mark: | :white_check_mark: | -| Provide redundant infrastructure | :white_check_mark: | | :white_check_mark: | -| Backup data | :white_check_mark: | :white_check_mark: | :white_check_mark: | -| Verify backup recovery capabilities | :white_check_mark: | :white_check_mark: | :white_check_mark: | -| Implement data retention settings | | :white_check_mark: | :white_check_mark: | -| Security configuration management | :white_check_mark: | | :white_check_mark: | -| Software and infrastructure vulnerability remediation | :white_check_mark: | | | -| Perform penetration tests | :white_check_mark: | | | -| Threat detection and response | :white_check_mark: | | :white_check_mark: | -| Security incident response | :white_check_mark: | | :white_check_mark: | - -## ClickHouse Cloud configurable security features {#clickhouse-cloud-configurable-security-features} - -
- Network connectivity - - | Setting | Status | Cloud | Service level | - |------------------------------------------------------------------------------------------------------|-----------|-------------------|----------------------| - | [IP filters](/cloud/security/setting-ip-filters) to restrict connections to services | Available | AWS, GCP, Azure | All | - | [Private link](/cloud/security/private-link-overview) to securely connect to services | Available | AWS, GCP, Azure | Scale or Enterprise | - -
-
- Access management - - | Setting | Status | Cloud | Service level | - |------------------------------------------------------------------------------------------------------|-----------|-------------------|-------------------------| - | [Standard role-based access](/cloud/security/cloud-access-management) in control plane | Available | AWS, GCP, Azure | All | - | [Multi-factor authentication (MFA)](/cloud/security/cloud-authentication#multi-factor-authentication) available | Available | AWS, GCP, Azure | All | - | [SAML Single Sign-On](/cloud/security/saml-setup) to control plane available | Preview | AWS, GCP, Azure | Enterprise | - | Granular [role-based access control](/cloud/security/cloud-access-management/overview#database-permissions) in databases | Available | AWS, GCP, Azure | All | - -
-
- Data security - - | Setting | Status | Cloud | Service level | - |------------------------------------------------------------------------------------------------------|-----------|-------------------|-------------------------| - | [Cloud provider and region](/cloud/reference/supported-regions) selections | Available | AWS, GCP, Azure | All | - | Limited [free daily backups](/cloud/manage/backups/overview#default-backup-policy) | Available | AWS, GCP, Azure | All | - | [Custom backup configurations](/cloud/manage/backups/overview#configurable-backups) available | Available | GCP, AWS, Azure | Scale or Enterprise | - | [Customer managed encryption keys (CMEK)](/cloud/security/cmek) for transparent
data encryption available | Available | AWS, GCP | Enterprise | - | [Field level encryption](/sql-reference/functions/encryption-functions) with manual key management for granular encryption | Available | GCP, AWS, Azure | All | - -
-
- Data retention - - | Setting | Status | Cloud | Service level | - |------------------------------------------------------------------------------------------------------|-----------|-------------------|-------------------------| - | [Time to live (TTL)](/sql-reference/statements/alter/ttl) settings to manage retention | Available | AWS, GCP, Azure | All | - | [ALTER TABLE DELETE](/sql-reference/statements/alter/delete) for heavy deletion actions | Available | AWS, GCP, Azure | All | - | [Lightweight DELETE](/sql-reference/statements/delete) for measured deletion activities | Available | AWS, GCP, Azure | All | - -
-
- Auditing and logging - - | Setting | Status | Cloud | Service level | - |------------------------------------------------------------------------------------------------------|-----------|-------------------|-------------------------| - | [Audit log](/cloud/security/audit-logging) for control plane activities | Available | AWS, GCP, Azure | All | - | [Session log](/operations/system-tables/session_log) for database activities | Available | AWS, GCP, Azure | All | - | [Query log](/operations/system-tables/query_log) for database activities | Available | AWS, GCP, Azure | All | - -
- -## ClickHouse Cloud compliance {#clickhouse-cloud-compliance} - - | Framework | Status | Cloud | Service level | - |------------------------------------------------------------------------------------------------------|-----------|-------------------|-------------------------| - | ISO 27001 compliance | Available | AWS, GCP, Azure | All | - | SOC 2 Type II compliance | Available | AWS, GCP, Azure | All | - | GDPR and CCPA compliance | Available | AWS, GCP, Azure | All | - | HIPAA compliance | Available | AWS, GCP | Enterprise | - | PCI compliance | Available | AWS | Enterprise | - - For more information on supported compliance frameworks, please review our [Security and Compliance](/cloud/security/compliance-overview) page. diff --git a/docs/cloud/features/06_security/02_cloud-access-management/cloud-authentication.md b/docs/cloud/features/06_security/02_cloud-access-management/cloud-authentication.md deleted file mode 100644 index 802fedba30b..00000000000 --- a/docs/cloud/features/06_security/02_cloud-access-management/cloud-authentication.md +++ /dev/null @@ -1,131 +0,0 @@ ---- -sidebar_label: 'Cloud Authentication' -slug: /cloud/security/cloud-authentication -title: 'Cloud Authentication' -description: 'This guide explains some good practices for configuring your authentication.' -doc_type: 'guide' ---- - -import ScalePlanFeatureBadge from '@theme/badges/ScalePlanFeatureBadge' -import EnterprisePlanFeatureBadge from '@theme/badges/EnterprisePlanFeatureBadge' - -# Cloud authentication - -ClickHouse Cloud provides a number of ways to authenticate. This guide explains some good practices for configuring your authentication. Always check with your security team when selecting authentication methods. - -## Password settings {#password-settings} - -Minimum password settings for our console and services (databases) currently comply with [NIST 800-63B](https://pages.nist.gov/800-63-3/sp800-63b.html#sec4) Authenticator Assurance Level 1: -- Minimum 12 characters -- Includes 3 of the following 4 items: - - 1 uppercase letter - - 1 lowercase letter - - 1 number - - 1 special character - -## Email and password {#email--password} - -ClickHouse Cloud allows you to authenticate with an email address and password. When using this method the best way to protect your ClickHouse account use a strong password. There are many online resources to help you devise a password you can remember. Alternatively, you can use a random password generator and store your password in a password manager for increased security. - -## SSO using Google or Microsoft social authentication {#sso-using-google-or-microsoft-social-authentication} - -If your company uses Google Workspace or Microsoft 365, you can leverage your current single sign-on setup within ClickHouse Cloud. To do this, simply sign up using your company email address and invite other users using their company email. The effect is that your users must login using your company's login flows, whether via your identity provider or directly through Google or Microsoft authentication, before they can authenticate into ClickHouse Cloud. - -## Multi-factor authentication {#multi-factor-authentication} - -Users with email + password or social authentication can further secure their account using multi-factor authentication (MFA). To set up MFA: -1. Log into [console.clickhouse.cloud](https://console.clickhouse.cloud/) -2. Click your initials in the upper left corner next to the ClickHouse logo -3. Select Profile -4. Select Security on the left -5. Click Set up in the Authenticator app tile -6. Use an authenticator app such as Authy, 1Password or Google Authenticator to scan the QR code -7. Enter the code to confirm -8. On the next screen, copy the recovery code and store it in a safe place -9. Check the box next to `I have safely recorded this code` -10. Click Continue - -## Account recovery {#account-recovery} - -
- Obtain recovery code - - If you previously enrolled in MFA and either did not create or misplaced your recovery code, follow these steps to get a new recovery code: - 1. Go to https://console.clickhouse.cloud - 2. Sign in with your credentials and MFA - 3. Go to your profile in the upper left corner - 4. Click Security on the left - 5. Click the trash can next to your Authenticator app - 6. Click Remove authenticator app - 7. Enter your code and click Continue - 8. Click Set up in the Authenticator app section - 9. Scan the QR code and input the new code - 10. Copy your recovery code and store it in a safe place - 11. Check the box next to `I have safely recorded this code` - 12. Click Continue - -
-
- Forgot password - - If you forgot your password, follow these steps for self-service recovery: - 1. Go to https://console.clickhouse.cloud - 2. Enter your email address and click Continue - 3. Click Forgot your password? - 4. Click Send password reset link - 5. Check your email and click Reset password from the email - 6. Enter your new password, confirm the password and click Update password - 7. Click Back to sign in - 8. Sign in normally with your new password - -
-
- Lost MFA device or token - - If you lost your MFA device or deleted your token, follow these steps to recover and create a new token: - 1. Go to https://console.clickhouse.cloud - 2. Enter your credentials and click Continue - 3. On the Multi-factor authentication screen click Cancel - 4. Click Recovery code - 5. Enter the code and press Continue - 6. Copy the new recovery code and store it somewhere safe - 7. Click the box next to `I have safely recorded this code` and click continue - 8. Once signed in, go to your profile in the upper left - 9. Click on security in the upper left - 10. Click the trash can icon next to Authenticator app to remove your old authenticator - 11. Click Remove authenticator app - 12. When prompted for your Multi-factor authentication, click Cancel - 13. Click Recovery code - 14. Enter your recovery code (this is the new code generated in step 7) and click Continue - 15. Copy the new recovery code and store it somewhere safe - this is a fail safe in case you leave the screen during the removal process - 16. Click the box next to `I have safely recorded this code` and click Continue - 17. Follow the process above to set up a new MFA factor - -
-
- Lost MFA and recovery code - - If you lost your MFA device AND recovery code or you lost your MFA device and never obtained a recovery code, follow these steps to request a reset: - - **Submit a ticket**: If you are in an organization that has other administrative users, even if you are attempting to access a single user organization, ask a member of your organization assigned the Admin role to log into the organization and submit a support ticket to reset your MFA on your behalf. Once we verify the request is authenticated, we will reset your MFA and notify the Admin. Sign in as usual without MFA and go to your profile settings to enroll a new factor if you wish. - - **Reset via email**: If you are the only user in the organization, submit a support case via email (support@clickhouse.com) using the email address associated with your account. Once we verify the request is coming from the correct email, we will reset your MFA AND password. Access your email to access the password reset link. Set up a new password then go to your profile settings to enroll a new factor if you wish. - -
- -## SAML SSO {#saml-sso} - - - -ClickHouse Cloud also supports security assertion markup language (SAML) single sign on (SSO). For more information, see [SAML SSO Setup](/cloud/security/saml-setup). - -## Database user ID and password {#database-user-id--password} - -Use the SHA256_hash method when [creating user accounts](/sql-reference/statements/create/user.md) to secure passwords. - -**TIP:** Since users with less than administrative privileges cannot set their own password, ask the user to hash their password using a generator -such as [this one](https://tools.keycdn.com/sha256-online-generator) before providing it to the admin to setup the account. Passwords should follow the [requirements](#password-settings) listed above. - -```sql -CREATE USER userName IDENTIFIED WITH sha256_hash BY 'hash'; -``` diff --git a/docs/cloud/features/06_security/02_cloud-access-management/index.md b/docs/cloud/features/06_security/02_cloud-access-management/index.md deleted file mode 100644 index b0fa9a8ffff..00000000000 --- a/docs/cloud/features/06_security/02_cloud-access-management/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -slug: /cloud/security/cloud-access-management -title: 'Cloud access management' -description: 'Cloud Access Management Table Of Contents' -doc_type: 'landing-page' ---- - -| Page | Description | -|----------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------| -| [Overview](/cloud/security/cloud-access-management/overview) | Overview of access control in ClickHouse Cloud | -| [Cloud Authentication](/cloud/security/cloud-authentication) | A guide which explores some good practices for configuring your authentication | diff --git a/docs/cloud/features/06_security/03_connectivity/01_private-link-overview.md b/docs/cloud/features/06_security/03_connectivity/01_private-link-overview.md deleted file mode 100644 index 4744e5a78f4..00000000000 --- a/docs/cloud/features/06_security/03_connectivity/01_private-link-overview.md +++ /dev/null @@ -1,17 +0,0 @@ ---- -sidebar_label: 'Private link overview' -slug: /cloud/security/private-link-overview -title: 'Private link overview' -description: 'Landing page for private link' -doc_type: 'landing-page' ---- - -# Private link overview - -ClickHouse Cloud provides the ability to connect your services to your cloud virtual network. - -Refer to the guides below for set up steps for your provider: - -- [AWS private Link](/manage/security/aws-privatelink) -- [GCP private service connect](/manage/security/gcp-private-service-connect) -- [Azure private link](/cloud/security/azure-privatelink) diff --git a/docs/cloud/features/06_security/03_connectivity/_category_.json b/docs/cloud/features/06_security/03_connectivity/_category_.json deleted file mode 100644 index f825cb0a36a..00000000000 --- a/docs/cloud/features/06_security/03_connectivity/_category_.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "label": "Connectivity", - "collapsible": true, - "collapsed": true, - "link": { "type": "doc", "id": "cloud/features/security/connectivity/index" } -} \ No newline at end of file diff --git a/docs/cloud/features/06_security/03_connectivity/index.md b/docs/cloud/features/06_security/03_connectivity/index.md deleted file mode 100644 index f347256dd00..00000000000 --- a/docs/cloud/features/06_security/03_connectivity/index.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -slug: /cloud/security/connectivity -title: 'connectivity overview' -description: 'Landing page for Connectivity' -doc_type: 'landing-page' ---- - -# Connectivity - -This section looks at connectivity and contains the following pages: - -| Page | Description | -|----------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------| -| [Private Networking](/cloud/security/private-link-overview) | Information on how to connect your services to your cloud virtual network. | diff --git a/docs/cloud/features/06_security/_category_.json b/docs/cloud/features/06_security/_category_.json deleted file mode 100644 index aed26fa7f7a..00000000000 --- a/docs/cloud/features/06_security/_category_.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "label": "Security", - "collapsible": true, - "collapsed": true, -} \ No newline at end of file diff --git a/docs/cloud/guides/SQL_console/_category_.json b/docs/cloud/guides/SQL_console/_category_.json deleted file mode 100644 index 07e636bd5ed..00000000000 --- a/docs/cloud/guides/SQL_console/_category_.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "label": "SQL console", - "collapsible": true, - "collapsed": true, -} \ No newline at end of file diff --git a/docs/cloud/guides/SQL_console/configure_org_and_service_role_assignments.md b/docs/cloud/guides/SQL_console/configure_org_and_service_role_assignments.md deleted file mode 100644 index 52671cad152..00000000000 --- a/docs/cloud/guides/SQL_console/configure_org_and_service_role_assignments.md +++ /dev/null @@ -1,85 +0,0 @@ ---- -slug: /cloud/guides/sql-console/configure-org-service-role-assignments -sidebar_label: 'Configuring organization and service role assignments' -title: 'Configuring organization and service role assignments within the console' -description: 'Guide showing how to configure org and service role assignments within the console' -doc_type: 'guide' ---- - -import Image from '@theme/IdealImage'; -import step_1 from '@site/static/images/cloud/guides/sql_console/org_level_access/1_org_settings.png' -import step_2 from '@site/static/images/cloud/guides/sql_console/org_level_access/2_org_settings.png' -import step_3 from '@site/static/images/cloud/guides/sql_console/org_level_access/3_org_settings.png' -import step_4 from '@site/static/images/cloud/guides/sql_console/org_level_access/4_org_settings.png' -import step_5 from '@site/static/images/cloud/guides/sql_console/org_level_access/5_org_settings.png' -import step_6 from '@site/static/images/cloud/guides/sql_console/org_level_access/6_org_settings.png' -import step_7 from '@site/static/images/cloud/guides/sql_console/org_level_access/7_org_settings.png' - -# Configuring organization and service role assignments within the console - -> This guide shows you how to configure role assignments at the organization and service level. - - - -## Access organization settings {#access-service-settings} - -From the services page, select the name of your organization: - - - -Select the `Users and roles` menu item from the popup menu. - - - -## Adjust access per user {#access-per-user} - -Select the menu item at the end of the row for the user that you which to modify -access for: - - - -Select `edit`: - - - -A tab will display on the right hand side of the page: - - - -Select the drop-down menu items to adjust console-wide access permissions and which features a user can access from within the ClickHouse console. -This manages high-level access and administrative settings for an organization: - -| Role | Description | -|-------------|----------------------------------------------------------------------------------| -| `Admin` | Perform all administrative activities for an organization, control all settings. | -| `Developer` | View everything except Services, create API keys with equal or lower access. | -| `Member` | Sign in only with ability to manage personal profile settings. | -| `Billing` | View usage and invoices, and manage payment methods | - -Select the drop-down menu items to adjust the access scope of the service role of the selected user. -This defines security and operational settings for individual services: - -| Access scope | -|---------------------| -| `All services` | -| `Specific services` | -| `No services` | - -When selecting `Specific services`, you can control the role of the user per -service: - - - -You can choose from the following roles: - -| Role | Description | -|-------------|--------------------------------------------------------------------| -| `Admin` | Full control over configuration and security. Can delete service. | -| `Read-only` | Can see service data and security settings. Can't modify anything. | -| `No access` | Doesn't know the service exists. | - -Save your changes with the `Save changes` button at the bottom of the tab: - - - - \ No newline at end of file diff --git a/docs/cloud/guides/best_practices/_category_.json b/docs/cloud/guides/best_practices/_category_.json index 21f95c55bca..635441db7e9 100644 --- a/docs/cloud/guides/best_practices/_category_.json +++ b/docs/cloud/guides/best_practices/_category_.json @@ -1,5 +1,5 @@ { "label": "Best practices", "collapsible": true, - "collapsed": true, + "collapsed": true } \ No newline at end of file diff --git a/docs/cloud/guides/security/connectivity/cloud-endpoints-api.md b/docs/cloud/guides/data_sources/01_cloud-endpoints-api.md similarity index 95% rename from docs/cloud/guides/security/connectivity/cloud-endpoints-api.md rename to docs/cloud/guides/data_sources/01_cloud-endpoints-api.md index babdc8c3688..7886c88fe95 100644 --- a/docs/cloud/guides/security/connectivity/cloud-endpoints-api.md +++ b/docs/cloud/guides/data_sources/01_cloud-endpoints-api.md @@ -1,7 +1,7 @@ --- -slug: /manage/security/cloud-endpoints-api -sidebar_label: 'Cloud IP Addresses' -title: 'Cloud IP Addresses' +slug: /manage/data-sources/cloud-endpoints-api +sidebar_label: 'Cloud IP addresses' +title: 'Cloud IP addresses' description: 'This page documents the Cloud Endpoints API security features within ClickHouse. It details how to secure your ClickHouse deployments by managing access through authentication and authorization mechanisms.' doc_type: 'reference' --- diff --git a/docs/cloud/guides/security/connectivity/accessing-s3-data-securely.md b/docs/cloud/guides/data_sources/02_accessing-s3-data-securely.md similarity index 98% rename from docs/cloud/guides/security/connectivity/accessing-s3-data-securely.md rename to docs/cloud/guides/data_sources/02_accessing-s3-data-securely.md index 85e2facfd61..a33a0d3a9e8 100644 --- a/docs/cloud/guides/security/connectivity/accessing-s3-data-securely.md +++ b/docs/cloud/guides/data_sources/02_accessing-s3-data-securely.md @@ -1,7 +1,7 @@ --- -slug: /cloud/security/secure-s3 -sidebar_label: 'Accessing S3 Data Securely' -title: 'Accessing S3 Data Securely' +slug: /cloud/data-sources/secure-s3 +sidebar_label: 'Accessing S3 data securely' +title: 'Accessing S3 data securely' description: 'This article demonstrates how ClickHouse Cloud customers can leverage role-based access to authenticate with Amazon Simple Storage Service(S3) and access their data securely.' doc_type: 'guide' --- diff --git a/docs/cloud/guides/data_sources/_category_.json b/docs/cloud/guides/data_sources/_category_.json new file mode 100644 index 00000000000..3810dd5bc80 --- /dev/null +++ b/docs/cloud/guides/data_sources/_category_.json @@ -0,0 +1,5 @@ +{ + "label": "Data sources", + "collapsible": true, + "collapsed": true +} \ No newline at end of file diff --git a/docs/cloud/guides/data_sources/index.md b/docs/cloud/guides/data_sources/index.md new file mode 100644 index 00000000000..8290d417670 --- /dev/null +++ b/docs/cloud/guides/data_sources/index.md @@ -0,0 +1,26 @@ +--- +slug: /cloud/guides/data-sources +title: 'Data sources' +hide_title: true +description: 'Table of contents page for the ClickHouse Cloud guides section' +doc_type: 'landing-page' +--- + +## Cloud integrations {#cloud-integrations} + +This section contains guides and references for integrating ClickHouse Cloud with external data sources that require additional configuration. + +| Page | Description | +|-----------------------------------------------------------------|-------------------------------------------------------------------------| +| [Cloud IP addresses](/manage/data-sources/cloud-endpoints-api) | Networking information needed for some table functions and connections | +| [Accessing S3 data securely](/cloud/data-sources/secure-s3) | Access external data sources in AWS S3 using role based access | + +## Additional connections for external data sources {#additional-connections-for-external-data-sources} + +### ClickPipes for data ingestion {#clickpipes-for-data-ingestion} + +ClickPipes allow customers to easily integrate streaming data from a number of sources. Refer to [ClickPipes](/integrations/clickpipes) in our Integrations documentation for additional information. + +### Table functions as external data sources {#table-functions-as-external-data-sources} + +ClickHouse supports a number of table functions to access external data sources. For more information, refer to [table functions](/sql-reference/table-functions) in the SQL reference section. diff --git a/docs/cloud/features/04_infrastructure/byoc.md b/docs/cloud/guides/infrastructure/01_deployment_options/01_byoc.md similarity index 100% rename from docs/cloud/features/04_infrastructure/byoc.md rename to docs/cloud/guides/infrastructure/01_deployment_options/01_byoc.md diff --git a/docs/cloud/guides/infrastructure/01_deployment_options/02_clickhouse-private.md b/docs/cloud/guides/infrastructure/01_deployment_options/02_clickhouse-private.md new file mode 100644 index 00000000000..399233d5132 --- /dev/null +++ b/docs/cloud/guides/infrastructure/01_deployment_options/02_clickhouse-private.md @@ -0,0 +1,82 @@ +--- +title: 'ClickHouse Private' +slug: /cloud/infrastructure/clickhouse-private +keywords: ['private', 'on-prem'] +description: 'Overview of ClickHouse Private offering' +doc_type: 'reference' +--- + +import Image from '@theme/IdealImage'; +import private_gov_architecture from '@site/static/images/cloud/reference/private-gov-architecture.png'; + +## Overview {#overview} + +ClickHouse Private is a self-deployed package consisting of the same proprietary version of ClickHouse that runs on ClickHouse Cloud and our ClickHouse Operator, configured for separation of compute and storage. It is deployed to Kubernetes environments with S3 compatible storage. + +This package is currently available for AWS and IBM Cloud, with bare metal deployments coming soon. + +:::note Note +ClickHouse Private is designed for large enterprises with the most rigorous compliance requirements, providing full control and management over their dedicated infrastructure. This option is only available by [contacting us](https://clickhouse.com/company/contact?loc=nav). +::: + +## Benefits over open-source {#benefits-over-os} + +The following features differentiate ClickHouse Private from self-managed open source deployments: + + + +### Enhanced performance {#enhanced-performance} +- Native separation of compute and storage +- Proprietary cloud features such as [shared merge tree](/cloud/reference/shared-merge-tree) and [warehouse](/cloud/reference/warehouses) functionality + +### Tested and proven through a variety of use cases and conditions {#tested-proven-through-variety-of-use-cases} +- Fully tested and validated in ClickHouse Cloud + +### Full featured roadmap with new features added regularly {#full-featured-roadmap} +Additional features that are coming soon include: +- API to programmatically manage resources + - Automated backups + - Automated vertical scaling operations +- Identity provider integration + + + +## Architecture {#architecture} + +ClickHouse Private is fully self-contained within your deployment environment and consists of compute managed within Kubernetes and storage within an S3 compatible storage solution. + +
+ +ClickHouse Private Architecture + +
+ +## Onboarding process {#onboarding-process} + +Customers can initiate onboarding by reaching out to [us](https://clickhouse.com/company/contact?loc=nav). For qualified customers, we will provide a detailed environment build guide and access to the images and Helm charts for deployment. + +## General requirements {#general-requirements} + +This section is intended to provide an overview of the resources required to deploy ClickHouse Private. Specific deployment guides are provided as part of onboarding. Instance/ server types and sizes depend on the use case. + +### ClickHouse Private on AWS {#clickhouse-private-aws} + +Required resources: +- [ECR](https://docs.aws.amazon.com/ecr/) to receive the images and Helm charts +- [EKS](https://docs.aws.amazon.com/eks/) cluster with [CNI](https://github.com/aws/amazon-vpc-cni-k8s), [EBS CSI Driver](https://github.com/kubernetes-sigs/aws-ebs-csi-driver), [DNS](https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html), [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md), [IMDS](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html) for authentication and an [OIDC](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) provider +- Server nodes run Amazon Linux +- Operator requires an x86 node group +- An S3 bucket in the same region as the EKS cluster +- If ingress is required, also configure an NLB +- One AWS role per ClickHouse cluster for clickhouse-server/keeper operations + +### ClickHouse Private on IBM Cloud {#clickhouse-private-ibm-cloud} + +Required resources: +- [Container Registry](https://cloud.ibm.com/docs/Registry?topic=Registry-getting-started) to receive the images and Helm charts +- [Cloud Kubernetes Service](https://cloud.ibm.com/docs/containers?topic=containers-getting-started) with [CNI](https://www.ibm.com/docs/en/cloud-private/3.2.x?topic=networking-kubernetes-network-model), [Cloud Block Storage for VPC](https://cloud.ibm.com/docs/containers?topic=containers-vpc-block), [Cloud DNS](https://www.ibm.com/products/dns), and [Cluster Autoscaler](https://cloud.ibm.com/docs/containers?topic=containers-cluster-scaling-install-addon-enable) +- Server nodes run Ubuntu +- Operator requires an x86 node group +- [Cloud Object Storage](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-getting-started-cloud-object-storage) in the same region as the Cloud Kubernetes Service cluster +- If ingress is required, also configure an NLB +- One service account per ClickHouse cluster for clickhouse-server/keeper operations diff --git a/docs/cloud/guides/infrastructure/01_deployment_options/03_clickhouse-government.md b/docs/cloud/guides/infrastructure/01_deployment_options/03_clickhouse-government.md new file mode 100644 index 00000000000..7e21939a23f --- /dev/null +++ b/docs/cloud/guides/infrastructure/01_deployment_options/03_clickhouse-government.md @@ -0,0 +1,75 @@ +--- +title: 'ClickHouse Government' +slug: /cloud/infrastructure/clickhouse-government +keywords: ['government', 'fips', 'fedramp', 'gov cloud'] +description: 'Overview of ClickHouse Government offering' +doc_type: 'reference' +--- + +import Image from '@theme/IdealImage'; +import private_gov_architecture from '@site/static/images/cloud/reference/private-gov-architecture.png'; + +## Overview {#overview} + +ClickHouse Government is a self-deployed package consisting of the same proprietary version of ClickHouse that runs on ClickHouse Cloud and our ClickHouse Operator, configured for separation of compute and storage and hardened to meet the rigorous demands of government agencies and public sector organizations. It is deployed to Kubernetes environments with S3 compatible storage. + +This package is currently available for AWS, with bare metal deployments coming soon. + +:::note Note +ClickHouse Government is designed for government agencies, public sector organizations, or cloud software companies selling to these agencies and organizations, providing full control and management over their dedicated infrastructure. This option is only available by [contacting us](https://clickhouse.com/government). +::: + +## Benefits over open-source {#benefits-over-os} + +The following features differentiate ClickHouse Government from self-managed open source deployments: + + + +### Enhanced performance {#enhanced-performance} +- Native separation of compute and storage +- Proprietary cloud features such as [shared merge tree](/cloud/reference/shared-merge-tree) and [warehouse](/cloud/reference/warehouses) functionality + +### Tested and proven through a variety of use cases and conditions {#tested-proven} +- Fully tested and validated in ClickHouse Cloud + +### Compliance package {#compliance-package} +- [NIST Risk Management Framework (RMF)](https://csrc.nist.gov/projects/risk-management/about-rmf) documentation to accelerate your Authorization to Operate (ATO) + +### Full featured roadmap with new features added regularly {#full-featured-roadmap} +Additional features that are coming soon include: +- API to programmatically manage resources + - Automated backups + - Automated vertical scaling operations +- Identity provider integration + + + +## Architecture {#architecture} + +ClickHouse Government is fully self-contained within your deployment environment and consists of compute managed within Kubernetes and storage within an S3 compatible storage solution. + +
+ +ClickHouse Government Architecture + +
+ +## Onboarding process {#onboarding-process} + +Customers can initiate onboarding by reaching out to [us](https://clickhouse.com/government). For qualified customers, we will provide a detailed environment build guide and access to the images and Helm charts for deployment. + +## General requirements {#general-requirements} + +This section is intended to provide an overview of the resources required to deploy ClickHouse Government. Specific deployment guides are provided as part of onboarding. Instance/server types and sizes depend on the use case. + +### ClickHouse Government on AWS {#clickhouse-government-aws} + +Required resources: +- [ECR](https://docs.aws.amazon.com/ecr/) to receive the images and Helm charts +- Certificate Authority capable of generating FIPS compliant certificates +- [EKS](https://docs.aws.amazon.com/eks/) cluster with [CNI](https://github.com/aws/amazon-vpc-cni-k8s), [EBS CSI Driver](https://github.com/kubernetes-sigs/aws-ebs-csi-driver), [DNS](https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html), [Cluster Autoscaler](https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md), [IMDS](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html) for authentication and an [OIDC](https://docs.aws.amazon.com/eks/latest/userguide/enable-iam-roles-for-service-accounts.html) provider +- Server nodes run Amazon Linux +- Operator requires an x86 node group +- An S3 bucket in the same region as the EKS cluster +- If ingress is required, also configure an NLB +- One AWS role per ClickHouse cluster for clickhouse-server/keeper operations diff --git a/docs/cloud/guides/infrastructure/01_deployment_options/_category_.json b/docs/cloud/guides/infrastructure/01_deployment_options/_category_.json new file mode 100644 index 00000000000..cdb3db52211 --- /dev/null +++ b/docs/cloud/guides/infrastructure/01_deployment_options/_category_.json @@ -0,0 +1,5 @@ +{ + "label": "Deployment options", + "collapsible": true, + "collapsed": true +} \ No newline at end of file diff --git a/docs/cloud/guides/infrastructure/_category_.json b/docs/cloud/guides/infrastructure/_category_.json new file mode 100644 index 00000000000..fc224829596 --- /dev/null +++ b/docs/cloud/guides/infrastructure/_category_.json @@ -0,0 +1,5 @@ +{ + "label": "Infrastructure", + "collapsible": true, + "collapsed": true +} \ No newline at end of file diff --git a/docs/cloud/guides/security/01_cloud_access_management/01_manage-my-account.md b/docs/cloud/guides/security/01_cloud_access_management/01_manage-my-account.md new file mode 100644 index 00000000000..32b37f8b7ca --- /dev/null +++ b/docs/cloud/guides/security/01_cloud_access_management/01_manage-my-account.md @@ -0,0 +1,111 @@ +--- +sidebar_label: 'Manage my account' +slug: /cloud/security/manage-my-account +title: 'Manage my account' +description: 'This page describes how users can accept invitations, manage MFA settings, and reset passwords' +doc_type: 'guide' +--- + +import EnterprisePlanFeatureBadge from '@theme/badges/EnterprisePlanFeatureBadge' + +## Accept an invitation {#accept-invitation} + +Users may use multiple methods to accept an invitation to join an organization. If this is your first invitation, select the appropriate authentication method for your organization below. + +If this is not your first organization, either sign in with your existing organization then accept the invitation from the lower left hand side of the page OR accept the invitation from your email and sign in with your existing account. + +:::note SAML Users +Organizations using SAML have a unique login per ClickHouse organization. Use the direct link provided by your administrator to log in. +::: + +### Email and password {#email-and-password} + +ClickHouse Cloud allows you to authenticate with an email address and password. When using this method the best way to protect your ClickHouse account is to use a strong password. There are many online resources to help you devise a password you can remember. Alternatively, you can use a random password generator and store your password in a password manager for increased security. + +Passwords must contain a minimum of 12 characters and meet 3 of 4 complexity requirements: upper case characters, lower case characters, numbers and/or special characters. + +### Social single sign-on (SSO) {#social-sso} + +Use `Continue with Google` or `Continue with Microsoft Account` to sign up for services or accept invitations. + +If your company uses Google Workspace or Microsoft 365, you can leverage your current single sign-on setup within ClickHouse Cloud. To do this, simply sign up using your company email address and invite other users using their company email. The effect is that your users must login using your company's login flows, whether via your identity provider or directly through Google or Microsoft authentication, before they can authenticate into ClickHouse Cloud. + +### SAML single sign-on (SSO) {#saml-sso} + + + +Users using SAML SSO are automatically added by their identity provider upon sign in. ClickHouse Cloud users with the Organization Admin role may [manage roles](/cloud/security/manage-cloud-users) assigned to SAML users and enforce SAML as the only authentication method. + +## Manage multi-factor authentication (MFA) {#mfa} + +Users with email + password or social authentication can further secure their account using multi-factor authentication (MFA). To set up MFA: + +1. Log into [console.clickhouse.cloud](https://console.clickhouse.cloud/) +2. Click your initials in the upper left corner next to the ClickHouse logo +3. Select Profile +4. Select Security on the left +5. Click Set up in the Authenticator app tile +6. Use an authenticator app such as Authy, 1Password or Google Authenticator to scan the QR code +7. Enter the code to confirm +8. On the next screen, copy the recovery code and store it in a safe place +9. Check the box next to `I have safely recorded this code` +10. Click Continue + +### Obtain a new recovery code {#obtain-recovery-code} + +If you previously enrolled in MFA and either did not create or misplaced your recovery code, follow these steps to get a new recovery code: +1. Go to https://console.clickhouse.cloud +2. Sign in with your credentials and MFA +3. Go to your profile in the upper left corner +4. Click Security on the left +5. Click the trash can next to your Authenticator app +6. Click Remove authenticator app +7. Enter your code and click Continue +8. Click Set up in the Authenticator app section +9. Scan the QR code and input the new code +10. Copy your recovery code and store it in a safe place +11. Check the box next to `I have safely recorded this code` +12. Click Continue + +## Account recovery {#account-recovery} + +### Forgot password {#forgot-password} + +If you forgot your password, follow these steps for self-service recovery: +1. Go to https://console.clickhouse.cloud +2. Enter your email address and click Continue +3. Click Forgot your password? +4. Click Send password reset link +5. Check your email and click Reset password from the email +6. Enter your new password, confirm the password and click Update password +7. Click Back to sign in +8. Sign in normally with your new password + +### MFA self-service recovery {#mfa-self-serivce-recovery} + +If you lost your MFA device or deleted your token, follow these steps to recover and create a new token: +1. Go to https://console.clickhouse.cloud +2. Enter your credentials and click Continue +3. On the Multi-factor authentication screen click Cancel +4. Click Recovery code +5. Enter the code and press Continue +6. Copy the new recovery code and store it somewhere safe +7. Click the box next to `I have safely recorded this code` and click continue +8. Once signed in, go to your profile in the upper left +9. Click on security in the upper left +10. Click the trash can icon next to Authenticator app to remove your old authenticator +11. Click Remove authenticator app +12. When prompted for your Multi-factor authentication, click Cancel +13. Click Recovery code +14. Enter your recovery code (this is the new code generated in step 7) and click Continue +15. Copy the new recovery code and store it somewhere safe - this is a fail safe in case you leave the screen during the removal process +16. Click the box next to `I have safely recorded this code` and click Continue +17. Follow the process above to set up a new MFA factor + +### Lost MFA and recovery code {#lost-mfa-and-recovery-code} + +If you lost your MFA device AND recovery code or you lost your MFA device and never obtained a recovery code, follow these steps to request a reset: + +**Submit a ticket**: If you are in an organization that has other administrative users, even if you are attempting to access a single user organization, ask a member of your organization assigned the Admin role to log into the organization and submit a support ticket to reset your MFA on your behalf. Once we verify the request is authenticated, we will reset your MFA and notify the Admin. Sign in as usual without MFA and go to your profile settings to enroll a new factor if you wish. + +**Reset via email**: If you are the only user in the organization, submit a support case via email (support@clickhouse.com) using the email address associated with your account. Once we verify the request is coming from the correct email, we will reset your MFA AND password. Access your email to access the password reset link. Set up a new password then go to your profile settings to enroll a new factor if you wish. diff --git a/docs/cloud/guides/security/01_cloud_access_management/02_manage-cloud-users.md b/docs/cloud/guides/security/01_cloud_access_management/02_manage-cloud-users.md new file mode 100644 index 00000000000..e9659c7b63d --- /dev/null +++ b/docs/cloud/guides/security/01_cloud_access_management/02_manage-cloud-users.md @@ -0,0 +1,109 @@ +--- +sidebar_label: 'Manage cloud users' +slug: /cloud/security/manage-cloud-users +title: 'Manage cloud users' +description: 'This page describes how administrators can add users, manage assignments, and remove users' +doc_type: 'guide' +--- + +import Image from '@theme/IdealImage'; +import step_1 from '@site/static/images/cloud/guides/sql_console/org_level_access/1_org_settings.png' +import step_2 from '@site/static/images/cloud/guides/sql_console/org_level_access/2_org_settings.png' +import step_3 from '@site/static/images/cloud/guides/sql_console/org_level_access/3_org_settings.png' +import step_4 from '@site/static/images/cloud/guides/sql_console/org_level_access/4_org_settings.png' +import step_5 from '@site/static/images/cloud/guides/sql_console/org_level_access/5_org_settings.png' +import step_6 from '@site/static/images/cloud/guides/sql_console/org_level_access/6_org_settings.png' +import step_7 from '@site/static/images/cloud/guides/sql_console/org_level_access/7_org_settings.png' +import EnterprisePlanFeatureBadge from '@theme/badges/EnterprisePlanFeatureBadge' + +This guide is intended for users with the Organization Admin role in ClickHouse Cloud. + +## Add users to your organization {#add-users} + +### Invite users {#invite-users} + +Administrators may invite up to three (3) users at a time and assign organization and service level roles at the time of invitation. + +To invite users: +1. Select the organization name in the lower left corner +2. Click `Users and roles` +3. Select `Invite members` in the upper left corner +4. Enter the email address of up to 3 new users +5. Select the organization and service roles that will be assigned to the users +6. Click `Send invites` + +Users will receive an email from which they can join the organization. For more information on accepting invitations, see [Manage my account](/cloud/security/manage-my-account). + +### Add users via SAML identity provider {#add-users-via-saml} + + + +If your organization is configured for [SAML SSO](/cloud/security/saml-setup) follow these steps to add users to your organization. + +1. Add users to your SAML application in your identity provider, the users will not appear in ClickHouse until they have logged in once +2. When the user logs in to ClickHouse Cloud they will automatically be assigned the `Member` role which may only log in and has no other access +3. Follow the instructions in the `Manage user role assignments` below to grant permissions + +### Enforcing SAML-only authentication {#enforce-saml} + +Once you have at least one SAML user in the organization assigned to the Organization Admin role, remove users with other authentication methods from the organization to enforce SAML only authentication for the organization. + +## Manage user role assignments {#manage-role-assignments} + +Users assigned the Organization Admin role may update permissions for other users at any time. + + + +### Access organization settings {#access-organization-settings} + +From the services page, select the name of your organization: + + + +### Access users and roles {#access-users-and-roles} + +Select the `Users and roles` menu item from the popup menu. + + + +### Select the user to update {#select-user-to-update} + +Select the menu item at the end of the row for the user that you which to modify access for: + + + +### Select `edit` {#select-edit} + + + +A tab will display on the right hand side of the page: + + + +### Update permissions {#update-permissions} + +Select the drop-down menu items to adjust console-wide access permissions and which features a user can access from within the ClickHouse console. Refer to [Console roles and permissions](/cloud/security/console-roles) for a listing of roles and associated permissions. + +Select the drop-down menu items to adjust the access scope of the service role of the selected user. When selecting `Specific services`, you can control the role of the user per service. + + + +### Save your changes {#save-changes} + +Save your changes with the `Save changes` button at the bottom of the tab: + + + + + +## Remove a user {#remove-user} +:::note Remove SAML users +SAML users that have been unassigned from the ClickHouse application in your identity provider are not able to log in to ClickHouse Cloud. The account is not removed from the console and will need to be manually removed. +::: + +Follow the steps below to remove a user. + +1. Select the organization name in the lower left corner +2. Click `Users and roles` +3. Click the three dots next to the user's name and select `Remove` +4. Confirm the action by clicking the `Remove user` button diff --git a/docs/cloud/guides/SQL_console/configure_sql_console_role_assignments.md b/docs/cloud/guides/security/01_cloud_access_management/03_manage-sql-console-role-assignments.md similarity index 75% rename from docs/cloud/guides/SQL_console/configure_sql_console_role_assignments.md rename to docs/cloud/guides/security/01_cloud_access_management/03_manage-sql-console-role-assignments.md index fc2de19b0b0..9c93b1e102e 100644 --- a/docs/cloud/guides/SQL_console/configure_sql_console_role_assignments.md +++ b/docs/cloud/guides/security/01_cloud_access_management/03_manage-sql-console-role-assignments.md @@ -1,8 +1,8 @@ --- -slug: /cloud/guides/sql-console/config-sql-console-role-assignments -sidebar_label: 'Configuring SQL console role assignments' -title: 'Configuring SQL console role assignments' -description: 'Guide showing how to configure SQL console role assignments' +slug: /cloud/guides/sql-console/manage-sql-console-role-assignments +sidebar_label: 'Manage SQL console role assignments' +title: 'Manage SQL console role assignments' +description: 'Guide showing how to manage SQL console role assignments' doc_type: 'guide' --- @@ -21,12 +21,11 @@ import step_7 from '@site/static/images/cloud/guides/sql_console/service_level_a determine console-wide access permissions and the features that a user can access within Cloud console. - + -## Access service settings {#access-service-settings} +### Access service settings {#access-service-settings} -From the services page, click the menu in the top right corner of the service -for which you want to adjust SQL console access settings. +From the services page, click the menu in the top right corner of the service for which you want to adjust SQL console access settings. @@ -34,12 +33,14 @@ Select `settings` from the popup menu. -## Adjust SQL console access {#adjust-sql-console-access} +### Adjust SQL console access {#adjust-sql-console-access} Under the "Security" section, find the "SQL console access" area: +### Update the settings for Service Admin {#update-settings-for-service-admin} + Select the drop-down menu for Service Admin to change the access control settings for Service Admin roles: @@ -52,6 +53,8 @@ You can choose from the following roles: | `Read only` | | `Full access` | +### Update the settings for Service Read Only {#update-settings-for-service-read-only} + Select the drop-down menu for Service Read Only to change the access control settings for Service Read Only roles: @@ -64,6 +67,8 @@ You can choose from the following roles: | `Read only` | | `Full access` | +### Review users with access {#review-users-with-access} + An overview of users for the service can be viewed by selecting the user count: @@ -72,4 +77,4 @@ A tab will open to the right of the page showing the total number of users and t - \ No newline at end of file + diff --git a/docs/cloud/features/06_security/02_cloud-access-management/cloud-access-management.md b/docs/cloud/guides/security/01_cloud_access_management/04_manage-database-users.md similarity index 65% rename from docs/cloud/features/06_security/02_cloud-access-management/cloud-access-management.md rename to docs/cloud/guides/security/01_cloud_access_management/04_manage-database-users.md index 9711b82daca..2045491cf04 100644 --- a/docs/cloud/features/06_security/02_cloud-access-management/cloud-access-management.md +++ b/docs/cloud/guides/security/01_cloud_access_management/04_manage-database-users.md @@ -1,47 +1,28 @@ --- -sidebar_label: 'Overview' -slug: /cloud/security/cloud-access-management/overview -title: 'Cloud access management' -description: 'Describes how access control in ClickHouse cloud works, including information on role types' +sidebar_label: 'Manage database users' +slug: /cloud/security/manage-database-users +title: 'Manage database users' +description: 'This page describes how administrators can add database users, manage assignments, and remove database users' doc_type: 'guide' --- import Image from '@theme/IdealImage'; import user_grant_permissions_options from '@site/static/images/cloud/security/cloud-access-management/user_grant_permissions_options.png'; -# Access control in ClickHouse Cloud {#access-control-in-clickhouse-cloud} - -ClickHouse Cloud controls access to the console itself and the features available within it. -A **console user** is the foundation of this access with all permissions, roles, and access controls assigned to and managed through these users. -When [database-level permissions are associated with console users](/cloud/security/common-access-management-queries#modifying-users-and-roles), those then govern their data access when querying via the SQL console. - -## Console users and roles {#console-users-and-roles} - -[Configure Organization and Service role assignments](/cloud/guides/sql-console/configure-org-service-role-assignments) within the Console > Users and roles page. -[Configure SQL Console role assignments](/cloud/guides/sql-console/config-sql-console-role-assignments) in the settings page for each service. - -Users must be assigned an organization level role and may optionally be assigned service roles for one or more services. Service roles may be optionally configured for users to access the SQL console in the service settings page. -- Users assigned the Organization Admin role are granted Service Admin by default. -- Users added to an organization via a SAML integration are automatically assigned the Member role, with least privilege and without access to any services until configured. -- Service Admin is assigned the SQL console admin role by default. SQL console permissions may be removed in the service settings page. - -| Context | Role | Description | -|:-------------|:-----------------------|:-------------------------------------------------| -| Organization | Admin | Perform all administrative activities for an organization and control all settings. Assigned to the first user in the organization by default. | -| Organization | Developer | View access to everything except Services, ability to generate read-only API keys. | -| Organization | Billing | View usage and invoices, and manage payment methods. | -| Organization | Member | Sign-in only with the ability to manage personal profile settings. Assigned to SAML SSO users by default. | -| Service | Service Admin | Manage service settings. | -| Service | Service Read Only | View services and settings. | -| SQL console | SQL console admin | Administrative access to databases within the service equivalent to the Default database role. | -| SQL console | SQL console read only | Read only access to databases within the service | -| SQL console | Custom | Configure using SQL [`GRANT`](/sql-reference/statements/grant) statement; assign the role to a SQL console user by naming the role after the user | - +This guide demonstrates two ways to manage database users, within SQL console and directly within the database. + +### SQL console passwordless authentication {#sql-console-passwordless-authentication} +SQL console users are created for each session and authenticated using X.509 certificates that are automatically rotated. The user is removed when the session is terminated. When generating access lists for audits, please navigate to the Settings tab for the service in the console and note the SQL console access in addition to the database users that exist in the database. If custom roles are configured, the user's access is listed in the role ending with the user's username. + +## SQL console users and roles {#sql-console-users-and-roles} + +Basic SQL console roles can be assigned to users with Service Read Only and Service Admin permissions. For more information, refer to [Manage SQL Console Role Assignments](/cloud/guides/sql-console/manage-sql-console-role-assignments). This guide demonstrates how to create a custom role for a SQL console user. + To create a custom role for a SQL console user and grant it a general role, run the following commands. The email address must match the user's email address in the console. -#### Create `database_developer` and grant permissions {#create-database_developer-and-grant-permissions} +#### Create `database_developer` and grant permissions {#create-role-grant-permissions} Create the `database_developer` role and grant `SHOW`, `CREATE`, `ALTER`, and `DELETE` permissions. @@ -53,7 +34,7 @@ GRANT ALTER ON * TO database_developer; GRANT DELETE ON * TO database_developer; ``` -#### Create SQL console user role {#create-sql-console-user-role} +#### Create SQL console user role {#create-sql-console-user-role} Create a role for the SQL console user my.user@domain.com and assign it the database_developer role. @@ -62,10 +43,37 @@ CREATE ROLE OR REPLACE `sql-console-role:my.user@domain.com`; GRANT database_developer TO `sql-console-role:my.user@domain.com`; ``` +#### The user is assigned the new role when they use SQL console {#use-assigned-new-role} + +The user will be assigned the role associated with their email address whenever they use SQL console. + -### SQL console passwordless authentication {#sql-console-passwordless-authentication} -SQL console users are created for each session and authenticated using X.509 certificates that are automatically rotated. The user is removed when the session is terminated. When generating access lists for audits, please navigate to the Settings tab for the service in the console and note the SQL console access in addition to the database users that exist in the database. If custom roles are configured, the user's access is listed in the role ending with the user's username. +## Database authentication {#database-authentication} + +### Database user ID and password {#database-user-id--password} + +Use the SHA256_hash method when [creating user accounts](/sql-reference/statements/create/user.md) to secure passwords. ClickHouse database passwords must contain a minimum of 12 characters and meet complexity requirements: upper case characters, lower case characters, numbers and/or special characters. + +:::tip Generate passwords securely +Since users with less than administrative privileges cannot set their own password, ask the user to hash their password using a generator +such as [this one](https://tools.keycdn.com/sha256-online-generator) before providing it to the admin to setup the account. +::: + +```sql +CREATE USER userName IDENTIFIED WITH sha256_hash BY 'hash'; +``` + +### Database user with secure shell (SSH) authentication {#database-ssh} + +To set up SSH authentication for a ClickHouse Cloud database user. + +1. Use ssh-keygen to create a keypair. +2. Use the public key to create the user. +3. Assign roles and/or permissions to the user. +4. Use the private key to authenticate against the service. + +For a detailed walkthrough with examples, check out [How to connect to ClickHouse Cloud using SSH keys](/knowledgebase/how-to-connect-to-ch-cloud-using-ssh-keys) in our Knowledgebase. ## Database permissions {#database-permissions} Configure the following within the services and databases using the SQL [GRANT](/sql-reference/statements/grant) statement. @@ -144,4 +152,8 @@ d. Scroll to the SQL console access section. e. Click the link for the number of users with access to the database `There are # users with access to this service.` to see the user listing. - \ No newline at end of file + + +## Warehouse users {#warehouse-users} + +Warehouse users are shared across services within the same warehouse. For more information, review [warehouse access controls](/cloud/reference/warehouses#access-controls). diff --git a/docs/cloud/guides/security/cloud_access_management/saml-sso-setup.md b/docs/cloud/guides/security/01_cloud_access_management/04_saml-sso-setup.md similarity index 88% rename from docs/cloud/guides/security/cloud_access_management/saml-sso-setup.md rename to docs/cloud/guides/security/01_cloud_access_management/04_saml-sso-setup.md index a76adfdd672..89ff0d9ceb3 100644 --- a/docs/cloud/guides/security/cloud_access_management/saml-sso-setup.md +++ b/docs/cloud/guides/security/01_cloud_access_management/04_saml-sso-setup.md @@ -1,7 +1,7 @@ --- -sidebar_label: 'SAML SSO Setup' +sidebar_label: 'SAML SSO setup' slug: /cloud/security/saml-setup -title: 'SAML SSO Setup' +title: 'SAML SSO setup' description: 'How to set up SAML SSO with ClickHouse Cloud' doc_type: 'guide' --- @@ -335,23 +335,13 @@ Azure (Microsoft) SAML may also be referred to as Azure Active Directory (AD) or ## How it works {#how-it-works} -### Service provider-initiated SSO {#service-provider-initiated-sso} - -We only utilize service provider-initiated SSO. This means users go to `https://console.clickhouse.cloud` and enter their email address to be redirected to the IdP for authentication. Users already authenticated via your IdP can use the direct link to automatically log in to your organization without entering their email address at the login page. - -### Assigning user roles {#assigning-user-roles} - -Users will appear in your ClickHouse Cloud console after they are assigned to your IdP application and log in for the first time. At least one SSO user should be assigned the Admin role in your organization and additional users that login with SSO will be created with the role of ["Member"](/cloud/security/cloud-access-management/overview#console-users-and-roles), meaning that by default they do not have access to any services and should have their access and roles updated by an Admin. - -Use social login or `https://console.clickhouse.cloud/?with=email` to log in with your original authentication method to update your SSO role. +### User management with SAML SSO {#user-management-with-saml-sso} -### Removing non-SSO users {#removing-non-sso-users} +For more information on managing user permissions and restricting access to only SAML connections, refer to [Manage cloud users](/cloud/security/manage-cloud-users). -Once you have SSO users set up and have assigned at least one user the Admin role, the Admin can remove users using other methods (e.g. social authentication or user ID + password). Google authentication will continue to work after SSO is set up. User ID + password users will be automatically redirected to SSO based on their email domain unless users use `https://console.clickhouse.cloud/?with=email`. - -### Managing users {#managing-users} +### Service provider-initiated SSO {#service-provider-initiated-sso} -ClickHouse Cloud currently implements SAML for SSO. We have not yet implemented SCIM to manage users. This means SSO users must be assigned to the application in your IdP to access your ClickHouse Cloud organization. Users must log in to ClickHouse Cloud once to appear in the **Users** area in the organization. When users are removed in your IdP, they will not be able to log in to ClickHouse Cloud using SSO. However, the SSO user will still show in your organization until and administrator manually removes the user. +We only utilize service provider-initiated SSO. This means users go to `https://console.clickhouse.cloud` and enter their email address to be redirected to the IdP for authentication. Users already authenticated via your IdP can use the direct link to automatically log in to your organization without entering their email address at the login page. ### Multi-org SSO {#multi-org-sso} @@ -363,6 +353,14 @@ Security is our top priority when it comes to authentication. For this reason, w - **We only process service provider-initiated authentication flows.** Users must navigate to `https://console.clickhouse.cloud` and enter an email address to be redirected to your identity provider. Instructions to add a bookmark application or shortcut are provided for your convenience so your users don't need to remember the URL. -- **All users assigned to your app via your IdP must have the same email domain.** If you have vendors, contractors or consultants you would like to have access to your ClickHouse account, they must have an email address with the same domain (e.g. user@domain.com) as your employees. - - **We do not automatically link SSO and non-SSO accounts.** You may see multiple accounts for your users in your ClickHouse user list even if they are using the same email address. + +## Troubleshooting Common Issues {#troubleshooting-common-issues} + +| Error | Cause | Solution | +|:------|:------|:---------| +| There could be a misconfiguration in the system or a service outage | Identity provider initiated login | To resolve this error try using the direct link `https://console.clickhouse.cloud/?connection={organizationid}`. Follow the instructions for your identity provider above to make this the default login method for your users | +| You are directed to your identity provider, then back to the login page | The identity provider does not have the email attribute mapping | Follow the instructions for your identity provider above to configure the user email attribute and log in again | +| User is not assigned to this application | The user has not been assigned to the ClickHouse application in the identity provider | Assign the user to the application in the identity provider and log in again | +| You have multiple ClickHouse organizations integrated with SAML SSO and you are always logged into the same organization, regardless of which link or tile you use | You are still logged in to the first organization | Log out, then log in to the other organization | +| The URL briefly shows `access denied` | Your email domain does not match the domain we have configured | Reach out to support for assistance resolving this error | diff --git a/docs/cloud/guides/security/cloud_access_management/common-access-management-queries.md b/docs/cloud/guides/security/01_cloud_access_management/05_common-access-management-queries.md similarity index 97% rename from docs/cloud/guides/security/cloud_access_management/common-access-management-queries.md rename to docs/cloud/guides/security/01_cloud_access_management/05_common-access-management-queries.md index 2420d7cc77d..d27a3bbfbc2 100644 --- a/docs/cloud/guides/security/cloud_access_management/common-access-management-queries.md +++ b/docs/cloud/guides/security/01_cloud_access_management/05_common-access-management-queries.md @@ -1,6 +1,6 @@ --- -sidebar_label: 'Common Access Management Queries' -title: 'Common Access Management Queries' +sidebar_label: 'Common access management queries' +title: 'Common access management queries' slug: /cloud/security/common-access-management-queries description: 'This article shows the basics of defining SQL users and roles and applying those privileges and permissions to databases, tables, rows, and columns.' doc_type: 'guide' diff --git a/docs/cloud/features/06_security/02_cloud-access-management/_category_.json b/docs/cloud/guides/security/01_cloud_access_management/_category_.json similarity index 75% rename from docs/cloud/features/06_security/02_cloud-access-management/_category_.json rename to docs/cloud/guides/security/01_cloud_access_management/_category_.json index 784ea5ce006..bdfe0505050 100644 --- a/docs/cloud/features/06_security/02_cloud-access-management/_category_.json +++ b/docs/cloud/guides/security/01_cloud_access_management/_category_.json @@ -1,5 +1,5 @@ { "label": "Cloud access management", "collapsible": true, - "collapsed": true, + "collapsed": true } \ No newline at end of file diff --git a/docs/cloud/guides/security/01_cloud_access_management/index.md b/docs/cloud/guides/security/01_cloud_access_management/index.md new file mode 100644 index 00000000000..9cfa38f3483 --- /dev/null +++ b/docs/cloud/guides/security/01_cloud_access_management/index.md @@ -0,0 +1,20 @@ +--- +sidebar_label: 'Cloud access management' +slug: /cloud/security/cloud_access_management +title: 'Cloud access management' +description: 'Learn more about cloud access management' +doc_type: 'landing-page' +--- + +# Cloud access management + +This section contains detailed guides for managing access in ClickHouse Cloud + +| Page | Description | +|--------------------------------------------------------|-------------------------------------------------------| +| [Manage my account](/cloud/security/manage-my-account) | Describes how to manage your own user account, including passwords, MFA and account recovery | +| [Manage cloud users](/cloud/security/manage-cloud-users) | An administrator's guide to managing user access in the ClickHouse Cloud console | +| [Manage SQL console role assignments](/cloud/guides/sql-console/manage-sql-console-role-assignments) | An administrator's guide to managing SQL console users | +| [Manage database users](/cloud/security/manage-database-users) | An administrator's guide to managing database users | +| [SAML SSO setup](/cloud/security/saml-setup) | An administrator's guide to configuring and troubleshooting SAML integrations | +| [Common access management queries](/cloud/security/common-access-management-queries) | Detailed examples of setting up and verifying database permissions | diff --git a/docs/cloud/guides/security/connectivity/setting-ip-filters.md b/docs/cloud/guides/security/02_connectivity/01_setting-ip-filters.md similarity index 97% rename from docs/cloud/guides/security/connectivity/setting-ip-filters.md rename to docs/cloud/guides/security/02_connectivity/01_setting-ip-filters.md index 2415b6df228..9dc14b12ba7 100644 --- a/docs/cloud/guides/security/connectivity/setting-ip-filters.md +++ b/docs/cloud/guides/security/02_connectivity/01_setting-ip-filters.md @@ -1,7 +1,7 @@ --- -sidebar_label: 'Setting IP Filters' +sidebar_label: 'Setting IP filters' slug: /cloud/security/setting-ip-filters -title: 'Setting IP Filters' +title: 'Setting IP filters' description: 'This page explains how to set IP filters in ClickHouse Cloud to control access to ClickHouse services.' doc_type: 'guide' --- @@ -27,7 +27,7 @@ Classless Inter-domain Routing (CIDR) notation, allows you to specify IP address ## Create or modify an IP access list {#create-or-modify-an-ip-access-list} :::note Applicable only to connections outside of PrivateLink -IP access lists only apply to connections from the public internet, outside of [PrivateLink](/cloud/security/private-link-overview). +IP access lists only apply to connections from the public internet, outside of [PrivateLink](/cloud/security/connectivity/private-networking). If you only want traffic from PrivateLink, set `DenyAll` in IP Allow list. ::: diff --git a/docs/cloud/guides/security/connectivity/_category_.json b/docs/cloud/guides/security/02_connectivity/_category_.json similarity index 71% rename from docs/cloud/guides/security/connectivity/_category_.json rename to docs/cloud/guides/security/02_connectivity/_category_.json index 6e137e0592d..2d0bb58f6f7 100644 --- a/docs/cloud/guides/security/connectivity/_category_.json +++ b/docs/cloud/guides/security/02_connectivity/_category_.json @@ -1,5 +1,5 @@ { "label": "Connectivity", "collapsible": true, - "collapsed": true, + "collapsed": true } \ No newline at end of file diff --git a/docs/cloud/guides/security/connectivity/private_networking/02_aws-privatelink.md b/docs/cloud/guides/security/02_connectivity/private_networking/02_aws-privatelink.md similarity index 99% rename from docs/cloud/guides/security/connectivity/private_networking/02_aws-privatelink.md rename to docs/cloud/guides/security/02_connectivity/private_networking/02_aws-privatelink.md index f1de0b79c4a..b49cd2aa05f 100644 --- a/docs/cloud/guides/security/connectivity/private_networking/02_aws-privatelink.md +++ b/docs/cloud/guides/security/02_connectivity/private_networking/02_aws-privatelink.md @@ -382,4 +382,4 @@ According to the [AWS PrivateLink documentation](https://docs.aws.amazon.com/whi > Use AWS PrivateLink when you have a client/server set up where you want to allow one or more consumer VPCs unidirectional access to a specific service or set of instances in the service provider VPC. Only the clients in the consumer VPC can initiate a connection to the service in the service provider VPC. -To do this, configure your AWS Security Groups to allow connections from ClickHouse Cloud to your internal/private database service. Check the [default egress IP addresses for ClickHouse Cloud regions](/manage/security/cloud-endpoints-api), along with the [available static IP addresses](https://api.clickhouse.cloud/static-ips.json). +To do this, configure your AWS Security Groups to allow connections from ClickHouse Cloud to your internal/private database service. Check the [default egress IP addresses for ClickHouse Cloud regions](/manage/data-sources/cloud-endpoints-api), along with the [available static IP addresses](https://api.clickhouse.cloud/static-ips.json). diff --git a/docs/cloud/guides/security/connectivity/private_networking/03_gcp-private-service-connect.md b/docs/cloud/guides/security/02_connectivity/private_networking/03_gcp-private-service-connect.md similarity index 99% rename from docs/cloud/guides/security/connectivity/private_networking/03_gcp-private-service-connect.md rename to docs/cloud/guides/security/02_connectivity/private_networking/03_gcp-private-service-connect.md index f77d27e61ed..ee5a61b0ab9 100644 --- a/docs/cloud/guides/security/connectivity/private_networking/03_gcp-private-service-connect.md +++ b/docs/cloud/guides/security/02_connectivity/private_networking/03_gcp-private-service-connect.md @@ -428,7 +428,7 @@ According to the [GCP Private Service Connect documentation](https://cloud.googl > Service-oriented design: Producer services are published through load balancers that expose a single IP address to the consumer VPC network. Consumer traffic that accesses producer services is unidirectional and can only access the service IP address, rather than having access to an entire peered VPC network. -To do this, configure your GCP VPC firewall rules to allow connections from ClickHouse Cloud to your internal/private database service. Check the [default egress IP addresses for ClickHouse Cloud regions](/manage/security/cloud-endpoints-api), along with the [available static IP addresses](https://api.clickhouse.cloud/static-ips.json). +To do this, configure your GCP VPC firewall rules to allow connections from ClickHouse Cloud to your internal/private database service. Check the [default egress IP addresses for ClickHouse Cloud regions](/manage/data-sources/cloud-endpoints-api), along with the [available static IP addresses](https://api.clickhouse.cloud/static-ips.json). ## More information {#more-information} diff --git a/docs/cloud/guides/security/connectivity/private_networking/04_azure-privatelink.md b/docs/cloud/guides/security/02_connectivity/private_networking/04_azure-privatelink.md similarity index 100% rename from docs/cloud/guides/security/connectivity/private_networking/04_azure-privatelink.md rename to docs/cloud/guides/security/02_connectivity/private_networking/04_azure-privatelink.md diff --git a/docs/cloud/guides/security/connectivity/private_networking/_category_.json b/docs/cloud/guides/security/02_connectivity/private_networking/_category_.json similarity index 73% rename from docs/cloud/guides/security/connectivity/private_networking/_category_.json rename to docs/cloud/guides/security/02_connectivity/private_networking/_category_.json index 1b1476fa879..e0d5898a96e 100644 --- a/docs/cloud/guides/security/connectivity/private_networking/_category_.json +++ b/docs/cloud/guides/security/02_connectivity/private_networking/_category_.json @@ -1,5 +1,5 @@ { "label": "Private networking", "collapsible": true, - "collapsed": true, + "collapsed": true } \ No newline at end of file diff --git a/docs/cloud/guides/security/02_connectivity/private_networking/index.md b/docs/cloud/guides/security/02_connectivity/private_networking/index.md new file mode 100644 index 00000000000..b9c2345abf2 --- /dev/null +++ b/docs/cloud/guides/security/02_connectivity/private_networking/index.md @@ -0,0 +1,15 @@ +--- +slug: /cloud/security/connectivity/private-networking +title: 'Private networking' +hide_title: true +description: 'Table of contents page for the ClickHouse Cloud private networking section' +doc_type: 'landing-page' +--- + +# Private networking + +ClickHouse Cloud provides the ability to connect your services to your cloud virtual network. Refer to the guides below for set up steps for your provider: + +- [AWS private Link](/manage/security/aws-privatelink) +- [GCP private service connect](/manage/security/gcp-private-service-connect) +- [Azure private link](/cloud/security/azure-privatelink) diff --git a/docs/cloud/guides/security/data_masking.md b/docs/cloud/guides/security/03_data-masking.md similarity index 98% rename from docs/cloud/guides/security/data_masking.md rename to docs/cloud/guides/security/03_data-masking.md index c5ed892d1a0..a818670a4b4 100644 --- a/docs/cloud/guides/security/data_masking.md +++ b/docs/cloud/guides/security/03_data-masking.md @@ -142,7 +142,7 @@ SELECT * FROM masked_orders Notice that the data returned from the view is partially masked, obfuscating the sensitive information. You can also create multiple views, with differing levels of obfuscation depending on the level of privileged access to information the viewer has. -To ensure that users are only able to access the view returning the masked data, and not the table with the original unmasked data, you should use [Role Based Access Control](/cloud/security/cloud-access-management/overview) to ensure that specific roles only have grants to select from the view. +To ensure that users are only able to access the view returning the masked data, and not the table with the original unmasked data, you should use [Role Based Access Control](/cloud/security/console-roles) to ensure that specific roles only have grants to select from the view. First create the role: @@ -232,7 +232,7 @@ ORDER BY user_id ASC └─────────┴───────────────┴───────────────────────────┴──────────────┴──────────────┴────────────┴────────────────────────────────────┴──────────────┴────────────────────┴──────────────┴────────────────────────────┘ ``` -To ensure that users are only able to access columns containing the masked data, you can again use [Role Based Access Control](/cloud/security/cloud-access-management/overview) to ensure that specific roles only have grants to select on masked columns from `orders`. +To ensure that users are only able to access columns containing the masked data, you can again use [Role Based Access Control](/cloud/security/console-roles) to ensure that specific roles only have grants to select on masked columns from `orders`. Recreate the role that we made previously: diff --git a/docs/cloud/features/06_security/cmek.md b/docs/cloud/guides/security/04_cmek.md similarity index 80% rename from docs/cloud/features/06_security/cmek.md rename to docs/cloud/guides/security/04_cmek.md index d0c4f683e19..7a145e25d0a 100644 --- a/docs/cloud/features/06_security/cmek.md +++ b/docs/cloud/guides/security/04_cmek.md @@ -1,8 +1,8 @@ --- -sidebar_label: 'Enhanced Encryption' +sidebar_label: 'Data encryption' slug: /cloud/security/cmek -title: 'Customer Managed Encryption Keys (CMEK)' -description: 'Learn more about customer managed encryption keys' +title: 'Data encryption' +description: 'Learn more about data encryption in ClickHouse Cloud' doc_type: 'guide' --- @@ -10,7 +10,16 @@ import Image from '@theme/IdealImage'; import EnterprisePlanFeatureBadge from '@theme/badges/EnterprisePlanFeatureBadge' import cmek_performance from '@site/static/images/_snippets/cmek-performance.png'; -# ClickHouse enhanced encryption +# Data encryption + +## Storage level encryption {#storage-encryption} + +ClickHouse Cloud is configured with encryption at rest by default utilizing cloud provider-managed AES 256 keys. For more information review: +- [AWS server-side encryption for S3](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingServerSideEncryption.html) +- [GCP default encryption at rest](https://cloud.google.com/docs/security/encryption/default-encryption) +- [Azure storage encryption for data at rest](https://learn.microsoft.com/en-us/azure/storage/common/storage-service-encryption) + +## Database level encryption {#database-encryption} @@ -18,7 +27,7 @@ Data at rest is encrypted by default using cloud provider-managed AES 256 keys. Enhanced encryption is currently available in AWS and GCP services. Azure is coming soon. -## Transparent Data Encryption (TDE) {#transparent-data-encryption-tde} +### Transparent Data Encryption (TDE) {#transparent-data-encryption-tde} TDE must be enabled on service creation. Existing services cannot be encrypted after creation. Once TDE is enabled, it cannot be disabled. All data in the service will remain encrypted. If you want to disable TDE after it has been enabled, you must create a new service and migrate your data there. @@ -28,7 +37,7 @@ TDE must be enabled on service creation. Existing services cannot be encrypted a 4. Click the drop-down for Enterprise features and toggle Enable Transparent Data Encryption (TDE) 5. Click Create service -## Customer Managed Encryption Keys (CMEK) {#customer-managed-encryption-keys-cmek} +### Customer Managed Encryption Keys (CMEK) {#customer-managed-encryption-keys-cmek} :::warning Deleting a KMS key used to encrypt a ClickHouse Cloud service will cause your ClickHouse service to be stopped and its data will be unretrievable, along with existing backups. To prevent accidental data loss when rotating keys you may wish to maintain old KMS keys for a period of time prior to deletion. @@ -90,22 +99,20 @@ Once a service is encrypted with TDE, customers may update the key to enable CME -## Key rotation {#key-rotation} +#### Key rotation {#key-rotation} Once you set up CMEK, rotate the key by following the procedures above for creating a new KMS key and granting permissions. Return to the service settings to paste the new ARN (AWS) or Key Resource Path (GCP) and save the settings. The service will restart to apply the new key. -## Backup and restore {#backup-and-restore} +#### KMS key poller {#kms-key-poller} -Backups are encrypted using the same key as the associated service. When you restore an encrypted backup, it creates an encrypted instance that uses the same KMS key as the original instance. If needed, you can rotate the KMS key after restoration; see [Key Rotation](#key-rotation) for more details. +When using CMEK, the validity of the provided KMS key is checked every 10 minutes. If access to the KMS key is invalid, the ClickHouse service will stop. To resume service, restore access to the KMS key by following the steps in this guide, and then restart the service. -## KMS key poller {#kms-key-poller} +### Backup and restore {#backup-and-restore} -When using CMEK, the validity of the provided KMS key is checked every 10 minutes. If access to the KMS key is invalid, the ClickHouse service will stop. To resume service, restore access to the KMS key by following the steps in this guide, and then restart the service. +Backups are encrypted using the same key as the associated service. When you restore an encrypted backup, it creates an encrypted instance that uses the same KMS key as the original instance. If needed, you can rotate the KMS key after restoration; see [Key Rotation](#key-rotation) for more details. ## Performance {#performance} -As specified in this page, we use ClickHouse's built-in [Virtual File System for Data Encryption feature](/operations/storing-data#encrypted-virtual-file-system) to encrypt and protect your data. - -The algorithm in use for this feature is `AES_256_CTR`, which is expected to have a performance penalty of 5-15% depending on the workload: +Database encryption leverages ClickHouse's built-in [Virtual File System for Data Encryption feature](/operations/storing-data#encrypted-virtual-file-system) to encrypt and protect your data. The algorithm in use for this feature is `AES_256_CTR`, which is expected to have a performance penalty of 5-15% depending on the workload: CMEK Performance Penalty diff --git a/docs/cloud/reference/09_security/audit-logging.md b/docs/cloud/guides/security/05_audit_logging/01_console-audit-log.md similarity index 64% rename from docs/cloud/reference/09_security/audit-logging.md rename to docs/cloud/guides/security/05_audit_logging/01_console-audit-log.md index 139dbf5d358..fc724131a03 100644 --- a/docs/cloud/reference/09_security/audit-logging.md +++ b/docs/cloud/guides/security/05_audit_logging/01_console-audit-log.md @@ -1,9 +1,9 @@ --- -sidebar_label: 'Audit Logging' -slug: /cloud/security/audit-logging -title: 'Audit Logging' -description: 'This page describes Audit Logging in ClickHouse Cloud. It explains how to access and interpret the audit logs, which record changes made to a ClickHouse Cloud organization.' -doc_type: 'reference' +sidebar_label: 'Console audit log' +slug: /cloud/security/audit-logging/console-audit-log +title: 'Console audit log' +description: 'This page describes how users can review the cloud audit log' +doc_type: 'guide' --- import Image from '@theme/IdealImage'; @@ -11,12 +11,24 @@ import activity_log_1 from '@site/static/images/cloud/security/activity_log1.png import activity_log_2 from '@site/static/images/cloud/security/activity_log2.png'; import activity_log_3 from '@site/static/images/cloud/security/activity_log3.png'; +# Console audit log {#console-audit-log} + +User console activities are recorded in the audit log, which is available to users with the Admin or Developer organization role to review and integrate with logging systems. Specific events included in the console audit log are shown in the + +## Access the console log via the user interface {#console-audit-log-ui} + + + +## Select Organization {#select-org} + In ClickHouse Cloud, navigate to your organization details. ClickHouse Cloud activity tab
+## Select Audit {#select-audit} + Select the **Audit** tab on the left menu to see what changes have been made to your ClickHouse Cloud organization - including who made the change and when it occurred. The **Activity** page displays a table containing a list of events logged about your organization. By default, this list is sorted in a reverse-chronological order (most-recent event at the top). Change the order of the table by clicking on the columns headers. Each item of the table contains the following fields: @@ -30,42 +42,22 @@ The **Activity** page displays a table containing a list of events logged about
+## Use the search bar {#use-search-bar} + You can use the search bar provided to isolate events based on some criteria like for example service name or IP address. You can also export this information in a CSV format for distribution or analysis in an external tool. + +
ClickHouse Cloud Activity CSV export
-## List of events logged {#list-of-events-logged} - -The different types of events captured for the organization are grouped in 3 categories: **Service**, **Organization** and **User**. The list of events logged contains: - -### Service {#service} - -- Service created -- Service deleted -- Service stopped -- Service started -- Service name changed -- Service IP access list changed -- Service password reset - -### Organization {#organization} - -- Organization created -- Organization deleted -- Organization name changed - -### User {#user} - -- User role changed -- User removed from organization -- User invited to organization -- User joined organization -- User invitation deleted -- User left organization - -## API for audit events {#api-for-audit-events} +## Access the console audit log via the API {#console-audit-log-api} Users can use the ClickHouse Cloud API `activity` endpoint to obtain an export of audit events. Further details can be found in the [API reference](https://clickhouse.com/docs/cloud/manage/api/swagger). + +## Log integrations {#log-integrations} + +Users can use the API to integrate with a logging platform of their choice. The following have supported out-of-the-box connectors: +- [ClickHouse Cloud Audit add-on for Splunk](/integrations/audit-splunk) diff --git a/docs/cloud/guides/security/05_audit_logging/02_database-audit-log.md b/docs/cloud/guides/security/05_audit_logging/02_database-audit-log.md new file mode 100644 index 00000000000..ba0dbeaba86 --- /dev/null +++ b/docs/cloud/guides/security/05_audit_logging/02_database-audit-log.md @@ -0,0 +1,58 @@ +--- +sidebar_label: 'Database audit log' +slug: /cloud/security/audit-logging/database-audit-log +title: 'Database audit log' +description: 'This page describes how users can review the database audit log' +doc_type: 'guide' +--- + +# Database audit log {#database-audit-log} + +ClickHouse provides database audit logs by default. This page focuses on security relevant logs. For more information on data recorded by the system, refer to the docs for [system tables](/operations/system-tables/overview). + +:::tip Log retention +Information is logged directly to the system tables and are retained for up to 30 days by default. This period can be longer or shorter and is affected by the frequency of merges in the system. Customers may take additional measures to store logs for longer or export logs to a security information and event management (SIEM) system for long term storage. Details below. +::: + +## Security relevant logs {#security-relevant-logs} + +ClickHouse logs security relevant database events primarily to session and query logs. + +The [system.session_log](/operations/system-tables/session_log) records successful and failed login attempts, as well as the location of the authentication attempt. This information can be used to identify credential stuffing or brute force attacks against a ClickHouse instance. + +Sample query showing login failures +```sql +select event_time + ,type + ,user + ,auth_type + ,client_address +FROM clusterAllReplicas('default',system.session_log) +WHERE type='LoginFailure' +LIMIT 100 +``` + +The [system.query_log](/operations/system-tables/query_log) captures query activity executed in a ClickHouse instance. This information can be useful to determine what queries a threat actor executed. + +Sample query to search for activities of a "compromised_account" user +```sql +SELECT event_time + ,address + ,initial_user + ,initial_address + ,forwarded_for + ,query +FROM clusterAllReplicas('default', system.query_log) +WHERE user=’compromised_account’ +``` + +## Retaining log data within services {#reatining-log-data-within-services} + +Customers needing longer retention or log durability can use materialized views to achieve these objectives. For more information on materialized views, what they are, benefits and how to implement review our [materialized views](/materialized-views) videos and documentation. + +## Exporting logs {#exporting-logs} + +System logs may be written or exported to a storage location using various formats that are compatible with SIEM systems. For more information, review our [table functions](/sql-reference/table-functions) docs. The most common methods are: +- [Write to S3](/sql-reference/table-functions/s3) +- [Write to GCS](/sql-reference/table-functions/gcs) +- [Write to Azure Blob Storage](/sql-reference/table-functions/azureBlobStorage) diff --git a/docs/cloud/guides/security/05_audit_logging/03_byoc-security-playbook.md b/docs/cloud/guides/security/05_audit_logging/03_byoc-security-playbook.md new file mode 100644 index 00000000000..e0c7020946f --- /dev/null +++ b/docs/cloud/guides/security/05_audit_logging/03_byoc-security-playbook.md @@ -0,0 +1,61 @@ +--- +sidebar_label: 'BYOC security playbook' +slug: /cloud/security/audit-logging/byoc-security-playbook +title: 'BYOC security playbook' +description: 'This page illustrates methods customers can use to identify potential security events' +doc_type: 'guide' +--- + +# BYOC security playbook {#byoc-security-playbook} + +ClickHouse operates Bring Your Own Cloud (BYOC) under a security shared responsibility model, which can be downloaded from our Trust Center at https://trust.clickhouse.com. The following information is provided for BYOC customers as examples of how to identify potential security events. Customers should consider this information in the context of their security program to determine if additional detections and alerts may be helpful. + +## Potentially compromised ClickHouse credentials {#compromised-clickhouse-credentials} + +Refer to the [database audit log](/cloud/security/audit-logging/database-audit-log) documentation for queries to detect credential based attacks and queries to investigate malicious activities. + +## Application layer denial of service attack {#application-layer-dos-attack} + +There are various methods to execute a Denial of Service (DoS) attack. If the attack is focused on crashing the ClickHouse instance through a specific payload, recover the system back to a running state, or reboot the system and restrict access to regain control. Use the following query to review the [system.crash_log](/operations/system-tables/crash_log) to get more information about the attack. + +```sql +SELECT * +FROM clusterAllReplicas('default',system.crash_log) +``` + +## Compromised ClickHouse created AWS roles {#compromised-clickhouse-created-aws-roles} + +ClickHouse utilizes pre-created roles to enable system functions. This section assumes the customer is using AWS with CloudTrail and has access to the CloudTrail logs. + +If an incident may be the result of a compromised role, review activities in CloudTrail and CloudWatch related to the ClickHouse IAM roles and actions. Refer to the [CloudFormation](/cloud/reference/byoc#cloudformation-iam-roles) stack or Terraform module provided as part of setup for a list of IAM roles. + +## Unauthorized access to EKS cluster {#unauthorized-access-eks-cluster} + +ClickHouse BYOC runs inside EKS. This section assumes the customer is using CloudTrail and CloudWatch in AWS and has access to the logs. + +If an incident may be the result of a compromised EKS cluster, use the queries below within the EKS CloudWatch logs to identify specific threats. + +List the number of Kubernetes API calls by username +```sql +fields user.username +| stats count(*) as count by user.username +``` + +Identify whether a user is a ClickHouse engineer +```sql +fields @timestamp,user.extra.sessionName.0, requestURI, verb,userAgent, @message, @logStream, @log +| sort @timestamp desc +| filter user.username like /clickhouse.com/ +| limit 10000 +``` + +Review user accessing Kubernetes secrets, filter out service roles +```sql +fields @timestamp,user.extra.sessionName.0, requestURI, verb,userAgent, @message, @logStream, @log +| sort @timestamp desc +| filter requestURI like /secret/ +| filter verb="get" +| filter ispresent(user.extra.sessionName.0) +| filter user.username not like /ClickHouseManagementRole/ +| filter user.username not like /data-plane-mgmt/ +``` diff --git a/docs/cloud/guides/security/05_audit_logging/_category_.json b/docs/cloud/guides/security/05_audit_logging/_category_.json new file mode 100644 index 00000000000..db6d935c02e --- /dev/null +++ b/docs/cloud/guides/security/05_audit_logging/_category_.json @@ -0,0 +1,5 @@ +{ + "label": "Audit logging", + "collapsible": true, + "collapsed": true +} \ No newline at end of file diff --git a/docs/cloud/guides/security/05_audit_logging/index.md b/docs/cloud/guides/security/05_audit_logging/index.md new file mode 100644 index 00000000000..75c47d362c3 --- /dev/null +++ b/docs/cloud/guides/security/05_audit_logging/index.md @@ -0,0 +1,16 @@ +--- +sidebar_label: 'Audit logging' +slug: /cloud/security/audit_logging +title: 'Audit logging' +hide_title: true +description: 'Table of contents page for the ClickHouse Cloud audit logging section' +doc_type: 'landing-page' +--- + +# Audit logging + +| Page | Description | +|---------------------------------------------------------------------|------------------------------------------------------------------------------------| +| [Console audit log](/cloud/security/audit-logging/console-audit-log) | Accessing and reviewing audited events in the ClickHouse Cloud console | +| [Database audit logs](/cloud/security/audit-logging/database-audit-log) | Relevant logs for database activities | +| [BYOC security playbook](/cloud/security/audit-logging/byoc-security-playbook) | Sample logs and queries customers can use in developing their BYOC security program | \ No newline at end of file diff --git a/docs/cloud/guides/security/06_compliance/_category_.json b/docs/cloud/guides/security/06_compliance/_category_.json new file mode 100644 index 00000000000..e07fd81e071 --- /dev/null +++ b/docs/cloud/guides/security/06_compliance/_category_.json @@ -0,0 +1,5 @@ +{ + "label": "Compliance", + "collapsible": true, + "collapsed": true +} \ No newline at end of file diff --git a/docs/cloud/guides/security/06_compliance/hipaa-onboarding.md b/docs/cloud/guides/security/06_compliance/hipaa-onboarding.md new file mode 100644 index 00000000000..d6772fa6d6a --- /dev/null +++ b/docs/cloud/guides/security/06_compliance/hipaa-onboarding.md @@ -0,0 +1,98 @@ +--- +sidebar_label: 'HIPAA onboarding' +slug: /cloud/security/compliance/hipaa-onboarding +title: 'HIPAA onboarding' +description: 'Learn more about how to onboard to HIPAA compliant services' +doc_type: 'guide' +--- + +import BetaBadge from '@theme/badges/BetaBadge'; +import EnterprisePlanFeatureBadge from '@theme/badges/EnterprisePlanFeatureBadge'; + +import Image from '@theme/IdealImage'; +import hipaa1 from '@site/static/images/cloud/security/compliance/hipaa_1.png'; +import hipaa2 from '@site/static/images/cloud/security/compliance/hipaa_2.png'; +import hipaa3 from '@site/static/images/cloud/security/compliance/hipaa_3.png'; +import hipaa4 from '@site/static/images/cloud/security/compliance/hipaa_4.png'; + + + +ClickHouse offers services that are compliant with the Health Information Portability and Accountability Act (HIPAA) of 1996's Security Rule. Customers may process protected health information (PHI) within these services after signing a Business Associate Agreement (BAA) and deploying services to a compliant region. + +For more information about ClickHouse's compliance program and third party audit report availability, review our [compliance overview](/cloud/security/compliance-overview) and [Trust Center](https://trust.clickhouse.com). Additionally, customers should review our [security features](/cloud/security) page to select and implement appropriate security controls for their workloads. + +This page describes the process for enabling deployment of HIPAA compliant services in ClickHouse Cloud. + +## Enable and deploy HIPAA compliant services {#enable-hipaa-compliant-services} + + + +### Sign up for Enterprise services {#sign-up-for-enterprise} + +1. Select your organization name in the lower left corner of the console. +2. Click **Billing**. +3. Review your **Plan** in the upper left corner. +4. If your **Plan** is **Enterprise**, then go to the next section. If not, click **Change plan**. +5. Select **Switch to Enterprise**. + +### Enable HIPAA for your organization {#enable-hipaa} + +1. Select your organization name in the lower left corner of the console. +2. Click **Organization details**. +3. Toggle **Enable HIPAA** on. + +
+ +Request HIPAA enablement + +
+ +4. Follow the instructions on the screen to submit a request to complete a BAA. + +
+ +Submit a BAA request + +
+ +5. Once the BAA is completed, HIPAA will be enabled for the organization. + +
+ +HIPAA enabled + +
+ +### Deploy services to HIPAA compliant regions {#deploy-hippa-services} + +1. Select **New service** in the upper left corner of the home screen in the console +2. Change the **Region type** to **HIPAA compliant** + +
+ +Deploy to HIPAA region + +
+ +3. Enter a name for the service and enter the remaining information + +For a complete listing of HIPAA compliant cloud providers and services, review our [Supported cloud regions](/cloud/reference/supported-regions) page. + + + +## Migrate existing services {#migrate-to-hipaa} + +Customers are strongly encouraged to deploy services to compliant environments where required. The process to migrate services from a standard region to a HIPAA compliant region involves restoring from a backup and may require some downtime. + +If migration from standard to HIPAA compliant regions is required, follow these steps to perform self-service migrations: + +1. Select the service to be migrated. +2. Click **Backups** on the left. +3. Select the three dots to the left of the backup to be restored. +4. Select the **Region type** to restore the backup to a HIPAA compliant region. +5. Once the restoration is complete, run a few queries to verify the schemas and record counts are as expected. +6. Delete the old service. + +:::info Restrictions +Services must remain in the same cloud provider and geographic region. This process migrates the service to the compliant environment in the same cloud provider and region. +::: diff --git a/docs/cloud/guides/security/06_compliance/pci-onboarding.md b/docs/cloud/guides/security/06_compliance/pci-onboarding.md new file mode 100644 index 00000000000..0c108f93a81 --- /dev/null +++ b/docs/cloud/guides/security/06_compliance/pci-onboarding.md @@ -0,0 +1,87 @@ +--- +sidebar_label: 'PCI onboarding' +slug: /cloud/security/compliance/pci-onboarding +title: 'PCI onboarding' +description: 'Learn more about how to onboard to PCI compliant services' +doc_type: 'guide' +--- + +import BetaBadge from '@theme/badges/BetaBadge'; +import EnterprisePlanFeatureBadge from '@theme/badges/EnterprisePlanFeatureBadge'; + +import Image from '@theme/IdealImage'; +import pci1 from '@site/static/images/cloud/security/compliance/pci_1.png'; +import pci2 from '@site/static/images/cloud/security/compliance/pci_2.png'; +import pci3 from '@site/static/images/cloud/security/compliance/pci_3.png'; + + + +ClickHouse offers services that are compliant with the Payment Card Industry Data Security Standard (PCI-DSS) and is audited to Level 1 Service Provider requirements. Customers may process primary account numbers (PAN) within these services by enabling this feature and deploying services to a compliant region. + +For more information about ClickHouse's compliance program and third party audit report availability, review our [compliance overview](/cloud/security/compliance-overview). For a copy of our PCI shared responsibility document, visit our [Trust Center](https://trust.clickhouse.com). Additionally, customers should review our [security features](/cloud/security) page to select and implement appropriate security controls for their workloads. + +This page describes the process for enabling deployment of PCI compliant services in ClickHouse Cloud. + + + +### Sign up for Enterprise services {#sign-up-for-enterprise} + +1. Select your organization name in the lower left corner of the console. +2. Click **Billing**. +3. Review your **Plan** in the upper left corner. +4. If your **Plan** is **Enterprise**, then go to the next section. If not, click **Change plan**. +5. Select **Switch to Enterprise**. + +### Enable PCI for your organization {#enable-hipaa} + +1. Select your organization name in the lower left corner of the console. +2. Click **Organization details**. +3. Toggle **Enable PCI** on. + +
+ +Enable PCI + +
+ +4. Once enabled, PCI services can be deployed within the organization. + +
+ +PCI enabled + +
+ +### Deploy services to PCI compliant regions {#deploy-pci-regions} + +1. Select **New service** in the upper left corner of the home screen in the console +2. Change the **Region type** to **HIPAA compliant** + +
+ +Deploy to PCI region + +
+ +3. Enter a name for the service and enter the remaining information + +For a complete listing of PCI compliant cloud providers and services, review our [Supported cloud regions](/cloud/reference/supported-regions) page. + + + +## Migrate existing services {#migrate-to-hipaa} + +Customers are strongly encouraged to deploy services to compliant environments where required. The process to migrate services from a standard region to a PCI compliant region involves restoring from a backup and may require some downtime. + +If migration from standard to PCI compliant regions is required, follow these steps to perform self-service migrations: + +1. Select the service to be migrated. +2. Click **Backups** on the left. +3. Select the three dots to the left of the backup to be restored. +4. Select the **Region type** to restore the backup to a PCI compliant region. +5. Once the restoration is complete, run a few queries to verify the schemas and record counts are as expected. +6. Delete the old service. + +:::info Restrictions +Services must remain in the same cloud provider and geographic region. This process migrates the service to the compliant environment in the same cloud provider and region. +::: diff --git a/docs/cloud/guides/security/_category_.json b/docs/cloud/guides/security/_category_.json index aed26fa7f7a..c682d43a9b1 100644 --- a/docs/cloud/guides/security/_category_.json +++ b/docs/cloud/guides/security/_category_.json @@ -1,5 +1,5 @@ { "label": "Security", "collapsible": true, - "collapsed": true, + "collapsed": true } \ No newline at end of file diff --git a/docs/cloud/guides/security/cloud_access_management/_category_.json b/docs/cloud/guides/security/cloud_access_management/_category_.json deleted file mode 100644 index abfdcebed27..00000000000 --- a/docs/cloud/guides/security/cloud_access_management/_category_.json +++ /dev/null @@ -1,5 +0,0 @@ -{ - "label": "Cloud Access Management", - "collapsible": true, - "collapsed": true, -} \ No newline at end of file diff --git a/docs/cloud/guides/security/cloud_access_management/inviting-new-users.md b/docs/cloud/guides/security/cloud_access_management/inviting-new-users.md deleted file mode 100644 index d462a1cba89..00000000000 --- a/docs/cloud/guides/security/cloud_access_management/inviting-new-users.md +++ /dev/null @@ -1,31 +0,0 @@ ---- -sidebar_label: 'Inviting new users' -slug: /cloud/security/inviting-new-users -title: 'Inviting new users' -description: 'This page describes how administrators can invite new users to their organisation and assign roles to them' -doc_type: 'guide' ---- - -import Image from '@theme/IdealImage'; -import users_and_roles from '@site/static/images/cloud/security/users_and_roles.png'; -import invite_user from '@site/static/images/cloud/security/invite-user.png'; - -Administrators can invite others to organization, assigning them the `Developer`, `Admin` or `Billing Admin` role. - -:::note -Admins and developers are different than database users. To create database users and roles, please use the SQL console. To learn more, visit our docs on [Users and Roles](/cloud/security/cloud-access-management). -::: - -To invite a user, select the organization and click `Users and roles`: - -ClickHouse Cloud users and roles page - -
- -Select `Invite members`, and enter the email address of up to 3 new users at once, selecting the role for each. - -ClickHouse Cloud invite user page - -
- -Click `Send invites`. Users will receive an email from which they can join the organization. diff --git a/docs/cloud/onboard/02_migrate/01_migration_guides/03_bigquery/01_overview.md b/docs/cloud/onboard/02_migrate/01_migration_guides/03_bigquery/01_overview.md index 2f68e320bf5..fbbccf486c8 100644 --- a/docs/cloud/onboard/02_migrate/01_migration_guides/03_bigquery/01_overview.md +++ b/docs/cloud/onboard/02_migrate/01_migration_guides/03_bigquery/01_overview.md @@ -51,7 +51,7 @@ Lastly, I/O scheduling allows users to restrict local and remote disk accesses f ### Permissions {#permissions} -ClickHouse Cloud [controls user access](/cloud/security/cloud-access-management) in two places, via the [cloud console](/cloud/get-started/sql-console) and via the database. Console access is managed via the [clickhouse.cloud](https://console.clickhouse.cloud) user interface. Database access is managed via database user accounts and roles. Additionally, console users can be granted roles within the database that enable the console user to interact with the database via our [SQL console](/integrations/sql-clients/sql-console). +ClickHouse Cloud controls user access in two places, via the [cloud console](/cloud/guides/sql-console/manage-sql-console-role-assignments) and via the [database](/cloud/security/manage-database-users). Console access is managed via the [clickhouse.cloud](https://console.clickhouse.cloud) user interface. Database access is managed via database user accounts and roles. Additionally, console users can be granted roles within the database that enable the console user to interact with the database via our [SQL console](/integrations/sql-clients/sql-console). ## Data types {#data-types} diff --git a/docs/cloud/reference/01_changelog/01_changelog.md b/docs/cloud/reference/01_changelog/01_changelog.md index d46b36e6524..6533295b209 100644 --- a/docs/cloud/reference/01_changelog/01_changelog.md +++ b/docs/cloud/reference/01_changelog/01_changelog.md @@ -449,7 +449,7 @@ Please see the [ClickHouse docs](/cloud/notifications) to learn more about how t ### ClickHouse Cloud now offers HIPAA-ready services in Beta for GCP {#clickhouse-cloud-now-offers-hipaa-ready-services-in-beta-for-gcp} -Customers looking for increased security for protected health information (PHI) can now onboard to ClickHouse Cloud in [Google Cloud Platform (GCP)](https://cloud.google.com/). ClickHouse has implemented administrative, physical and technical safeguards prescribed by the [HIPAA Security Rule](https://www.hhs.gov/hipaa/for-professionals/security/index.html) and now has configurable security settings that can be implemented, depending on your specific use case and workload. For more information on available security settings, please review our [Security Shared Responsibility Model](/cloud/security/shared-responsibility-model). +Customers looking for increased security for protected health information (PHI) can now onboard to ClickHouse Cloud in [Google Cloud Platform (GCP)](https://cloud.google.com/). ClickHouse has implemented administrative, physical and technical safeguards prescribed by the [HIPAA Security Rule](https://www.hhs.gov/hipaa/for-professionals/security/index.html) and now has configurable security settings that can be implemented, depending on your specific use case and workload. For more information on available security settings, please review our [Security features page](/cloud/security). Services are available in GCP `us-central-1` to customers with the **Dedicated** service type and require a Business Associate Agreement (BAA). Contact [sales](mailto:sales@clickhouse.com) or [support](https://clickhouse.com/support/program) to request access to this feature or join the wait list for additional GCP, AWS, and Azure regions. @@ -954,7 +954,7 @@ This release brings usability and performance improvements in the SQL console, b This release brings general availability of ClickPipes for Kafka, Confluent Cloud, and Amazon MSK and the Kafka Connect ClickHouse Sink, self-service workflow to secure access to Amazon S3 via IAM roles, and AI-assisted query suggestions ( private preview). ### Console changes {#console-changes-11} -- Added a self-service workflow to secure [access to Amazon S3 via IAM roles](/cloud/security/secure-s3) +- Added a self-service workflow to secure [access to Amazon S3 via IAM roles](/cloud/data-sources/secure-s3) - Introduced AI-assisted query suggestions in private preview (please [contact ClickHouse Cloud support](https://console.clickhouse.cloud/support) to try it out.) ### Integrations changes {#integrations-changes-11} diff --git a/docs/cloud/reference/09_security/01_console-roles.md b/docs/cloud/reference/09_security/01_console-roles.md new file mode 100644 index 00000000000..2e9947776c2 --- /dev/null +++ b/docs/cloud/reference/09_security/01_console-roles.md @@ -0,0 +1,37 @@ +--- +sidebar_label: 'Console roles and permissions' +slug: /cloud/security/console-roles +title: 'Console roles and permissions' +description: 'This page describes the standard roles and associated permissions in ClickHouse Cloud console' +doc_type: 'reference' +--- + +## Organization roles {#organization-roles} +Refer to [Manage cloud users](/cloud/security/manage-cloud-users) for instructions on assigning organization roles. + +ClickHouse has four organization level roles available for user management. Only the admin role has default access to services. All other roles must be combined with service level roles to interact with services. + +| Role | Description | +|-----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Admin | Perform all administrative activities for an organization and control all settings. This role is assigned to the first user in the organization by default and automatically has Service Admin permissions on all services. | +| Developer | View access to the organization and ability to generate API keys with the same or lower permissions. | +| Billing | View usage and invoices, and manage payment methods. | +| Member | Sign-in only with the ability to manage personal profile settings. Assigned to SAML SSO users by default. | + +## Service roles {#service-roles} +Refer to [Manage cloud users](/cloud/security/manage-cloud-users) for instructions on assigning service roles. + +Service permissions must be explicitly granted by an admin to users with roles other than admin. Service Admin is pre-configured with SQL console admin access, but may be modified to reduce or remove permissions. + +| Role | Description | +|-------------------|-----------------------------| +| Service read only | View services and settings. | +| Service admin | Manage service settings. | + +## SQL console roles {#sql-console-roles} +Refer to [Manage SQL console role assignments](/cloud/guides/sql-console/manage-sql-console-role-assignments) for instructions on assigning SQL console roles. + +| Role | Description | +|-----------------------|------------------------------------------------------------------------------------------------| +| SQL console read only | Read only access to databases within the service. | +| SQL console admin | Administrative access to databases within the service equivalent to the Default database role. | \ No newline at end of file diff --git a/docs/cloud/reference/09_security/02_audit-logging.md b/docs/cloud/reference/09_security/02_audit-logging.md new file mode 100644 index 00000000000..80f71dac9f1 --- /dev/null +++ b/docs/cloud/reference/09_security/02_audit-logging.md @@ -0,0 +1,38 @@ +--- +sidebar_label: 'Console audit log events' +slug: /cloud/security/audit-logging +title: 'Console audit log events' +description: 'This page describes the events that are recorded to the console audit log.' +doc_type: 'reference' +--- + +## Console audit log events {#console-audit-log-events} + +The different types of events captured for the organization are grouped in 3 categories: **Organization**, **Service** and **User**. For more information about the audit log and how to export or add an API integration, review the [console audit log](/cloud/security/audit-logging/console-audit-log) documentation in the Guides section above. + +The following events are recorded to the audit log. + +### Organization {#organization} + +- Organization created +- Organization deleted +- Organization name changed + +### Service {#service} + +- Service created +- Service deleted +- Service stopped +- Service started +- Service name changed +- Service IP access list changed +- Service password reset + +### User {#user} + +- User role changed +- User removed from organization +- User invited to organization +- User joined organization +- User invitation deleted +- User left organization diff --git a/docs/cloud/reference/09_security/privacy_and_compliance/compliance-overview.md b/docs/cloud/reference/09_security/03_compliance-overview.md similarity index 59% rename from docs/cloud/reference/09_security/privacy_and_compliance/compliance-overview.md rename to docs/cloud/reference/09_security/03_compliance-overview.md index db226cad8e7..2c5a6a295ba 100644 --- a/docs/cloud/reference/09_security/privacy_and_compliance/compliance-overview.md +++ b/docs/cloud/reference/09_security/03_compliance-overview.md @@ -1,65 +1,48 @@ --- -title: 'Security and compliance reports' +title: 'Compliance overview' slug: /cloud/security/compliance-overview description: 'Overview of ClickHouse Cloud security and compliance certifications including SOC 2, ISO 27001, U.S. DPF, and HIPAA' -doc_type: 'guide' +doc_type: 'reference' --- import BetaBadge from '@theme/badges/BetaBadge'; import EnterprisePlanFeatureBadge from '@theme/badges/EnterprisePlanFeatureBadge'; # Security and compliance reports -ClickHouse Cloud evaluates the security and compliance needs of our customers and is continuously expanding the program as additional reports are requested. For additional information or to download the reports visit our [Trust Center](https://trust.clickhouse.com). +ClickHouse evaluates the security and compliance needs of our customers and is continuously expanding the program as additional reports are requested. For additional information or to download the reports visit our [Trust Center](https://trust.clickhouse.com). -### SOC 2 Type II (since 2022) {#soc-2-type-ii-since-2022} +## SOC 2 Type II (since 2022) {#soc-2-type-ii-since-2022} System and Organization Controls (SOC) 2 is a report focusing on security, availability, confidentiality, processing integrity and privacy criteria contained in the Trust Services Criteria (TSC) as applied to an organization's systems and is designed to provide assurance about these controls to relying parties (our customers). ClickHouse works with independent external auditors to undergo an audit at least once per year addressing security, availability and processing integrity of our systems and confidentiality and privacy of the data processed by our systems. The report addresses both our ClickHouse Cloud and Bring Your Own Cloud (BYOC) offerings. -### ISO 27001 (Since 2023) {#iso-27001-since-2023} +## ISO 27001 (Since 2023) {#iso-27001-since-2023} International Standards Organization (ISO) 27001 is an international standard for information security. It requires companies to implement an Information Security Management System (ISMS) that includes processes for managing risks, creating and communicating policies, implementing security controls, and monitoring to ensure components remain relevant and effective. ClickHouse conducts internal audits and works with independent external auditors to undergo audits and interim inspections for the 2 years between certificate issuance. -### U.S. DPF (since 2024) {#us-dpf-since-2024} +## U.S. DPF (since 2024) {#us-dpf-since-2024} The U.S. Data Privacy Framework was developed to provide U.S. organizations with reliable mechanisms for personal data transfers to the United States from the European Union/ European Economic Area, the United Kingdom, and Switzerland that are consistent with EU, UK and Swiss law (https://dataprivacyframework.gov/Program-Overview). ClickHouse self-certified to the framework and is listed on the [Data Privacy Framework List](https://dataprivacyframework.gov/list). -### HIPAA (since 2024) {#hipaa-since-2024} +## HIPAA (since 2024) {#hipaa-since-2024} -Customers that wish to deploy services to a HIPAA compliant region to load electronic protected health information (ePHI) may visit the **Organization** page in the console to request the feature to be enabled. A sales associate will reach out to obtain a signed Business Associate Agreement (BAA) to complete the setup. Customers deploying to HIPAA compliant regions should review our [shared responsibility model](/cloud/security/shared-responsibility-model), select and implement appropriate controls for their use case. - The Health Insurance Portability and Accountability Act (HIPAA) of 1996 is a United States based privacy law focused on management of protected health information (PHI). HIPAA has several requirements, including the [Security Rule](https://www.hhs.gov/hipaa/for-professionals/security/index.html), which is focused on protecting electronic personal health information (ePHI). ClickHouse has implemented administrative, physical and technical safeguards to ensure the confidentiality, integrity and security of ePHI stored in designated services. These activities are incorporated in our SOC 2 Type II report available for download in our [Trust Center](https://trust.clickhouse.com). -### PCI service provider (since 2025) {#pci-service-provider-since-2025} +Refer to [HIPAA onboarding](//cloud/security/compliance/hipaa-onboarding) for steps to complete a Business Associate Agreement (BAA) and deploy HIPAA compliant services. - +## PCI service provider (since 2025) {#pci-service-provider-since-2025} -Customers that wish to deploy services to PCI compliant regions to load cardholder data may visit the **Organization** page in the console to enable the feature. Once enabled, customers may select from a "PCI Compliant" region type when deploying new services. Customers deploying to PCI compliant regions should review our PCI responsibility overview available in our [Trust Center](https://trust.clickhouse.com), select and implement appropriate controls for their use case. + The [Payment Card Industry Data Security Standard (PCI DSS)](https://www.pcisecuritystandards.org/standards/pci-dss/) is a set of rules created by the PCI Security Standards Council to protect credit card payment data. ClickHouse has undergone an external audit with a Qualified Security Assessor (QSA) that resulted in a passing Report on Compliance (ROC) against PCI criteria relevant to storing credit card data. To download a copy of our Attestation on Compliance (AOC) and PCI responsibility overview, please visit our [Trust Center](https://trust.clickhouse.com). -# Privacy compliance - -In addition to the items above, ClickHouse maintains internal compliance programs addressing the General Data Protection Regulation (GDPR), California Consumer Privacy Act (CCPA) and other relevant privacy frameworks. Details on personal data that ClickHouse collects, how it is used, how it is protected and other privacy related information can be found in the following locations. - -### Legal documents {#legal-documents} - -- [Privacy Policy](https://clickhouse.com/legal/privacy-policy) -- [Cookie Policy](https://clickhouse.com/legal/cookie-policy) -- [Data Privacy Framework Notification](https://clickhouse.com/legal/data-privacy-framework) -- [Data Processing Addendum (DPA)](https://clickhouse.com/legal/agreements/data-processing-addendum) - -### Processing locations {#processing-locations} - -- [Sub-Processors and Affiliates](https://clickhouse.com/legal/agreements/subprocessors) -- [Data Processing Locations](https://trust.clickhouse.com) +Refer to [PCI onboarding](//cloud/security/compliance/pci-onboarding) for steps to deploy PCI compliant services. -### Additional procedures {#additional-procedures} +## Privacy compliance {#privacy-compliance} -- [Personal Data Access](/cloud/security/personal-data-access) -- [Delete Account](/cloud/manage/close_account) +In addition to the items above, ClickHouse maintains internal compliance programs addressing the General Data Protection Regulation (GDPR), California Consumer Privacy Act (CCPA) and other relevant privacy frameworks. -# Payment compliance +## Payment compliance {#payment-compliance} ClickHouse provides a secure method to pay by credit card that is compliant with [PCI SAQ A v4.0](https://www.pcisecuritystandards.org/document_library/). diff --git a/docs/cloud/reference/09_security/privacy_and_compliance/_category_.json b/docs/cloud/reference/09_security/privacy_and_compliance/_category_.json deleted file mode 100644 index 99beeb3e924..00000000000 --- a/docs/cloud/reference/09_security/privacy_and_compliance/_category_.json +++ /dev/null @@ -1,6 +0,0 @@ -{ - "label": "Privacy and compliance", - "collapsible": true, - "collapsed": true, - "link": { "type": "doc", "id": "cloud/reference/security/privacy_and_compliance/index" } -} \ No newline at end of file diff --git a/docs/cloud/reference/09_security/privacy_and_compliance/index.md b/docs/cloud/reference/09_security/privacy_and_compliance/index.md deleted file mode 100644 index 32f0491c4fa..00000000000 --- a/docs/cloud/reference/09_security/privacy_and_compliance/index.md +++ /dev/null @@ -1,16 +0,0 @@ ---- -sidebar_label: 'Privacy and Compliance Overview' -slug: /cloud/security/privacy-compliance-overview -title: 'Privacy and Compliance' -description: 'Landing page for privacy and compliance' -doc_type: 'landing-page' ---- - -# Privacy and compliance - -This section contains the following pages: - -| Page | Description | -|----------------------------------------------------------------------------|--------------------------------------------------------------| -| [Security and Compliance](/cloud/security/compliance-overview) | Security reports and privacy compliance of ClickHouse Cloud. | -| [Personal Data Access](/cloud/security/personal-data-access) | Information on how to access your personal data. | diff --git a/docs/cloud/reference/09_security/privacy_and_compliance/personal-data-access.md b/docs/cloud/reference/10_personal-data-access.md similarity index 99% rename from docs/cloud/reference/09_security/privacy_and_compliance/personal-data-access.md rename to docs/cloud/reference/10_personal-data-access.md index 7a301d023cb..a721d793feb 100644 --- a/docs/cloud/reference/09_security/privacy_and_compliance/personal-data-access.md +++ b/docs/cloud/reference/10_personal-data-access.md @@ -1,6 +1,6 @@ --- sidebar_label: 'Personal data access' -slug: /cloud/security/personal-data-access +slug: /cloud/manage/personal-data-access title: 'Personal data access' description: 'As a registered user, ClickHouse allows you to view and manage your personal account data, including contact information.' doc_type: 'reference' diff --git a/docs/cloud/reference/10_account-close.md b/docs/cloud/reference/11_account-close.md similarity index 100% rename from docs/cloud/reference/10_account-close.md rename to docs/cloud/reference/11_account-close.md diff --git a/docs/guides/sre/user-management/index.md b/docs/guides/sre/user-management/index.md index 69856c34c32..441c8daaf0f 100644 --- a/docs/guides/sre/user-management/index.md +++ b/docs/guides/sre/user-management/index.md @@ -34,7 +34,7 @@ You can't manage the same access entity by both configuration methods simultaneo ::: :::note -If you are looking to manage ClickHouse Cloud console users, please refer to this [page](/cloud/security/cloud-access-management) +If you are looking to manage ClickHouse Cloud console users, please refer to this [page](/cloud/security/manage-cloud-users) ::: To see all users, roles, profiles, etc. and all their grants use [`SHOW ACCESS`](/sql-reference/statements/show#show-access) statement. @@ -163,7 +163,7 @@ Management queries: ## Defining SQL users and roles {#defining-sql-users-and-roles} :::tip -If you are working in ClickHouse Cloud, please see [Cloud access management](/cloud/security/cloud-access-management). +If you are working in ClickHouse Cloud, please see [Cloud access management](/cloud/security/console-roles). ::: This article shows the basics of defining SQL users and roles and applying those privileges and permissions to databases, tables, rows, and columns. diff --git a/docs/integrations/data-ingestion/clickpipes/kafka/05_faq.md b/docs/integrations/data-ingestion/clickpipes/kafka/05_faq.md index 890dc76cbae..7ccadab57ba 100644 --- a/docs/integrations/data-ingestion/clickpipes/kafka/05_faq.md +++ b/docs/integrations/data-ingestion/clickpipes/kafka/05_faq.md @@ -31,7 +31,7 @@ ClickPipes is a separate cloud service that runs independently of the ClickHouse What are the requirements for using ClickPipes for Kafka? -In order to use ClickPipes for Kafka, you will need a running Kafka broker and a ClickHouse Cloud service with ClickPipes enabled. You will also need to ensure that ClickHouse Cloud can access your Kafka broker. This can be achieved by allowing remote connection on the Kafka side, whitelisting [ClickHouse Cloud Egress IP addresses](/manage/security/cloud-endpoints-api) in your Kafka setup. Alternatively, you can use [AWS PrivateLink](/integrations/clickpipes/aws-privatelink) to connect ClickPipes for Kafka to your Kafka brokers. +In order to use ClickPipes for Kafka, you will need a running Kafka broker and a ClickHouse Cloud service with ClickPipes enabled. You will also need to ensure that ClickHouse Cloud can access your Kafka broker. This can be achieved by allowing remote connection on the Kafka side, whitelisting [ClickHouse Cloud Egress IP addresses](/manage/data-sources/cloud-endpoints-api) in your Kafka setup. Alternatively, you can use [AWS PrivateLink](/integrations/clickpipes/aws-privatelink) to connect ClickPipes for Kafka to your Kafka brokers.
diff --git a/docs/integrations/data-ingestion/clickpipes/object-storage.md b/docs/integrations/data-ingestion/clickpipes/object-storage.md index 2d42d918f36..1f16324f094 100644 --- a/docs/integrations/data-ingestion/clickpipes/object-storage.md +++ b/docs/integrations/data-ingestion/clickpipes/object-storage.md @@ -162,7 +162,7 @@ Both publicly accessible and protected S3 buckets are supported. Public buckets need to allow both the `s3:GetObject` and the `s3:ListBucket` actions in their Policy. Protected buckets can be accessed using either [IAM credentials](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) or an [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html). -To use an IAM Role, you will need to create the IAM Role as specified [in this guide](/cloud/security/secure-s3). Copy the new IAM Role Arn after creation and paste it into the ClickPipe configuration as the "IAM ARN role". +To use an IAM Role, you will need to create the IAM Role as specified [in this guide](/cloud/data-sources/secure-s3). Copy the new IAM Role Arn after creation and paste it into the ClickPipe configuration as the "IAM ARN role". ### GCS {#gcs} Like S3, you can access public buckets with no configuration, and with protected buckets you can use [HMAC Keys](https://cloud.google.com/storage/docs/authentication/managing-hmackeys) in place of the AWS IAM credentials. You can read this guide from Google Cloud on [how to setup such keys](https://cloud.google.com/storage/docs/authentication/hmackeys). diff --git a/docs/use-cases/observability/clickstack/production.md b/docs/use-cases/observability/clickstack/production.md index a340da5da0e..ece3a35540d 100644 --- a/docs/use-cases/observability/clickstack/production.md +++ b/docs/use-cases/observability/clickstack/production.md @@ -86,7 +86,7 @@ Additionally, we recommend enabling TLS for OTLP endpoints and creating a [dedic ## ClickHouse {#clickhouse} -For production deployments, we recommend using [ClickHouse Cloud](https://clickhouse.com/cloud), which applies industry-standard [security practices](/cloud/security/shared-responsibility-model) by default - including [enhanced encryption](/cloud/security/cmek), [authentication and connectivity](/cloud/security/connectivity), and [managed access controls](/cloud/security/cloud-access-management). See ["ClickHouse Cloud"](#clickhouse-cloud-production) for a step-by-step guide of using ClickHouse Cloud with best practices. +For production deployments, we recommend using [ClickHouse Cloud](https://clickhouse.com/cloud), which applies industry-standard [security practices](/cloud/security) by default - including enhanced encryption, authentication and connectivity, and managed access controls. See ["ClickHouse Cloud"](#clickhouse-cloud-production) for a step-by-step guide of using ClickHouse Cloud with best practices. ### User permissions {#user-permissions} diff --git a/docs/whats-new/changelog/index.md b/docs/whats-new/changelog/index.md index c5c638730ba..84afb5ddc50 100644 --- a/docs/whats-new/changelog/index.md +++ b/docs/whats-new/changelog/index.md @@ -8,3 +8,1885 @@ title: '2025 Changelog' doc_type: 'changelog' --- +### Table of Contents +**[ClickHouse release v25.9, 2025-09-25](#259)**
+**[ClickHouse release v25.8 LTS, 2025-08-28](#258)**
+**[ClickHouse release v25.7, 2025-07-24](#257)**
+**[ClickHouse release v25.6, 2025-06-26](#256)**
+**[ClickHouse release v25.5, 2025-05-22](#255)**
+**[ClickHouse release v25.4, 2025-04-22](#254)**
+**[ClickHouse release v25.3 LTS, 2025-03-20](#253)**
+**[ClickHouse release v25.2, 2025-02-27](#252)**
+**[ClickHouse release v25.1, 2025-01-28](#251)**
+**[Changelog for 2024](https://clickhouse.com/docs/whats-new/changelog/2024/)**
+**[Changelog for 2023](https://clickhouse.com/docs/whats-new/changelog/2023/)**
+**[Changelog for 2022](https://clickhouse.com/docs/whats-new/changelog/2022/)**
+**[Changelog for 2021](https://clickhouse.com/docs/whats-new/changelog/2021/)**
+**[Changelog for 2020](https://clickhouse.com/docs/whats-new/changelog/2020/)**
+**[Changelog for 2019](https://clickhouse.com/docs/whats-new/changelog/2019/)**
+**[Changelog for 2018](https://clickhouse.com/docs/whats-new/changelog/2018/)**
+**[Changelog for 2017](https://clickhouse.com/docs/whats-new/changelog/2017/)**
+ + +### ClickHouse release 25.9, 2025-09-25 {#259} + +#### Backward Incompatible Change +* Disable nonsensical binary operations with IPv4/IPv6: Plus / minus of a IPv4/IPv6 with a non-integer type is disabled. Before it would allow operations with floating types and throw logical errors with some other types (such as DateTime). [#86336](https://github.com/ClickHouse/ClickHouse/pull/86336) ([Raúl Marín](https://github.com/Algunenano)). +* Deprecate setting `allow_dynamic_metadata_for_data_lakes`. Now all iceberg tables try to fetch up-to-date table schema from storage before executing of each query. [#86366](https://github.com/ClickHouse/ClickHouse/pull/86366) ([Daniil Ivanik](https://github.com/divanik)). +* Changed resolving of the coalesced column from `OUTER JOIN ... USING` clause to be more consistent: previously, when selecting both the USING column and qualified columns (`a, t1.a, t2.a`) in a OUTER JOIN, the USING column would incorrectly be resolved to `t1.a`, showing 0/NULL for rows from the right table with no left match. Now identifiers from USING clause are always resolved to the coalesced column, while qualified identifiers resolve to the non-coalesced columns, regardless of which other identifiers are present in the query. For example: ```sql SELECT a, t1.a, t2.a FROM (SELECT 1 as a WHERE 0) t1 FULL JOIN (SELECT 2 as a) t2 USING (a) -- Before: a=0, t1.a=0, t2.a=2 (incorrect - 'a' resolved to t1.a) -- After: a=2, t1.a=0, t2.a=2 (correct - 'a' is coalesced). [#80848](https://github.com/ClickHouse/ClickHouse/pull/80848) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Increase replicated deduplication window up to 10000. This is fully compatible, but we can imagine scenarios when this change could lead to high resource consumption in the presence of a large number of tables. [#86820](https://github.com/ClickHouse/ClickHouse/pull/86820) ([Sema Checherinda](https://github.com/CheSema)). + +#### New Feature +* Users can now use NATS JetStream to consume messages by specifying the new settings of `nats_stream` and `nats_consumer` for the NATS engine. [#84799](https://github.com/ClickHouse/ClickHouse/pull/84799) ([Dmitry Novikov](https://github.com/dmitry-sles-novikov)). +* Added support for authentication and SSL in the `arrowFlight` table function. [#87120](https://github.com/ClickHouse/ClickHouse/pull/87120) ([Vitaly Baranov](https://github.com/vitlibar)). +* Add new parameter to `S3` table engine and `s3` table function named `storage_class_name` which allows to specify intelligent tiring supported by AWS. Supported both in key-value format and in positional (deprecated) format). [#87122](https://github.com/ClickHouse/ClickHouse/pull/87122) ([alesapin](https://github.com/alesapin)). +* `ALTER UPDATE` for Iceberg table engine. [#86059](https://github.com/ClickHouse/ClickHouse/pull/86059) ([scanhex12](https://github.com/scanhex12)). +* Add system table `iceberg_metadata_log` to retrieve Iceberg metadata files during SELECT statements. [#86152](https://github.com/ClickHouse/ClickHouse/pull/86152) ([scanhex12](https://github.com/scanhex12)). +* `Iceberg` and `DeltaLake` tables support custom disk configuration via storage level setting `disk`. [#86778](https://github.com/ClickHouse/ClickHouse/pull/86778) ([scanhex12](https://github.com/scanhex12)). +* Support Azure for data lakes disks. [#87173](https://github.com/ClickHouse/ClickHouse/pull/87173) ([scanhex12](https://github.com/scanhex12)). +* Support `Unity` catalog on top of Azure blob storage. [#80013](https://github.com/ClickHouse/ClickHouse/pull/80013) ([Smita Kulkarni](https://github.com/SmitaRKulkarni)). +* Support more formats (`ORC`, `Avro`) in `Iceberg` writes. This closes [#86179](https://github.com/ClickHouse/ClickHouse/issues/86179). [#87277](https://github.com/ClickHouse/ClickHouse/pull/87277) ([scanhex12](https://github.com/scanhex12)). +* Add a new system table `database_replicas` with information about database replicas. [#83408](https://github.com/ClickHouse/ClickHouse/pull/83408) ([Konstantin Morozov](https://github.com/k-morozov)). +* Added function `arrayExcept` that subtracts one array as a set from another. [#82368](https://github.com/ClickHouse/ClickHouse/pull/82368) ([Joanna Hulboj](https://github.com/jh0x)). +* Adds a new `system.aggregated_zookeeper_log` table. The table contains statistics (e.g. number of operations, average latency, errors) of ZooKeeper operations grouped by session id, parent path and operation type, and periodically flushed to disk. [#85102](https://github.com/ClickHouse/ClickHouse/pull/85102) [#87208](https://github.com/ClickHouse/ClickHouse/pull/87208) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). +* A new function, `isValidASCII`. Returns 1 if the input string or FixedString contains only ASCII bytes (0x00–0x7F) else 0. Closes [#85377](https://github.com/ClickHouse/ClickHouse/issues/85377). ... [#85786](https://github.com/ClickHouse/ClickHouse/pull/85786) ([rajat mohan](https://github.com/rajatmohan22)). +* Boolean settings can be specified without arguments, e.g., `SET use_query_cache;`, which is equivalent to setting it to true. [#85800](https://github.com/ClickHouse/ClickHouse/pull/85800) ([thraeka](https://github.com/thraeka)). +* New configuration options: `logger.startupLevel` & `logger.shutdownLevel` allow for overriding the log level during the startup & shutdown of Clickhouse respectively. [#85967](https://github.com/ClickHouse/ClickHouse/pull/85967) ([Lennard Eijsackers](https://github.com/Blokje5)). +* Aggregate functions `timeSeriesChangesToGrid` and `timeSeriesResetsToGrid`. Behaves similarly to `timeSeriesRateToGrid`, accepting parameters for start timestamp, end timestamp, step, and look back window, as well as two arguments for the timestamps and values, but requiring at least 1 sample per window instead of 2. Calculates a PromQL `changes`/`resets`, counting the number of times the sample value changes or decreases in the specified window for each timestamp in the time grid defined by the parameters. The return type is `Array(Nullable(Float64))`. [#86010](https://github.com/ClickHouse/ClickHouse/pull/86010) ([Stephen Chi](https://github.com/stephchi0)). +* Allows users to create temporary views with a similar syntax to temporary tables (`CREATE TEMPORARY VIEW`). [#86432](https://github.com/ClickHouse/ClickHouse/pull/86432) ([Aly Kafoury](https://github.com/AlyHKafoury)). +* Add warnings for CPU and memory usage to the `system.warnings` table. [#86838](https://github.com/ClickHouse/ClickHouse/pull/86838) ([Bharat Nallan](https://github.com/bharatnc)). +* Support the `oneof` indicator in `Protobuf` inputs. A special column may be used to indicate the presence of part of oneof. If a message contains [oneof](https://protobuf.dev/programming-guides/proto3/#oneof) and `input_format_protobuf_oneof_presence` is set, ClickHouse fills column that indicates which field of oneof was found. [#82885](https://github.com/ClickHouse/ClickHouse/pull/82885) ([Ilya Golshtein](https://github.com/ilejn)). +* Improve allocation profiling based on jemalloc's internal tooling. Global jemalloc profiler can now be enabled with config `jemalloc_enable_global_profiler`. Sampled global allocations and deallocations can be stored in `system.trace_log` under `JemallocSample` type by enabling config `jemalloc_collect_global_profile_samples_in_trace_log`. Jemalloc profiling can now be enabled for each query independently using setting `jemalloc_enable_profiler`. Storing samples in `system.trace_log` can be controlled per query using setting `jemalloc_collect_profile_samples_in_trace_log`. Update jemalloc to newer version. [#85438](https://github.com/ClickHouse/ClickHouse/pull/85438) ([Antonio Andelic](https://github.com/antonio2368)). +* A new setting to delete files on dropping Iceberg tables. This closes [#86211](https://github.com/ClickHouse/ClickHouse/issues/86211). [#86501](https://github.com/ClickHouse/ClickHouse/pull/86501) ([scanhex12](https://github.com/scanhex12)). + +#### Experimental Feature +* The inverted text index was reworked from scratch to be scalable for datasets that don't fit into RAM. [#86485](https://github.com/ClickHouse/ClickHouse/pull/86485) ([Anton Popov](https://github.com/CurtizJ)). +* Join reordering now uses statistics. The feature can be enabled by setting `allow_statistics_optimize = 1` and `query_plan_optimize_join_order_limit = 10`. [#86822](https://github.com/ClickHouse/ClickHouse/pull/86822) ([Han Fei](https://github.com/hanfei1991)). +* Support `alter table ... materialize statistics all` will materialize all the statistics of a table. [#87197](https://github.com/ClickHouse/ClickHouse/pull/87197) ([Han Fei](https://github.com/hanfei1991)). + +#### Performance Improvement +* Support filtering data parts using skip indexes during reading to reduce unnecessary index reads. Controlled by the new setting `use_skip_indexes_on_data_read` (disabled by default). This addresses [#75774](https://github.com/ClickHouse/ClickHouse/issues/75774). This includes some common groundwork shared with [#81021](https://github.com/ClickHouse/ClickHouse/issues/81021). [#81526](https://github.com/ClickHouse/ClickHouse/pull/81526) ([Amos Bird](https://github.com/amosbird)). +* Added JOIN order optimization that can automatically reorder JOINs for better performance (controlled by `query_plan_optimize_join_order_limit` setting). Note that the join order optimization currently has limited statistics support and primarily relies on row count estimates from storage engines - more sophisticated statistics collection and cardinality estimation will be added in future releases. **If you encounter issues with JOIN queries after upgrading**, you can temporarily disable the new implementation by setting `SET query_plan_use_new_logical_join_step = 0` and report the issue for investigation. **Note about resolution of identifiers from USING clause**: Changed resolving of the coalesced column from `OUTER JOIN ... USING` clause to be more consistent: previously, when selecting both the USING column and qualified columns (`a, t1.a, t2.a`) in a OUTER JOIN, the USING column would incorrectly be resolved to `t1.a`, showing 0/NULL for rows from the right table with no left match. Now identifiers from USING clause are always resolved to the coalesced column, while qualified identifiers resolve to the non-coalesced columns, regardless of which other identifiers are present in the query. For example: ```sql SELECT a, t1.a, t2.a FROM (SELECT 1 as a WHERE 0) t1 FULL JOIN (SELECT 2 as a) t2 USING (a) -- Before: a=0, t1.a=0, t2.a=2 (incorrect - 'a' resolved to t1.a) -- After: a=2, t1.a=0, t2.a=2 (correct - 'a' is coalesced). [#80848](https://github.com/ClickHouse/ClickHouse/pull/80848) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Distributed `INSERT SELECT` for data lakes. [#86783](https://github.com/ClickHouse/ClickHouse/pull/86783) ([scanhex12](https://github.com/scanhex12)). +* Improve PREWHERE optimization for conditions like `func(primary_column) = 'xx'` and `column in (xxx)`. [#85529](https://github.com/ClickHouse/ClickHouse/pull/85529) ([李扬](https://github.com/taiyang-li)). +* Implemented rewriting of JOIN: 1. Convert `LEFT ANY JOIN` and `RIGHT ANY JOIN` to `SEMI`/`ANTI` JOIN if the filter condition is always false for matched or non-matched rows. This optimization is controlled by a new setting `query_plan_convert_any_join_to_semi_or_anti_join`. 2. Convert `FULL ALL JOIN` to `LEFT ALL` or `RIGHT ALL` JOIN if the filter condition is always false for non-matched rows from one side. [#86028](https://github.com/ClickHouse/ClickHouse/pull/86028) ([Dmitry Novik](https://github.com/novikd)). +* Improved performance of vertical merges after executing a lightweight delete. [#86169](https://github.com/ClickHouse/ClickHouse/pull/86169) ([Anton Popov](https://github.com/CurtizJ)). +* `HashJoin` performance optimised slightly in the case of `LEFT/RIGHT` join having a lot of unmatched rows. [#86312](https://github.com/ClickHouse/ClickHouse/pull/86312) ([Nikita Taranov](https://github.com/nickitat)). +* Radix sort: help the compiler use SIMD and do better prefetching. Uses dynamic dispatch to use software prefetching with Intel CPUs only. Continues the work by @taiyang-li in https://github.com/ClickHouse/ClickHouse/pull/77029. [#86378](https://github.com/ClickHouse/ClickHouse/pull/86378) ([Raúl Marín](https://github.com/Algunenano)). +* Improves performance of short queries with lots of parts in tables (by optimizing `MarkRanges` by using `devector` over `deque`). [#86933](https://github.com/ClickHouse/ClickHouse/pull/86933) ([Azat Khuzhin](https://github.com/azat)). +* Improved performance of applying patch parts in the join mode. [#87094](https://github.com/ClickHouse/ClickHouse/pull/87094) ([Anton Popov](https://github.com/CurtizJ)). +* Added setting `query_condition_cache_selectivity_threshold` (default value: 1.0) which excludes scan results of predicates with low selectivity from insertion into the query condition cache. This allows to reduce the memory consumption of the query condition cache at the cost of a worse cache hit rate. [#86076](https://github.com/ClickHouse/ClickHouse/pull/86076) ([zhongyuankai](https://github.com/zhongyuankai)). +* Reduce memory usage in Iceberg writes. [#86544](https://github.com/ClickHouse/ClickHouse/pull/86544) ([scanhex12](https://github.com/scanhex12)). + +#### Improvement +* Support writing multiple data files in Iceberg in a single insertion. Add new settings, `iceberg_insert_max_rows_in_data_file` and `iceberg_insert_max_bytes_in_data_file` to control the limits. [#86275](https://github.com/ClickHouse/ClickHouse/pull/86275) ([scanhex12](https://github.com/scanhex12)). +* Add rows/bytes limit for inserted data files in delta lake. Controlled by settings `delta_lake_insert_max_rows_in_data_file` and `delta_lake_insert_max_bytes_in_data_file`. [#86357](https://github.com/ClickHouse/ClickHouse/pull/86357) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Support more types for partitions in Iceberg writes. This closes [#86206](https://github.com/ClickHouse/ClickHouse/issues/86206). [#86298](https://github.com/ClickHouse/ClickHouse/pull/86298) ([scanhex12](https://github.com/scanhex12)). +* Make S3 retry strategy configurable and make settings of S3 disk can be hot reload if change the config XML file. [#82642](https://github.com/ClickHouse/ClickHouse/pull/82642) ([RinChanNOW](https://github.com/RinChanNOWWW)). +* Improved S3(Azure)Queue table engine to allow it to survive zookeeper connection loss without potential duplicates. Requires enabling S3Queue setting `use_persistent_processing_nodes` (changeable by `ALTER TABLE MODIFY SETTING`). [#85995](https://github.com/ClickHouse/ClickHouse/pull/85995) ([Kseniia Sumarokova](https://github.com/kssenii)). +* You can use query parameters after `TO` when creating a materialized view, for example: `CREATE MATERIALIZED VIEW mv TO {to_table:Identifier} AS SELECT * FROM src_table`. [#84899](https://github.com/ClickHouse/ClickHouse/pull/84899) ([Diskein](https://github.com/Diskein)). +* Give more clear instruction for users when incorrect settings are specified for the `Kafka2` table engine. [#83701](https://github.com/ClickHouse/ClickHouse/pull/83701) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). +* It's no longer possible to specify time zones for the `Time` type (it didn't make sense). [#84689](https://github.com/ClickHouse/ClickHouse/pull/84689) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Simplified (and avoided some bugs) a logic related to parsing Time/Time64 in the `best_effort` mode. [#84730](https://github.com/ClickHouse/ClickHouse/pull/84730) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Added `deltaLakeAzureCluster` function (similar to `deltaLakeAzure` for the cluster mode) and `deltaLakeS3Cluster` (alias to `deltaLakeCluster`) function. Resolves [#85358](https://github.com/ClickHouse/ClickHouse/issues/85358). [#85547](https://github.com/ClickHouse/ClickHouse/pull/85547) ([Smita Kulkarni](https://github.com/SmitaRKulkarni)). +* Apply `azure_max_single_part_copy_size` setting for normal copy operations in the same way as for backup. [#85767](https://github.com/ClickHouse/ClickHouse/pull/85767) ([Ilya Golshtein](https://github.com/ilejn)). +* Slow down S3 client threads on retryable errors in S3 Object Storage. This extends the previous setting `backup_slow_all_threads_after_retryable_s3_error` to S3 disks and renames it to the more general `s3_slow_all_threads_after_retryable_error`. [#85918](https://github.com/ClickHouse/ClickHouse/pull/85918) ([Julia Kartseva](https://github.com/jkartseva)). +* Mark settings allow_experimental_variant/dynamic/json and enable_variant/dynamic/json as obsolete. Now all three types are enabled unconditionally. [#85934](https://github.com/ClickHouse/ClickHouse/pull/85934) ([Pavel Kruglov](https://github.com/Avogar)). +* Support filtering by complete URL string (`full_url` directive) in `http_handlers` (including schema and host:port). [#86155](https://github.com/ClickHouse/ClickHouse/pull/86155) ([Azat Khuzhin](https://github.com/azat)). +* Add a new setting, `allow_experimental_delta_lake_writes`. [#86180](https://github.com/ClickHouse/ClickHouse/pull/86180) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix detection of systemd in init.d script (fixes "Install packages" check). [#86187](https://github.com/ClickHouse/ClickHouse/pull/86187) ([Azat Khuzhin](https://github.com/azat)). +* Add a new `startup_scripts_failure_reason` dimensional metric. This metric is needed to distinguish between different error types that result in failing startup scripts. In particular, for alerting purposes, we need to distinguish between transient (e.g., `MEMORY_LIMIT_EXCEEDED` or `KEEPER_EXCEPTION`) and non-transient errors. [#86202](https://github.com/ClickHouse/ClickHouse/pull/86202) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). +* Allow to omit `identity` function for partition for Iceberg table. [#86314](https://github.com/ClickHouse/ClickHouse/pull/86314) ([scanhex12](https://github.com/scanhex12)). +* Add ability to enable JSON logging only for specific channel, for this set `logger.formatting.channel` to one of `syslog`/`console`/`errorlog`/`log`. [#86331](https://github.com/ClickHouse/ClickHouse/pull/86331) ([Azat Khuzhin](https://github.com/azat)). +* Allow using native numbers in `WHERE`. They are already allowed to be arguments of logical functions. This simplifies filter-push-down and move-to-prewhere optimizations. [#86390](https://github.com/ClickHouse/ClickHouse/pull/86390) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fixed error in case of executing `SYSTEM DROP REPLICA` against a Catalog with corrupted metadata. [#86391](https://github.com/ClickHouse/ClickHouse/pull/86391) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Add extra retries for disk access check (`skip_access_check = 0`) in Azure because it may be provisioning access for quite a long time. [#86419](https://github.com/ClickHouse/ClickHouse/pull/86419) ([Alexander Tokmakov](https://github.com/tavplubix)). +* Make the staleness window in `timeSeries*()` functions left-open and right-closed. [#86588](https://github.com/ClickHouse/ClickHouse/pull/86588) ([Vitaly Baranov](https://github.com/vitlibar)). +* Add `FailedInternal*Query` profile events. [#86627](https://github.com/ClickHouse/ClickHouse/pull/86627) ([Shane Andrade](https://github.com/mauidude)). +* Fixes handling of users with a dot in the name when added via config file. [#86633](https://github.com/ClickHouse/ClickHouse/pull/86633) ([Mikhail Koviazin](https://github.com/mkmkme)). +* Add asynchronous metric for memory usage in queries (`QueriesMemoryUsage` and `QueriesPeakMemoryUsage`). [#86669](https://github.com/ClickHouse/ClickHouse/pull/86669) ([Azat Khuzhin](https://github.com/azat)). +* You can use `clickhouse-benchmark --precise` flag for more precise reporting of QPS and other per-interval metrics. It helps to get consistent QPS in case if durations of queries are comparable to the reporting interval `--delay D`. [#86684](https://github.com/ClickHouse/ClickHouse/pull/86684) ([Sergei Trifonov](https://github.com/serxa)). +* Make nice values of Linux threads configurable to assign some threads (merge/mutate, query, materialized view, zookeeper client) higher or lower priorities. [#86703](https://github.com/ClickHouse/ClickHouse/pull/86703) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). +* Fix misleading “specified upload does not exist” error, which occurs when the original exception is lost in multipart upload because of a race condition. [#86725](https://github.com/ClickHouse/ClickHouse/pull/86725) ([Julia Kartseva](https://github.com/jkartseva)). +* Limit query plan description in the `EXPLAIN` query. Do not calculate the description for queries other than `EXPLAIN`. Added a steeing `query_plan_max_step_description_length`. [#86741](https://github.com/ClickHouse/ClickHouse/pull/86741) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Add ability to tune pending signals in attempt to overcome CANNOT_CREATE_TIMER (for query profilers, `query_profiler_real_time_period_ns`/`query_profiler_cpu_time_period_ns`). And also collect `SigQ` from the `/proc/self/status` for introspection (if `ProcessSignalQueueSize` is near to `ProcessSignalQueueLimit`, then you will likely get `CANNOT_CREATE_TIMER` errors). [#86760](https://github.com/ClickHouse/ClickHouse/pull/86760) ([Azat Khuzhin](https://github.com/azat)). +* Improve performance of `RemoveRecursive` request in Keeper. [#86789](https://github.com/ClickHouse/ClickHouse/pull/86789) ([Antonio Andelic](https://github.com/antonio2368)). +* Remove extra whitespace in `PrettyJSONEachRow` during JSON type output. [#86819](https://github.com/ClickHouse/ClickHouse/pull/86819) ([Pavel Kruglov](https://github.com/Avogar)). +* Now we write blobs sizes of for `prefix.path` when directory is removed for plain rewriteable disk. [#86908](https://github.com/ClickHouse/ClickHouse/pull/86908) ([alesapin](https://github.com/alesapin)). +* Support performance tests against remote ClickHouse instances, including ClickHouse Cloud. Usage example: `tests/performance/scripts/perf.py tests/performance/math.xml --runs 10 --user --password --host --port --secure`. [#86995](https://github.com/ClickHouse/ClickHouse/pull/86995) ([Raufs Dunamalijevs](https://github.com/rienath)). +* Respect memory limits in some places that are known to allocate significant (>16MiB) amount of memory (sorting, async inserts, file log). [#87035](https://github.com/ClickHouse/ClickHouse/pull/87035) ([Azat Khuzhin](https://github.com/azat)). +* Throw an exception if setting `network_compression_method` is not a supported generic codec. [#87097](https://github.com/ClickHouse/ClickHouse/pull/87097) ([Robert Schulze](https://github.com/rschu1ze)). +* System table `system.query_cache` now returns _all_ query result cache entries, whereas it previously returned only shared entries or non-shared entries of the same user and role. That is okay as non-shared entries are supposed to not reveal _query results_, whereas `system.query_cache` returns _query strings_. This makes the behavior of the system table more similar to `system.query_log`. [#87104](https://github.com/ClickHouse/ClickHouse/pull/87104) ([Robert Schulze](https://github.com/rschu1ze)). +* Enable short circuit evaluation for `parseDateTime` function. [#87184](https://github.com/ClickHouse/ClickHouse/pull/87184) ([Pavel Kruglov](https://github.com/Avogar)). +* Add a new column `statistics` in `system.parts_columns`. [#87259](https://github.com/ClickHouse/ClickHouse/pull/87259) ([Han Fei](https://github.com/hanfei1991)). + +#### Bug Fix (user-visible misbehavior in an official stable release) +* The results of alter queries are only validated on the initiator node for replicated databases and internally replicated tables. This will fix situations where an already committed alter query could get stuck on other nodes. [#83849](https://github.com/ClickHouse/ClickHouse/pull/83849) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). +* Limit the number of tasks of each type in `BackgroundSchedulePool`. Avoid situations when all slots are occupied by task of one type, while other tasks are starving. Also avoids deadlocks when tasks wait for each other. This is controlled by `background_schedule_pool_max_parallel_tasks_per_type_ratio` server setting. [#84008](https://github.com/ClickHouse/ClickHouse/pull/84008) ([Alexander Tokmakov](https://github.com/tavplubix)). +* Shutdown tables properly when recovering database replica. Improper shutdown would lead to LOGICAL_ERROR for some table engines during database replica recovery. [#84744](https://github.com/ClickHouse/ClickHouse/pull/84744) ([Antonio Andelic](https://github.com/antonio2368)). +* Check access rights during typo correction hints generation for the database name. [#85371](https://github.com/ClickHouse/ClickHouse/pull/85371) ([Dmitry Novik](https://github.com/novikd)). +* 1. LowCardinality for hive columns 2. Fill hive columns before virtual columns (required for https://github.com/ClickHouse/ClickHouse/pull/81040) 3. LOGICAL_ERROR on empty format for hive [#85528](https://github.com/ClickHouse/ClickHouse/issues/85528) 4. Fix check for hive partition columns being the only columns 5. Assert all hive columns are specified in the schema 6. Partial fix for parallel_replicas_cluster with hive 7. Use ordered container in extractkeyValuePairs for hive utils (required for https://github.com/ClickHouse/ClickHouse/pull/81040). [#85538](https://github.com/ClickHouse/ClickHouse/pull/85538) ([Arthur Passos](https://github.com/arthurpassos)). +* Prevent unnecessary optimization of the first argument of `IN` functions sometimes resulting in error when array mapping is used. [#85546](https://github.com/ClickHouse/ClickHouse/pull/85546) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* Mapping between iceberg source ids and parquet names was not adjusted to the schema when the parquet file was written. This PR processes schema relevant for each iceberg data file, not a current one. [#85829](https://github.com/ClickHouse/ClickHouse/pull/85829) ([Daniil Ivanik](https://github.com/divanik)). +* Fix reading file size separately from opening it. Relates to https://github.com/ClickHouse/ClickHouse/pull/33372, which was introduced in response to a bug in Linux kernels prior to `5.10` release. [#85837](https://github.com/ClickHouse/ClickHouse/pull/85837) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* ClickHouse Keeper no longer fails to start on systems where IPv6 is disabled at the kernel level (e.g., RHEL with ipv6.disable=1). It now attempts to fall back to an IPv4 listener if the initial IPv6 listener fails. [#85901](https://github.com/ClickHouse/ClickHouse/pull/85901) ([jskong1124](https://github.com/jskong1124)). +* This PR closes [#77990](https://github.com/ClickHouse/ClickHouse/issues/77990). Add TableFunctionRemote support for parallel replicas in globalJoin. [#85929](https://github.com/ClickHouse/ClickHouse/pull/85929) ([zoomxi](https://github.com/zoomxi)). +* Fix null pointer in orcschemareader::initializeifneeded(). this pr addresses the following issue: [#85292](https://github.com/ClickHouse/ClickHouse/issues/85292) ### documentation entry for user-facing changes. [#85951](https://github.com/ClickHouse/ClickHouse/pull/85951) ([yanglongwei](https://github.com/ylw510)). +* Add a check to allow correlated subqueries in the FROM clause only if they use columns from the outer query. Fixes [#85469](https://github.com/ClickHouse/ClickHouse/issues/85469). Fixes [#85402](https://github.com/ClickHouse/ClickHouse/issues/85402). [#85966](https://github.com/ClickHouse/ClickHouse/pull/85966) ([Dmitry Novik](https://github.com/novikd)). +* Fix alter update of a column with a subcolumn used in other column materialized expression. Previously materialized column with subcolumn in its expression was not updated properly. [#85985](https://github.com/ClickHouse/ClickHouse/pull/85985) ([Pavel Kruglov](https://github.com/Avogar)). +* Forbid altering columns whose subcolumns are used in PK or partition expression. [#86005](https://github.com/ClickHouse/ClickHouse/pull/86005) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix reading subcolumns with non-default column mapping mode in storage DeltaLake. [#86064](https://github.com/ClickHouse/ClickHouse/pull/86064) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix using wrong default values for path with Enum hint inside JSON. [#86065](https://github.com/ClickHouse/ClickHouse/pull/86065) ([Pavel Kruglov](https://github.com/Avogar)). +* DataLake hive catalog url parsing with input sanitisation. Closes [#86018](https://github.com/ClickHouse/ClickHouse/issues/86018). [#86092](https://github.com/ClickHouse/ClickHouse/pull/86092) ([rajat mohan](https://github.com/rajatmohan22)). +* Fix logical error during filesystem cache dynamic resize. Closes [#86122](https://github.com/ClickHouse/ClickHouse/issues/86122). Closes https://github.com/ClickHouse/clickhouse-core-incidents/issues/473. [#86130](https://github.com/ClickHouse/ClickHouse/pull/86130) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Use `NonZeroUInt64` for `logs_to_keep` in DatabaseReplicatedSettings. [#86142](https://github.com/ClickHouse/ClickHouse/pull/86142) ([Tuan Pham Anh](https://github.com/tuanpach)). +* Exception was thrown by a `FINAL` query with skip index if the table (e.g `ReplacingMergeTree`) was created with setting`index_granularity_bytes = 0`. That exception has been fixed now. [#86147](https://github.com/ClickHouse/ClickHouse/pull/86147) ([Shankar Iyer](https://github.com/shankar-iyer)). +* Removes UB and fixes problems with parsing of Iceberg partition expression. [#86166](https://github.com/ClickHouse/ClickHouse/pull/86166) ([Daniil Ivanik](https://github.com/divanik)). +* Fix crash in case of const and non-const blocks in one INSERT. [#86230](https://github.com/ClickHouse/ClickHouse/pull/86230) ([Azat Khuzhin](https://github.com/azat)). +* Process includes from `/etc/metrika.xml` as a default when creating disks from SQL. [#86232](https://github.com/ClickHouse/ClickHouse/pull/86232) ([alekar](https://github.com/alekar)). +* Fix accurateCastOrNull/accurateCastOrDefault from String to JSON. [#86240](https://github.com/ClickHouse/ClickHouse/pull/86240) ([Pavel Kruglov](https://github.com/Avogar)). +* Support directories without '/' in iceberg engine. [#86249](https://github.com/ClickHouse/ClickHouse/pull/86249) ([scanhex12](https://github.com/scanhex12)). +* Fix crash with replaceRegex, a FixedString haystack and an empty needle. [#86270](https://github.com/ClickHouse/ClickHouse/pull/86270) ([Raúl Marín](https://github.com/Algunenano)). +* Fix crash during ALTER UPDATE Nullable(JSON). [#86281](https://github.com/ClickHouse/ClickHouse/pull/86281) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix missing column definer in system.tables. [#86295](https://github.com/ClickHouse/ClickHouse/pull/86295) ([Raúl Marín](https://github.com/Algunenano)). +* Fix cast from LowCardinality(Nullable(T)) to Dynamic. [#86365](https://github.com/ClickHouse/ClickHouse/pull/86365) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix logical error during writes to DeltaLake. Closes [#86175](https://github.com/ClickHouse/ClickHouse/issues/86175). [#86367](https://github.com/ClickHouse/ClickHouse/pull/86367) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix `416 The range specified is invalid for the current size of the resource. The range specified is invalid for the current size of the resource` when reading empty blobs from Azure blob storage for plain_rewritable disk. [#86400](https://github.com/ClickHouse/ClickHouse/pull/86400) ([Julia Kartseva](https://github.com/jkartseva)). +* Fix GROUP BY Nullable(JSON). [#86410](https://github.com/ClickHouse/ClickHouse/pull/86410) ([Pavel Kruglov](https://github.com/Avogar)). +* Fixed a bug in Materialized Views: an MV might not work if it was created, dropped, and then created again with the same name. [#86413](https://github.com/ClickHouse/ClickHouse/pull/86413) ([Alexander Tokmakov](https://github.com/tavplubix)). +* Fail if all replicas are unavailable when reading from *cluster functions. [#86414](https://github.com/ClickHouse/ClickHouse/pull/86414) ([Julian Maicher](https://github.com/jmaicher)). +* Fix leaking of `MergesMutationsMemoryTracking` due to `Buffer` tables and fix `query_views_log` for streaming from `Kafka` (and others). [#86422](https://github.com/ClickHouse/ClickHouse/pull/86422) ([Azat Khuzhin](https://github.com/azat)). +* Fix show tables after dropping reference table of alias storage. [#86433](https://github.com/ClickHouse/ClickHouse/pull/86433) ([RinChanNOW](https://github.com/RinChanNOWWW)). +* Fix missing chunk header when send_chunk_header is enabled and UDF is invoked via HTTP protocol. [#86469](https://github.com/ClickHouse/ClickHouse/pull/86469) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Fix possible deadlock in case of jemalloc profile flushes enabled. [#86473](https://github.com/ClickHouse/ClickHouse/pull/86473) ([Azat Khuzhin](https://github.com/azat)). +* Fix reading subcolumns in DeltaLake table engine. Closes [#86204](https://github.com/ClickHouse/ClickHouse/issues/86204). [#86477](https://github.com/ClickHouse/ClickHouse/pull/86477) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Handling loopback host ID properly to avoid collision when processing DDL tasks:. [#86479](https://github.com/ClickHouse/ClickHouse/pull/86479) ([Tuan Pham Anh](https://github.com/tuanpach)). +* Fix detach/attach for postgres database engine tables with numeric/decimal columns. [#86480](https://github.com/ClickHouse/ClickHouse/pull/86480) ([Julian Maicher](https://github.com/jmaicher)). +* Fix use of uninitialized memory in getSubcolumnType. [#86498](https://github.com/ClickHouse/ClickHouse/pull/86498) ([Raúl Marín](https://github.com/Algunenano)). +* Functions `searchAny` and `searchAll` when called with empty needles now return `true` (aka. "matches everything"). Previously, they returned `false`. (issue [#86300](https://github.com/ClickHouse/ClickHouse/issues/86300)). [#86500](https://github.com/ClickHouse/ClickHouse/pull/86500) ([Elmi Ahmadov](https://github.com/ahmadov)). +* Fix function `timeSeriesResampleToGridWithStaleness()` when the first bucket has no value. [#86507](https://github.com/ClickHouse/ClickHouse/pull/86507) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix crash caused by `merge_tree_min_read_task_size` being set to 0. [#86527](https://github.com/ClickHouse/ClickHouse/pull/86527) ([yanglongwei](https://github.com/ylw510)). +* While reading takes format for each data file from Iceberg metadata (earlier it was taken from table arguments). [#86529](https://github.com/ClickHouse/ClickHouse/pull/86529) ([Daniil Ivanik](https://github.com/divanik)). +* Ignore exceptions during flushing log on shutdown and make shutdown more safe (to avoid SIGSEGV). [#86546](https://github.com/ClickHouse/ClickHouse/pull/86546) ([Azat Khuzhin](https://github.com/azat)). +* Fix Backup db engine raising exception on query with zero sized part files. [#86563](https://github.com/ClickHouse/ClickHouse/pull/86563) ([Max Justus Spransy](https://github.com/maxjustus)). +* Fix missing chunk header when send_chunk_header is enabled and UDF is invoked via HTTP protocol. [#86606](https://github.com/ClickHouse/ClickHouse/pull/86606) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Fix S3Queue logical error "Expected current processor {} to be equal to {}", which happened because of keeper session expiration. [#86615](https://github.com/ClickHouse/ClickHouse/pull/86615) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Nullablity bugs in insert and pruning. This closes [#86407](https://github.com/ClickHouse/ClickHouse/issues/86407). [#86630](https://github.com/ClickHouse/ClickHouse/pull/86630) ([scanhex12](https://github.com/scanhex12)). +* Do not disable file system cache if Iceberg metadata cache is disabled. [#86635](https://github.com/ClickHouse/ClickHouse/pull/86635) ([Daniil Ivanik](https://github.com/divanik)). +* Fixed 'Deadlock in Parquet::ReadManager (single-threaded)' error in parquet reader v3. [#86644](https://github.com/ClickHouse/ClickHouse/pull/86644) ([Michael Kolupaev](https://github.com/al13n321)). +* Fix support for IPv6 in `listen_host` for ArrowFlight. [#86664](https://github.com/ClickHouse/ClickHouse/pull/86664) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix shutdown in `ArrowFlight` handler. This PR fixes [#86596](https://github.com/ClickHouse/ClickHouse/issues/86596). [#86665](https://github.com/ClickHouse/ClickHouse/pull/86665) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix distributed queries with `describe_compact_output=1`. [#86676](https://github.com/ClickHouse/ClickHouse/pull/86676) ([Azat Khuzhin](https://github.com/azat)). +* Fix window definition parsing and applying query parameters. [#86720](https://github.com/ClickHouse/ClickHouse/pull/86720) ([Azat Khuzhin](https://github.com/azat)). +* Fix exception `Partition strategy wildcard can not be used without a '_partition_id' wildcard.` when creating a table with `PARTITION BY`, but without partition wildcard, which used to work in versions before 25.8. Closes https://github.com/ClickHouse/clickhouse-private/issues/37567. [#86748](https://github.com/ClickHouse/ClickHouse/pull/86748) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix LogicalError if parallel queries are trying to acquire single lock. [#86751](https://github.com/ClickHouse/ClickHouse/pull/86751) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* Fix writing NULL into JSON shared data in RowBinary input format and add some additional validations in ColumnObject. [#86812](https://github.com/ClickHouse/ClickHouse/pull/86812) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix empty Tuple permutation with limit. [#86828](https://github.com/ClickHouse/ClickHouse/pull/86828) ([Pavel Kruglov](https://github.com/Avogar)). +* Do not use separate keeper node for persistent processing nodes. Fix for https://github.com/ClickHouse/ClickHouse/pull/85995. Closes [#86406](https://github.com/ClickHouse/ClickHouse/issues/86406). [#86841](https://github.com/ClickHouse/ClickHouse/pull/86841) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix TimeSeries engine table breaking creation of new replica in Replicated Database. [#86845](https://github.com/ClickHouse/ClickHouse/pull/86845) ([Nikolay Degterinsky](https://github.com/evillique)). +* Fix querying `system.distributed_ddl_queue` in cases where tasks are missing certain Keeper nodes. [#86848](https://github.com/ClickHouse/ClickHouse/pull/86848) ([Antonio Andelic](https://github.com/antonio2368)). +* Fix seeking at the end of the decompressed block. [#86906](https://github.com/ClickHouse/ClickHouse/pull/86906) ([Pavel Kruglov](https://github.com/Avogar)). +* Process exception which is thrown during asyncronous execution of Iceberg Iterator. [#86932](https://github.com/ClickHouse/ClickHouse/pull/86932) ([Daniil Ivanik](https://github.com/divanik)). +* Fix saving of big preprocessed XML configs. [#86934](https://github.com/ClickHouse/ClickHouse/pull/86934) ([c-end](https://github.com/c-end)). +* Fix date field populating in system.iceberg_metadata_log table. [#86961](https://github.com/ClickHouse/ClickHouse/pull/86961) ([Daniil Ivanik](https://github.com/divanik)). +* Fixed infinite recalculation of `TTL` with `WHERE`. [#86965](https://github.com/ClickHouse/ClickHouse/pull/86965) ([Anton Popov](https://github.com/CurtizJ)). +* Fixed possible incorrect result of `uniqExact` function with `ROLLUP` and `CUBE` modifiers. [#87014](https://github.com/ClickHouse/ClickHouse/pull/87014) ([Nikita Taranov](https://github.com/nickitat)). +* Fix resolving table schema with `url()` table function when `parallel_replicas_for_cluster_functions` setting is set to 1. [#87029](https://github.com/ClickHouse/ClickHouse/pull/87029) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Correctly cast output of PREWHERE after splitting it into multiple steps. [#87040](https://github.com/ClickHouse/ClickHouse/pull/87040) ([Antonio Andelic](https://github.com/antonio2368)). +* Fixed lightweight updates with `ON CLUSTER` clause. [#87043](https://github.com/ClickHouse/ClickHouse/pull/87043) ([Anton Popov](https://github.com/CurtizJ)). +* Fix compatibility of some aggregate function states with String argument. [#87049](https://github.com/ClickHouse/ClickHouse/pull/87049) ([Pavel Kruglov](https://github.com/Avogar)). +* Fixes an issue where model name from OpenAI wasn't passed through. [#87100](https://github.com/ClickHouse/ClickHouse/pull/87100) ([Kaushik Iska](https://github.com/iskakaushik)). +* EmbeddedRocksDB: Path must be inside user_files. [#87109](https://github.com/ClickHouse/ClickHouse/pull/87109) ([Raúl Marín](https://github.com/Algunenano)). +* Fix KeeperMap tables created before 25.1, leaving data in ZooKeeper after the DROP query. [#87112](https://github.com/ClickHouse/ClickHouse/pull/87112) ([Nikolay Degterinsky](https://github.com/evillique)). +* Fix maps and arrays field ids reading parquet. [#87136](https://github.com/ClickHouse/ClickHouse/pull/87136) ([scanhex12](https://github.com/scanhex12)). +* Fix reading array with array sizes subcolumn in lazy materialization. [#87139](https://github.com/ClickHouse/ClickHouse/pull/87139) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix CASE function with Dynamic arguments. [#87177](https://github.com/ClickHouse/ClickHouse/pull/87177) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix reading empty array from empty string in CSV. [#87182](https://github.com/ClickHouse/ClickHouse/pull/87182) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix possible wrong result of non-correlated `EXISTS`. It was broken with `execute_exists_as_scalar_subquery=1` which was introduced in https://github.com/ClickHouse/ClickHouse/pull/85481 and affects `25.8`. Fixes [#86415](https://github.com/ClickHouse/ClickHouse/issues/86415). [#87207](https://github.com/ClickHouse/ClickHouse/pull/87207) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Throws an error if iceberg_metadata_log is not configured, but user tries to get debug iceberg metadata info. Fixes nullptr access. [#87250](https://github.com/ClickHouse/ClickHouse/pull/87250) ([Daniil Ivanik](https://github.com/divanik)). + +#### Build/Testing/Packaging Improvement +* Fix compatibility with abseil-cpp 20250814.0, https://github.com/abseil/abseil-cpp/issues/1923. [#85970](https://github.com/ClickHouse/ClickHouse/pull/85970) ([Yuriy Chernyshov](https://github.com/georgthegreat)). +* Put building standalone WASM lexer under a flag. [#86505](https://github.com/ClickHouse/ClickHouse/pull/86505) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Fix crc32c build on older ARM CPUs without support for the `vmull_p64` instruction. [#86521](https://github.com/ClickHouse/ClickHouse/pull/86521) ([Pablo Marcos](https://github.com/pamarcos)). +* Use `openldap` 2.6.10. [#86623](https://github.com/ClickHouse/ClickHouse/pull/86623) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Do not try to intercept `memalign` on darwin. [#86769](https://github.com/ClickHouse/ClickHouse/pull/86769) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Use `krb5` 1.22.1-final. [#86836](https://github.com/ClickHouse/ClickHouse/pull/86836) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Fix unpacking Rust crate names in `list-licenses.sh`. [#87305](https://github.com/ClickHouse/ClickHouse/pull/87305) ([Konstantin Bogdanov](https://github.com/thevar1able)). + + +### ClickHouse release 25.8 LTS, 2025-08-28 {#258} + +#### Backward Incompatible Change +* Infer `Array(Dynamic)` instead of unnamed `Tuple` for arrays of values with different types in JSON. To use the previous behaviour, disable setting `input_format_json_infer_array_of_dynamic_from_array_of_different_types`. [#80859](https://github.com/ClickHouse/ClickHouse/pull/80859) ([Pavel Kruglov](https://github.com/Avogar)). +* Move S3 latency metrics to histograms for homogeneity and simplicity. [#82305](https://github.com/ClickHouse/ClickHouse/pull/82305) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). +* Require backticks around identifiers with dots in default expressions to prevent them from being parsed as compound identifiers. [#83162](https://github.com/ClickHouse/ClickHouse/pull/83162) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* Lazy materialization is enabled only with analyzer (which is the default) to avoid maintenance without analyzer, — which, in our experience, have some issues (for example, when using `indexHint()` in conditions). [#83791](https://github.com/ClickHouse/ClickHouse/pull/83791) ([Igor Nikonov](https://github.com/devcrafter)). +* Write values of `Enum` type as `BYTE_ARRAY` with `ENUM` logical type in Parquet output format by default. [#84169](https://github.com/ClickHouse/ClickHouse/pull/84169) ([Pavel Kruglov](https://github.com/Avogar)). +* Enable MergeTree setting `write_marks_for_substreams_in_compact_parts` by default. It significantly improves performance of subcolumns reading from newly created Compact parts. Servers with version less then 25.5 won't be able to read new Compact parts. [#84171](https://github.com/ClickHouse/ClickHouse/pull/84171) ([Pavel Kruglov](https://github.com/Avogar)). +* The previous `concurrent_threads_scheduler` default value was `round_robin`, which proved unfair in the presence of a high number of single-threaded queries (e.g., INSERTs). This change makes a safer alternative `fair_round_robin` scheduler, the default. [#84747](https://github.com/ClickHouse/ClickHouse/pull/84747) ([Sergei Trifonov](https://github.com/serxa)). +* ClickHouse supports PostgreSQL-style heredoc syntax: `$tag$ string contents... $tag$`, also known as dollar-quoted string literals. In previous versions, there were fewer restrictions on tags: they could contain arbitrary characters, including punctuation and whitespace. This introduces parsing ambiguity with identifiers that can also start with a dollar character. At the same time, PostgreSQL only allows word characters for tags. To resolve the problem, we now restrict heredoc tags only to contain word characters. Closes [#84731](https://github.com/ClickHouse/ClickHouse/issues/84731). [#84846](https://github.com/ClickHouse/ClickHouse/pull/84846) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* The functions `azureBlobStorage`, `deltaLakeAzure`, and `icebergAzure` have been updated to properly validate `AZURE` permissions. All cluster-variant functions (`-Cluster` functions) now verify permissions against their corresponding non-clustered counterparts. Additionally, the `icebergLocal` and `deltaLakeLocal` functions now enforce `FILE` permission checks. [#84938](https://github.com/ClickHouse/ClickHouse/pull/84938) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Enables `allow_dynamic_metadata_for_data_lakes` setting (Table Engine level setting) by default. [#85044](https://github.com/ClickHouse/ClickHouse/pull/85044) ([Daniil Ivanik](https://github.com/divanik)). +* Disable quoting 64 bit integers in JSON formats by default. [#74079](https://github.com/ClickHouse/ClickHouse/pull/74079) ([Pavel Kruglov](https://github.com/Avogar)) + +#### New Feature +* Basic support for the PromQL dialect is added. To use it, set `dialect='promql'` in clickhouse-client, point it to the TimeSeries table using the setting `promql_table_name='X'` and execute queries like `rate(ClickHouseProfileEvents_ReadCompressedBytes[1m])[5m:1m]`. In addition you can wrap the PromQL query with SQL: `SELECT * FROM prometheusQuery('up', ...);`. So far only functions `rate`, `delta` and `increase` are supported. No unary/binary operators. No HTTP API. [#75036](https://github.com/ClickHouse/ClickHouse/pull/75036) ([Vitaly Baranov](https://github.com/vitlibar)). +* AI Powered SQL generation can now infer from env ANTHROPIC_API_KEY and OPENAI_API_KEY if available, this is to make it so that we can have a zero config option to use this feature. [#83787](https://github.com/ClickHouse/ClickHouse/pull/83787) ([Kaushik Iska](https://github.com/iskakaushik)). +* Implement support for [ArrowFlight RPC](https://arrow.apache.org/docs/format/Flight.html) protocol by adding: - new table function `arrowflight`. [#74184](https://github.com/ClickHouse/ClickHouse/pull/74184) ([zakr600](https://github.com/zakr600)). +* Now all tables support the `_table` virtual column (not only tables with the `Merge` engine), which is especially useful for queries with UNION ALL. [#63665](https://github.com/ClickHouse/ClickHouse/pull/63665) ([Xiaozhe Yu](https://github.com/wudidapaopao)). +* Allow to use any storage policy (i.e. object storage, such as S3) for external aggregation/sorting. [#84734](https://github.com/ClickHouse/ClickHouse/pull/84734) ([Azat Khuzhin](https://github.com/azat)). +* Implement AWS S3 authentication with an explicitly provided IAM role. Implement OAuth for GCS. These features were recently only available in ClickHouse Cloud and are now open-sourced. Synchronize some interfaces such as serialization of the connection parameters for object storages. [#84011](https://github.com/ClickHouse/ClickHouse/pull/84011) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Support position deletes for Iceberg TableEngine. [#83094](https://github.com/ClickHouse/ClickHouse/pull/83094) ([Daniil Ivanik](https://github.com/divanik)). +* Support Iceberg Equality Deletes. [#85843](https://github.com/ClickHouse/ClickHouse/pull/85843) ([Han Fei](https://github.com/hanfei1991)). +* Iceberg writes for create. Closes [#83927](https://github.com/ClickHouse/ClickHouse/issues/83927). [#83983](https://github.com/ClickHouse/ClickHouse/pull/83983) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Glue catalogs for writes. [#84136](https://github.com/ClickHouse/ClickHouse/pull/84136) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Iceberg Rest catalogs for writes. [#84684](https://github.com/ClickHouse/ClickHouse/pull/84684) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Merge all iceberg position delete files into data files. This will reduce amount and sizes of parquet files in iceberg storage. Syntax: `OPTIMIZE TABLE table_name`. [#85250](https://github.com/ClickHouse/ClickHouse/pull/85250) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Support `drop table` for iceberg (Removing from REST/Glue catalogs + removing metadata about table). [#85395](https://github.com/ClickHouse/ClickHouse/pull/85395) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Support alter delete mutations for iceberg in merge-on-read format. [#85549](https://github.com/ClickHouse/ClickHouse/pull/85549) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Support writes into DeltaLake. Closes [#79603](https://github.com/ClickHouse/ClickHouse/issues/79603). [#85564](https://github.com/ClickHouse/ClickHouse/pull/85564) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Added setting `delta_lake_snapshot_version` to allow reading specific snapshot version in table engine `DeltaLake`. [#85295](https://github.com/ClickHouse/ClickHouse/pull/85295) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Write more iceberg statistics (column sizes, lower and upper bounds) in metadata (manifest entries) for min-max pruning. [#85746](https://github.com/ClickHouse/ClickHouse/pull/85746) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Support add/drop/modify columns in iceberg for simple types. [#85769](https://github.com/ClickHouse/ClickHouse/pull/85769) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Iceberg: support writing version-hint file. This closes [#85097](https://github.com/ClickHouse/ClickHouse/issues/85097). [#85130](https://github.com/ClickHouse/ClickHouse/pull/85130) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Views, created by ephemeral users, will now store a copy of an actual user and will no longer be invalidated after the ephemeral user is deleted. [#84763](https://github.com/ClickHouse/ClickHouse/pull/84763) ([pufit](https://github.com/pufit)). +* The vector similarity index now supports binary quantization. Binary quantization significantly reduces the memory consumption and speeds up the process of building a vector index (due to faster distance calculation). Also, the existing setting `vector_search_postfilter_multiplier `was made obsolete and replaced by a more general setting : `vector_search_index_fetch_multiplier`. [#85024](https://github.com/ClickHouse/ClickHouse/pull/85024) ([Shankar Iyer](https://github.com/shankar-iyer)). +* Allow key value arguments in `s3` or `s3Cluster` table engine/function, e.g. for example `s3('url', CSV, structure = 'a Int32', compression_method = 'gzip')`. [#85134](https://github.com/ClickHouse/ClickHouse/pull/85134) ([Kseniia Sumarokova](https://github.com/kssenii)). +* A new system table to keep erroneous incoming messages from engines like kafka ("dead letter queue"). [#68873](https://github.com/ClickHouse/ClickHouse/pull/68873) ([Ilya Golshtein](https://github.com/ilejn)). +* The new SYSTEM RESTORE DATABASE REPLICA for Replicated databases, similar to the existing functionality for restore in ReplicatedMergeTree. [#73100](https://github.com/ClickHouse/ClickHouse/pull/73100) ([Konstantin Morozov](https://github.com/k-morozov)). +* PostgreSQL protocol now supports the `COPY` command. [#74344](https://github.com/ClickHouse/ClickHouse/pull/74344) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Support C# client for mysql protocol. This closes [#83992](https://github.com/ClickHouse/ClickHouse/issues/83992). [#84397](https://github.com/ClickHouse/ClickHouse/pull/84397) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Add support for hive partition style reads and writes. [#76802](https://github.com/ClickHouse/ClickHouse/pull/76802) ([Arthur Passos](https://github.com/arthurpassos)). +* Add `zookeeper_connection_log` system table to store historical information about ZooKeeper connections. [#79494](https://github.com/ClickHouse/ClickHouse/pull/79494) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). +* Server setting `cpu_slot_preemption` enables preemptive CPU scheduling for workloads and ensures max-min fair allocation of CPU time among workloads. New workload settings for CPU throttling are added: `max_cpus`, `max_cpu_share` and `max_burst_cpu_seconds`. More details: https://clickhouse.com/docs/operations/workload-scheduling#cpu_scheduling. [#80879](https://github.com/ClickHouse/ClickHouse/pull/80879) ([Sergei Trifonov](https://github.com/serxa)). +* Drop TCP connection after a configured number of queries or time threshold. This makes sense for a more uniform connection distribution between cluster nodes behind a load balancer. Resolves [#68000](https://github.com/ClickHouse/ClickHouse/issues/68000). [#81472](https://github.com/ClickHouse/ClickHouse/pull/81472) ([Kenny Sun](https://github.com/hwabis)). +* Parallel replicas now support using projections for queries. [#82659](https://github.com/ClickHouse/ClickHouse/issues/82659). [#82807](https://github.com/ClickHouse/ClickHouse/pull/82807) ([zoomxi](https://github.com/zoomxi)). +* Support DESCRIBE SELECT in addition to DESCRIBE (SELECT ...). [#82947](https://github.com/ClickHouse/ClickHouse/pull/82947) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Force secure connection for mysql_port and postgresql_port. [#82962](https://github.com/ClickHouse/ClickHouse/pull/82962) ([tiandiwonder](https://github.com/tiandiwonder)). +* Users can now do case-insensitive JSON key lookups using `JSONExtractCaseInsensitive` (and other variants of `JSONExtract`). [#83770](https://github.com/ClickHouse/ClickHouse/pull/83770) ([Alistair Evans](https://github.com/alistairjevans)). +* Introduction of `system.completions` table. Closes [#81889](https://github.com/ClickHouse/ClickHouse/issues/81889). [#83833](https://github.com/ClickHouse/ClickHouse/pull/83833) ([|2ustam](https://github.com/RuS2m)). +* Added a new function `nowInBlock64`. Example usage: `SELECT nowInBlock64(6)` returns `2025-07-29 17:09:37.775725`. [#84178](https://github.com/ClickHouse/ClickHouse/pull/84178) ([Halersson Paris](https://github.com/halersson)). +* Add extra_credentials to AzureBlobStorage to authenticate with client_id and tenant_id. [#84235](https://github.com/ClickHouse/ClickHouse/pull/84235) ([Pablo Marcos](https://github.com/pamarcos)). +* Added function `dateTimeToUUIDv7` to convert a DateTime value to a UUIDv7. Example usage: `SELECT dateTimeToUUIDv7(toDateTime('2025-08-15 18:57:56'))` returns `0198af18-8320-7a7d-abd3-358db23b9d5c`. [#84319](https://github.com/ClickHouse/ClickHouse/pull/84319) ([samradovich](https://github.com/samradovich)). +* `timeSeriesDerivToGrid` and `timeSeriesPredictLinearToGrid` aggregate functions to re-sample data to a time grid defined by the specified start timestamp, end timestamp, and step; calculates PromQL-like `deriv` and `predict_linear`, respectively. [#84328](https://github.com/ClickHouse/ClickHouse/pull/84328) ([Stephen Chi](https://github.com/stephchi0)). +* Add two new TimeSeries functions: - `timeSeriesRange(start_timestamp, end_timestamp, step)`, - `timeSeriesFromGrid(start_timestamp, end_timestamp, step, values)`,. [#85435](https://github.com/ClickHouse/ClickHouse/pull/85435) ([Vitaly Baranov](https://github.com/vitlibar)). +* New syntax added `GRANT READ ON S3('s3://foo/.*') TO user`. [#84503](https://github.com/ClickHouse/ClickHouse/pull/84503) ([pufit](https://github.com/pufit)). +* Added `Hash` as a new output format. It calculates a single hash value for all columns and rows of the result. This is useful for calculating a "fingerprint" of the result, for example, in use cases where data transfer is a bottleneck. Example: `SELECT arrayJoin(['abc', 'def']), 42 FORMAT Hash` returns `e5f9e676db098fdb9530d2059d8c23ef`. [#84607](https://github.com/ClickHouse/ClickHouse/pull/84607) ([Robert Schulze](https://github.com/rschu1ze)). +* Add the ability to set up arbitrary watches in Keeper Multi queries. [#84964](https://github.com/ClickHouse/ClickHouse/pull/84964) ([Mikhail Artemenko](https://github.com/Michicosun)). +* Adds an option `--max-concurrency` for the `clickhouse-benchmark` tool that enables a mode with a gradual increase in the number of parallel queries. [#85623](https://github.com/ClickHouse/ClickHouse/pull/85623) ([Sergei Trifonov](https://github.com/serxa)). +* TODO: what's that? Support partially aggregated metrics. [#85328](https://github.com/ClickHouse/ClickHouse/pull/85328) ([Mikhail Artemenko](https://github.com/Michicosun)). + +#### Experimental Feature +* Enable correlated subqueries support by default, they are no longer experimental. [#85107](https://github.com/ClickHouse/ClickHouse/pull/85107) ([Dmitry Novik](https://github.com/novikd)). +* Unity, Glue, Rest, and Hive Metastore data lake catalogs are promoted from experimental to beta. [#85848](https://github.com/ClickHouse/ClickHouse/pull/85848) ([Melvyn Peignon](https://github.com/melvynator)). +* Lightweight updates and deletes are promoted from experimental to beta. +* Approximate vector search with vector similarity indexes is now GA. [#85888](https://github.com/ClickHouse/ClickHouse/pull/85888) ([Robert Schulze](https://github.com/rschu1ze)). +* Ytsaurus table engine and table function. [#77606](https://github.com/ClickHouse/ClickHouse/pull/77606) ([MikhailBurdukov](https://github.com/MikhailBurdukov)). +* Previously, the text index data would be separated into multiple segments (each segment size by default was 256 MiB). This might reduce the memory consumption while building the text index, however this increases the space requirement on the disk and increase the query response time. [#84590](https://github.com/ClickHouse/ClickHouse/pull/84590) ([Elmi Ahmadov](https://github.com/ahmadov)). + +#### Performance Improvement +* New parquet reader implementation. It's generally faster and supports page-level filter pushdown and PREWHERE. Currently experimental. Use setting `input_format_parquet_use_native_reader_v3` to enable. [#82789](https://github.com/ClickHouse/ClickHouse/pull/82789) ([Michael Kolupaev](https://github.com/al13n321)). +* Replaced the official HTTP transport in Azure library with our own HTTP client implementation for Azure Blob Storage. Introduced multiple settings for this clients which mirror settings from S3. Introduced aggressive connection timeouts for both Azure and S3. Improved introspection into Azure profile events and metrics. New client is enabled by default, provide much better latencies for cold queries on top of Azure Blob Storage. Old `Curl` client can be returned back by setting `azure_sdk_use_native_client=false`. [#83294](https://github.com/ClickHouse/ClickHouse/pull/83294) ([alesapin](https://github.com/alesapin)). The previous, official implementation of Azure client was unsuitable for production due to terrible latency spikes, ranging from five seconds to minutes. We have ditched that terrible implementation and are very proud of that. +* Processes indexes in increasing order of file size. The net index ordering prioritizes minmax and vector indexes (due to simplicity and selectivity respectively), and small indexes thereafter. Within the minmax/vector indexes smaller indexes are also preferred. [#84094](https://github.com/ClickHouse/ClickHouse/pull/84094) ([Maruth Goyal](https://github.com/maruthgoyal)). +* Enable MergeTree setting `write_marks_for_substreams_in_compact_parts` by default. It significantly improves performance of subcolumns reading from newly created Compact parts. Servers with version less then 25.5 won't be able to read new Compact parts. [#84171](https://github.com/ClickHouse/ClickHouse/pull/84171) ([Pavel Kruglov](https://github.com/Avogar)). +* `azureBlobStorage` table engine: cache and reuse managed identity authentication tokens when possible to avoid throttling. [#79860](https://github.com/ClickHouse/ClickHouse/pull/79860) ([Nick Blakely](https://github.com/niblak)). +* `ALL` `LEFT/INNER` JOINs will be automatically converted to `RightAny` if the right side is functionally determined by the join key columns (all rows have unique join key values). [#84010](https://github.com/ClickHouse/ClickHouse/pull/84010) ([Nikita Taranov](https://github.com/nickitat)). +* Add `max_joined_block_size_bytes` in addition to `max_joined_block_size_rows` to limit the memory usage of JOINs with heavy columns. [#83869](https://github.com/ClickHouse/ClickHouse/pull/83869) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Added new logic (controlled by the setting `enable_producing_buckets_out_of_order_in_aggregation`, enabled by default) that allows sending some buckets out of order during memory-efficient aggregation. When some aggregation buckets take significantly longer to merge than others, it improves performance by allowing the initiator to merge buckets with higher bucket id-s in the meantime. The downside is potentially higher memory usage (shouldn't be significant). [#80179](https://github.com/ClickHouse/ClickHouse/pull/80179) ([Nikita Taranov](https://github.com/nickitat)). +* Introduced the `optimize_rewrite_regexp_functions` setting (enabled by default), which allows the optimizer to rewrite certain `replaceRegexpAll`, `replaceRegexpOne`, and `extract` calls into simpler and more efficient forms when specific regular expression patterns are detected. (issue [#81981](https://github.com/ClickHouse/ClickHouse/issues/81981)). [#81992](https://github.com/ClickHouse/ClickHouse/pull/81992) ([Amos Bird](https://github.com/amosbird)). +* Process `max_joined_block_rows` outside of hash JOIN main loop. Slightly better performance for ALL JOIN. [#83216](https://github.com/ClickHouse/ClickHouse/pull/83216) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Process higher granularity min-max indexes first. Closes [#75381](https://github.com/ClickHouse/ClickHouse/issues/75381). [#83798](https://github.com/ClickHouse/ClickHouse/pull/83798) ([Maruth Goyal](https://github.com/maruthgoyal)). +* Make `DISTINCT` window aggregates run in linear time and fix a bug in `sumDistinct`. Closes [#79792](https://github.com/ClickHouse/ClickHouse/issues/79792). Closes [#52253](https://github.com/ClickHouse/ClickHouse/issues/52253). [#79859](https://github.com/ClickHouse/ClickHouse/pull/79859) ([Nihal Z. Miaji](https://github.com/nihalzp)). +* Vector search queries using a vector similarity index complete with lower latency due to reduced storage reads and reduced CPU usage. [#83803](https://github.com/ClickHouse/ClickHouse/pull/83803) ([Shankar Iyer](https://github.com/shankar-iyer)). +* Rendezvous hashing for improve cache locality of workload distribution among parallel replicas. [#82511](https://github.com/ClickHouse/ClickHouse/pull/82511) ([Anton Ivashkin](https://github.com/ianton-ru)). +* Implement addManyDefaults for If combinators, so now aggregate functions with If combinators work faster. [#83870](https://github.com/ClickHouse/ClickHouse/pull/83870) ([Raúl Marín](https://github.com/Algunenano)). +* Calculate serialized key columnarly when group by multiple string or number columns. [#83884](https://github.com/ClickHouse/ClickHouse/pull/83884) ([李扬](https://github.com/taiyang-li)). +* Eliminated full scans for the cases when index analysis results in empty ranges for parallel replicas reading. [#84971](https://github.com/ClickHouse/ClickHouse/pull/84971) ([Eduard Karacharov](https://github.com/korowa)). +* Try -falign-functions=64 in attempt for more stable perf tests. [#83920](https://github.com/ClickHouse/ClickHouse/pull/83920) ([Azat Khuzhin](https://github.com/azat)). +* The bloom filter index is now used for conditions like `has([c1, c2, ...], column)`, where `column` is not of an `Array` type. This improves performance for such queries, making them as efficient as the `IN` operator. [#83945](https://github.com/ClickHouse/ClickHouse/pull/83945) ([Doron David](https://github.com/dorki)). +* Reduce unnecessary memcpy calls in CompressedReadBufferBase::readCompressedData. [#83986](https://github.com/ClickHouse/ClickHouse/pull/83986) ([Raúl Marín](https://github.com/Algunenano)). +* Optimize `largestTriangleThreeBuckets` by removing temporary data. [#84479](https://github.com/ClickHouse/ClickHouse/pull/84479) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Optimize string deserialization by simplifying the code. Closes [#38564](https://github.com/ClickHouse/ClickHouse/issues/38564). [#84561](https://github.com/ClickHouse/ClickHouse/pull/84561) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Fixed the calculation of the minimal task size for parallel replicas. [#84752](https://github.com/ClickHouse/ClickHouse/pull/84752) ([Nikita Taranov](https://github.com/nickitat)). +* Improved performance of applying patch parts in `Join` mode. [#85040](https://github.com/ClickHouse/ClickHouse/pull/85040) ([Anton Popov](https://github.com/CurtizJ)). +* Remove zero byte. Closes [#85062](https://github.com/ClickHouse/ClickHouse/issues/85062). A few minor bugs were fixed. Functions `structureToProtobufSchema`, `structureToCapnProtoSchema` didn't correctly put a zero-terminating byte and were using a newline instead of it. That was leading to a missing newline in the output, and could lead to buffer overflows while using other functions that depend on the zero byte (such as `logTrace`, `demangle`, `extractURLParameter`, `toStringCutToZero`, and `encrypt`/`decrypt`). The `regexp_tree` dictionary layout didn't support processing strings with zero bytes. The `formatRowNoNewline` function, called with `Values` format or with any other format without a newline at the end of rows, erroneously cuts the last character of the output. Function `stem` contained an exception-safety error that could lead to a memory leak in a very rare scenario. The `initcap` function worked in the wrong way for `FixedString` arguments: it didn't recognize the start of the word at the start of the string if the previous string in a block ended with a word character. Fixed a security vulnerability of the Apache `ORC` format, which could lead to the exposure of uninitialized memory. Changed behavior of the function `replaceRegexpAll` and the corresponding alias, `REGEXP_REPLACE`: now it can do an empty match at the end of the string even if the previous match processed the whole string, such as in the case of `^a*|a*$` or `^|.*` - this corresponds to the semantic of JavaScript, Perl, Python, PHP, Ruby, but differs to the semantic of PostgreSQL. Implementation of many functions has been simplified and optimized. Documentation for several functions was wrong and has now been fixed. Keep in mind that the output of `byteSize` for String columns and complex types, which consisted of String columns, has changed (from 9 bytes per empty string to 8 bytes per empty string), and this is normal. [#85063](https://github.com/ClickHouse/ClickHouse/pull/85063) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Optimize the materialization of constants in cases when we do this materialization only to return a single row. [#85071](https://github.com/ClickHouse/ClickHouse/pull/85071) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Improve parallel files processing with delta-kernel-rs backend. [#85642](https://github.com/ClickHouse/ClickHouse/pull/85642) ([Azat Khuzhin](https://github.com/azat)). +* A new setting, enable_add_distinct_to_in_subqueries, has been introduced. When enabled, ClickHouse will automatically add DISTINCT to subqueries in IN clauses for distributed queries. This can significantly reduce the size of temporary tables transferred between shards and improve network efficiency. Note: This is a trade-off—while network transfer is reduced, additional merging (deduplication) work is required on each node. Enable this setting when network transfer is a bottleneck and the merging cost is acceptable. [#81908](https://github.com/ClickHouse/ClickHouse/pull/81908) ([fhw12345](https://github.com/fhw12345)). +* Reduce query memory tracking overhead for executable user-defined functions. [#83929](https://github.com/ClickHouse/ClickHouse/pull/83929) ([Eduard Karacharov](https://github.com/korowa)). +* Implement internal `delta-kernel-rs` filtering (statistics and partition pruning) in storage `DeltaLake`. [#84006](https://github.com/ClickHouse/ClickHouse/pull/84006) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Disable skipping indexes that depend on columns updated on the fly or by patch parts more granularly. Now, skipping indexes are not used only in parts affected by on-the-fly mutations or patch parts; previously, those indexes were disabled for all parts. [#84241](https://github.com/ClickHouse/ClickHouse/pull/84241) ([Anton Popov](https://github.com/CurtizJ)). +* Allocate the minimum amount of memory needed for encrypted_buffer for encrypted named collections. [#84432](https://github.com/ClickHouse/ClickHouse/pull/84432) ([Pablo Marcos](https://github.com/pamarcos)). +* Improved support for bloom filter indexes (regular, ngram, and token) to be utilized when the first argument is a constant array (the set) and the second is the indexed column (the subset), enabling more efficient query execution. [#84700](https://github.com/ClickHouse/ClickHouse/pull/84700) ([Doron David](https://github.com/dorki)). +* Reduce contention on storage lock in Keeper. [#84732](https://github.com/ClickHouse/ClickHouse/pull/84732) ([Antonio Andelic](https://github.com/antonio2368)). +* Add missing support of `read_in_order_use_virtual_row` for `WHERE`. It allows to skip reading more parts for queries with filters that were not fully pushed to `PREWHERE`. [#84835](https://github.com/ClickHouse/ClickHouse/pull/84835) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Allows asynchronously iterating objects from Iceberg table without storing objects for each data file explicitly. [#85369](https://github.com/ClickHouse/ClickHouse/pull/85369) ([Daniil Ivanik](https://github.com/divanik)). +* Execute non-correlated `EXISTS` as a scalar subquery. This allows using a scalar subquery cache and constant-folding the result, which is helpful for indexes. For compatibility, the new setting `execute_exists_as_scalar_subquery=1` is added. [#85481](https://github.com/ClickHouse/ClickHouse/pull/85481) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). + +#### Improvement +* Add `database_replicated` settings defining the default values of DatabaseReplicatedSettings. If the setting is not present in the Replicated DB create query, the value from this setting is used. [#85127](https://github.com/ClickHouse/ClickHouse/pull/85127) ([Tuan Pham Anh](https://github.com/tuanpach)). +* Made the table columns in the web UI (play) resizable. [#84012](https://github.com/ClickHouse/ClickHouse/pull/84012) ([Doron David](https://github.com/dorki)). +* Support compressed `.metadata.json` file via `iceberg_metadata_compression_method` setting. It supports all clickhouse compression methods. This closes [#84895](https://github.com/ClickHouse/ClickHouse/issues/84895). [#85196](https://github.com/ClickHouse/ClickHouse/pull/85196) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Show the number of ranges to be read in the output of `EXPLAIN indexes = 1`. [#79938](https://github.com/ClickHouse/ClickHouse/pull/79938) ([Christoph Wurm](https://github.com/cwurm)). +* Introduce settings to set ORC compression block size, and update its default value from 64KB to 256KB to keep consistent with spark or hive. [#80602](https://github.com/ClickHouse/ClickHouse/pull/80602) ([李扬](https://github.com/taiyang-li)). +* Add `columns_substreams.txt` file to Wide part to track all substreams stored in the part. It helps to track dynamic streams in JSON and Dynamic types and so avoid reading sample of these columns to get the list of dynamic streams (for example for columns sizes calculation). Also now all dynamic streams are reflected in `system.parts_columns`. [#81091](https://github.com/ClickHouse/ClickHouse/pull/81091) ([Pavel Kruglov](https://github.com/Avogar)). +* Add a CLI flag --show_secrets to clickhouse format to hide sensitive data by default. [#81524](https://github.com/ClickHouse/ClickHouse/pull/81524) ([Nikolai Ryzhov](https://github.com/Dolaxom)). +* S3 read and write requests are throttled on the HTTP socket level (instead of whole S3 requests) to avoid issues with `max_remote_read_network_bandwidth_for_server` and `max_remote_write_network_bandwidth_for_server` throttling. [#81837](https://github.com/ClickHouse/ClickHouse/pull/81837) ([Sergei Trifonov](https://github.com/serxa)). +* Allow to mix different collations for the same column in different windows (for window functions). [#82877](https://github.com/ClickHouse/ClickHouse/pull/82877) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* Add a tool to simulate, visualize and compare merge selectors. [#71496](https://github.com/ClickHouse/ClickHouse/pull/71496) ([Sergei Trifonov](https://github.com/serxa)). +* Add support of `remote*` table functions with parallel replicas if cluster is provided in `address_expression` argument. Also, fixes [#73295](https://github.com/ClickHouse/ClickHouse/issues/73295). [#82904](https://github.com/ClickHouse/ClickHouse/pull/82904) ([Igor Nikonov](https://github.com/devcrafter)). +* Set all log messages for writing backup files to TRACE. [#82907](https://github.com/ClickHouse/ClickHouse/pull/82907) ([Hans Krutzer](https://github.com/hkrutzer)). +* User-defined functions with unusual names and codecs can be formatted inconsistently by the SQL formatter. This closes [#83092](https://github.com/ClickHouse/ClickHouse/issues/83092). [#83644](https://github.com/ClickHouse/ClickHouse/pull/83644) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Users can now use Time and Time64 types inside the JSON type. [#83784](https://github.com/ClickHouse/ClickHouse/pull/83784) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Joins with parallel replicas now use the join logical step. In case of any issues with join queries using parallel replicas, try `SET query_plan_use_new_logical_join_step=0` and report an issue. [#83801](https://github.com/ClickHouse/ClickHouse/pull/83801) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Fix compatibility for cluster_function_process_archive_on_multiple_nodes. [#83968](https://github.com/ClickHouse/ClickHouse/pull/83968) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Support changing mv insert settings on `S3Queue` table level. Added new `S3Queue` level settings: `min_insert_block_size_rows_for_materialized_views` and `min_insert_block_size_bytes_for_materialized_views`. By default profile level settings will be used and `S3Queue` level settings will override those. [#83971](https://github.com/ClickHouse/ClickHouse/pull/83971) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Added profile event `MutationAffectedRowsUpperBound` that shows the number of affected rows in a mutation (e.g., the total number of rows that satisfy the condition in `ALTER UPDATE` or `ALTER DELETE` query. [#83978](https://github.com/ClickHouse/ClickHouse/pull/83978) ([Anton Popov](https://github.com/CurtizJ)). +* Use information from cgroup (if applicable, i.e. `memory_worker_use_cgroup` and cgroups are available) to adjust memory tracker (`memory_worker_correct_memory_tracker`). [#83981](https://github.com/ClickHouse/ClickHouse/pull/83981) ([Azat Khuzhin](https://github.com/azat)). +* MongoDB: Implicit parsing of strings to numeric types. Previously, if a string value was received from a MongoDB source for a numeric column in a ClickHouse table, an exception was thrown. Now, the engine attempts to parse the numeric value from the string automatically. Closes [#81167](https://github.com/ClickHouse/ClickHouse/issues/81167). [#84069](https://github.com/ClickHouse/ClickHouse/pull/84069) ([Kirill Nikiforov](https://github.com/allmazz)). +* Highlight digit groups in `Pretty` formats for `Nullable` numbers. [#84070](https://github.com/ClickHouse/ClickHouse/pull/84070) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Dashboard: the tooltip will not overflow the container at the top. [#84072](https://github.com/ClickHouse/ClickHouse/pull/84072) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Slightly better-looking dots on the dashboard. [#84074](https://github.com/ClickHouse/ClickHouse/pull/84074) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Dashboard now has a slightly better favicon. [#84076](https://github.com/ClickHouse/ClickHouse/pull/84076) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Web UI: Give browsers a chance to save the password. Also, it will remember the URL values. [#84087](https://github.com/ClickHouse/ClickHouse/pull/84087) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Add support for applying extra ACL on specific Keeper nodes using `apply_to_children` config. [#84137](https://github.com/ClickHouse/ClickHouse/pull/84137) ([Antonio Andelic](https://github.com/antonio2368)). +* Fix usage of "compact" Variant discriminators serialization in MergeTree. Perviously it wasn't used in some cases when it could be used. [#84141](https://github.com/ClickHouse/ClickHouse/pull/84141) ([Pavel Kruglov](https://github.com/Avogar)). +* Added a server setting, `logs_to_keep` to database replicated settings, that allows changing the default `logs_to_keep` parameter for replicated databases. Lower values reduce the number of ZNodes (especially if there are many databases), while higher values allow a missing replica to catch up after a longer period of time. [#84183](https://github.com/ClickHouse/ClickHouse/pull/84183) ([Alexey Khatskevich](https://github.com/Khatskevich)). +* Add a setting `json_type_escape_dots_in_keys` to escape dots in JSON keys during JSON type parsing. The setting is disabled by default. [#84207](https://github.com/ClickHouse/ClickHouse/pull/84207) ([Pavel Kruglov](https://github.com/Avogar)). +* Check if connection is cancelled before checking for EOF to prevent reading from closed connection. Fixes [#83893](https://github.com/ClickHouse/ClickHouse/issues/83893). [#84227](https://github.com/ClickHouse/ClickHouse/pull/84227) ([Raufs Dunamalijevs](https://github.com/rienath)). +* Slightly better colors of text selection in Web UI. The difference is significant only for selected table cells in the dark mode. In previous versions, there was not enough contrast between the text and the selection background. [#84258](https://github.com/ClickHouse/ClickHouse/pull/84258) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Improved server shutdown handling for client connections by simplifying internal checks. [#84312](https://github.com/ClickHouse/ClickHouse/pull/84312) ([Raufs Dunamalijevs](https://github.com/rienath)). +* Added a setting `delta_lake_enable_expression_visitor_logging` to turn off expression visitor logs as they can be too verbose even for test log level when debugging something. [#84315](https://github.com/ClickHouse/ClickHouse/pull/84315) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Cgroup-level and system-wide metrics are reported now altogether. Cgroup-level metrics have names `CGroup` and OS-level metrics (collected from procfs) have names `OS`. [#84317](https://github.com/ClickHouse/ClickHouse/pull/84317) ([Nikita Taranov](https://github.com/nickitat)). +* Slightly better charts in Web UI. Not much, but better. [#84326](https://github.com/ClickHouse/ClickHouse/pull/84326) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Change the default of the Replicated database setting `max_retries_before_automatic_recovery` to 10, so it will recover faster in some cases. [#84369](https://github.com/ClickHouse/ClickHouse/pull/84369) ([Alexander Tokmakov](https://github.com/tavplubix)). +* Fix formatting of CREATE USER with query parameters (i.e. `CREATE USER {username:Identifier} IDENTIFIED WITH no_password`). [#84376](https://github.com/ClickHouse/ClickHouse/pull/84376) ([Azat Khuzhin](https://github.com/azat)). +* Introduce `backup_restore_s3_retry_initial_backoff_ms`, `backup_restore_s3_retry_max_backoff_ms`, `backup_restore_s3_retry_jitter_factor` to configure the S3 retry backoff strategy used during backup and restore operations. [#84421](https://github.com/ClickHouse/ClickHouse/pull/84421) ([Julia Kartseva](https://github.com/jkartseva)). +* S3Queue ordered mode fix: quit earlier if shutdown was called. [#84463](https://github.com/ClickHouse/ClickHouse/pull/84463) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Support iceberg writes to read from pyiceberg. [#84466](https://github.com/ClickHouse/ClickHouse/pull/84466) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Allow set values type casting when pushing down `IN` / `GLOBAL IN` filters over KeyValue storage primary keys (e.g., EmbeddedRocksDB, KeeperMap). [#84515](https://github.com/ClickHouse/ClickHouse/pull/84515) ([Eduard Karacharov](https://github.com/korowa)). +* Bump chdig to [25.7.1](https://github.com/azat/chdig/releases/tag/v25.7.1). [#84521](https://github.com/ClickHouse/ClickHouse/pull/84521) ([Azat Khuzhin](https://github.com/azat)). +* Low-level errors during UDF execution now fail with error code `UDF_EXECUTION_FAILED`, whereas previously different error codes could be returned. [#84547](https://github.com/ClickHouse/ClickHouse/pull/84547) ([Xu Jia](https://github.com/XuJia0210)). +* Add `get_acl` command to KeeperClient. [#84641](https://github.com/ClickHouse/ClickHouse/pull/84641) ([Antonio Andelic](https://github.com/antonio2368)). +* Adds snapshot version to data lake table engines. [#84659](https://github.com/ClickHouse/ClickHouse/pull/84659) ([Pete Hampton](https://github.com/pjhampton)). +* Add a dimensional metric for the size of `ConcurrentBoundedQueue`, labelled by the queue type (i.e. what the queue is there for) and queue id (i.e. randomly generated id for the current instance of the queue). [#84675](https://github.com/ClickHouse/ClickHouse/pull/84675) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). +* The `system.columns` table now provides `column` as an alias for the existing `name` column. [#84695](https://github.com/ClickHouse/ClickHouse/pull/84695) ([Yunchi Pang](https://github.com/yunchipang)). +* New MergeTree setting `search_orphaned_parts_drives` to limit scope to look for parts e.g. by disks with local metadata. [#84710](https://github.com/ClickHouse/ClickHouse/pull/84710) ([Ilya Golshtein](https://github.com/ilejn)). +* Add 4LW in Keeper, `lgrq`, for toggling request logging of received requests. [#84719](https://github.com/ClickHouse/ClickHouse/pull/84719) ([Antonio Andelic](https://github.com/antonio2368)). +* Match external auth forward_headers in case-insensitive way. [#84737](https://github.com/ClickHouse/ClickHouse/pull/84737) ([ingodwerust](https://github.com/ingodwerust)). +* The `encrypt_decrypt` tool now supports encrypted ZooKeeper connections. [#84764](https://github.com/ClickHouse/ClickHouse/pull/84764) ([Roman Vasin](https://github.com/rvasin)). +* Add format string column to `system.errors`. This column is needed to group by the same error type in alerting rules. [#84776](https://github.com/ClickHouse/ClickHouse/pull/84776) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). +* Updated `clickhouse-format` to accept `--highlight` as an alias for `--hilite`. - Updated `clickhouse-client` to accept `--hilite` as an alias for `--highlight`. - Updated `clickhouse-format` documentation to reflect the change. [#84806](https://github.com/ClickHouse/ClickHouse/pull/84806) ([Rishabh Bhardwaj](https://github.com/rishabh1815769)). +* Fix iceberg reading by field ids for complex types. [#84821](https://github.com/ClickHouse/ClickHouse/pull/84821) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Introduce a new `backup_slow_all_threads_after_retryable_s3_error` setting to reduce pressure on S3 during retry storms caused by errors such as `SlowDown`, by slowing down all threads once a single retryable error is observed. [#84854](https://github.com/ClickHouse/ClickHouse/pull/84854) ([Julia Kartseva](https://github.com/jkartseva)). +* Skip creating and renaming the old temp table of non-append RMV DDLs in Replicated DBs. [#84858](https://github.com/ClickHouse/ClickHouse/pull/84858) ([Tuan Pham Anh](https://github.com/tuanpach)). +* Limit Keeper log entry cache size by number of entries using `keeper_server.coordination_settings.latest_logs_cache_entry_count_threshold` and `keeper_server.coordination_settings.commit_logs_cache_entry_count_threshold`. [#84877](https://github.com/ClickHouse/ClickHouse/pull/84877) ([Antonio Andelic](https://github.com/antonio2368)). +* Allow using `simdjson` on unsupported architectures (previously leads to `CANNOT_ALLOCATE_MEMORY` errors). [#84966](https://github.com/ClickHouse/ClickHouse/pull/84966) ([Azat Khuzhin](https://github.com/azat)). +* Async logging: Make limits tuneable and add introspection. [#85105](https://github.com/ClickHouse/ClickHouse/pull/85105) ([Raúl Marín](https://github.com/Algunenano)). +* Collect all removed objects to execute single object storage remove operation. [#85316](https://github.com/ClickHouse/ClickHouse/pull/85316) ([Mikhail Artemenko](https://github.com/Michicosun)). +* Iceberg's current implementation of positional delete files keeps all data in RAM. This can be quite expensive if the positional delete files are large, which is often the case. My implementation keeps only the last row-group of Parquet delete files in RAM, which is significantly cheaper. [#85329](https://github.com/ClickHouse/ClickHouse/pull/85329) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* chdig: fix leftovers on the screen, fix crash after edit query in editor, search in `path` for `editor`, update to [25.8.1](https://github.com/azat/chdig/releases/tag/v25.8.1). [#85341](https://github.com/ClickHouse/ClickHouse/pull/85341) ([Azat Khuzhin](https://github.com/azat)). +* Add missing `partition_columns_in_data_file` to azure configuration. [#85373](https://github.com/ClickHouse/ClickHouse/pull/85373) ([Arthur Passos](https://github.com/arthurpassos)). +* Allow zero step in functions `timeSeries*ToGrid` This is part of [#75036](https://github.com/ClickHouse/ClickHouse/pull/75036). [#85390](https://github.com/ClickHouse/ClickHouse/pull/85390) ([Vitaly Baranov](https://github.com/vitlibar)). +* Added show_data_lake_catalogs_in_system_tables flag to manage adding data lake tables in system.tables. Resolves [#85384](https://github.com/ClickHouse/ClickHouse/issues/85384). [#85411](https://github.com/ClickHouse/ClickHouse/pull/85411) ([Smita Kulkarni](https://github.com/SmitaRKulkarni)). +* Added support for macro expansion in `remote_fs_zero_copy_zookeeper_path`. [#85437](https://github.com/ClickHouse/ClickHouse/pull/85437) ([Mikhail Koviazin](https://github.com/mkmkme)). +* AI in clickhouse-client will look slightly better. [#85447](https://github.com/ClickHouse/ClickHouse/pull/85447) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Enable trace_log.symbolize for old deployments by default. [#85456](https://github.com/ClickHouse/ClickHouse/pull/85456) ([Azat Khuzhin](https://github.com/azat)). +* Support resolution of more cases for compound identifiers. Particularly, it improves the compatibility of `ARRAY JOIN` with the old analyzer. Introduce a new setting `analyzer_compatibility_allow_compound_identifiers_in_unflatten_nested` to keep the old behaviour. [#85492](https://github.com/ClickHouse/ClickHouse/pull/85492) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Ignore UNKNOWN_DATABASE while obtaining table columns sizes for system.columns. [#85632](https://github.com/ClickHouse/ClickHouse/pull/85632) ([Azat Khuzhin](https://github.com/azat)). +* Added a limit (table setting `max_uncompressed_bytes_in_patches`) for total uncompressed bytes in patch parts. It prevents significant slowdowns of SELECT queries after lightweight updates and prevents possible misuse of lightweight updates. [#85641](https://github.com/ClickHouse/ClickHouse/pull/85641) ([Anton Popov](https://github.com/CurtizJ)). +* Add a `parameter` column to `system.grants` to determine source type for `GRANT READ/WRITE` and the table engine for `GRANT TABLE ENGINE`. [#85643](https://github.com/ClickHouse/ClickHouse/pull/85643) ([MikhailBurdukov](https://github.com/MikhailBurdukov)). +* Fix parsing of a trailing comma in columns of the CREATE DICTIONARY query after a column with parameters, for example, Decimal(8). Closes [#85586](https://github.com/ClickHouse/ClickHouse/issues/85586). [#85653](https://github.com/ClickHouse/ClickHouse/pull/85653) ([Nikolay Degterinsky](https://github.com/evillique)). +* Support inner arrays for the function `nested`. [#85719](https://github.com/ClickHouse/ClickHouse/pull/85719) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* All the allocations done by external libraries are now visible to ClickHouse's memory tracker and accounted properly. This may result in "increased" reported memory usage for certain queries or failures with `MEMORY_LIMIT_EXCEEDED`. [#84082](https://github.com/ClickHouse/ClickHouse/pull/84082) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). + +#### Bug Fix (user-visible misbehavior in an official stable release) + +* This pr fixes the metadata resolution when querying iceberg tables through rest catalog. ... [#80562](https://github.com/ClickHouse/ClickHouse/pull/80562) ([Saurabh Kumar Ojha](https://github.com/saurabhojha)). +* Fix markReplicasActive in DDLWorker and DatabaseReplicatedDDLWorker. [#81395](https://github.com/ClickHouse/ClickHouse/pull/81395) ([Tuan Pham Anh](https://github.com/tuanpach)). +* Fix rollback of Dynamic column on parsing failure. [#82169](https://github.com/ClickHouse/ClickHouse/pull/82169) ([Pavel Kruglov](https://github.com/Avogar)). +* If function `trim` called with all-constant inputs now produces a constant output string. (Bug [#78796](https://github.com/ClickHouse/ClickHouse/issues/78796)). [#82900](https://github.com/ClickHouse/ClickHouse/pull/82900) ([Robert Schulze](https://github.com/rschu1ze)). +* Fix logical error with duplicate subqueries when `optimize_syntax_fuse_functions` is enabled, close [#75511](https://github.com/ClickHouse/ClickHouse/issues/75511). [#83300](https://github.com/ClickHouse/ClickHouse/pull/83300) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Fixed incorrect result of queries with `WHERE ... IN ()` clause and enabled query condition cache (setting `use_query_condition_cache`). [#83445](https://github.com/ClickHouse/ClickHouse/pull/83445) ([LB7666](https://github.com/acking-you)). +* Historically, `gcs` function did not require any access to use. Now it will check `GRANT READ ON S3` permission for usage. Closes [#70567](https://github.com/ClickHouse/ClickHouse/issues/70567). [#83503](https://github.com/ClickHouse/ClickHouse/pull/83503) ([pufit](https://github.com/pufit)). +* Skip unavailable nodes during INSERT SELECT from s3Cluster() into replicated MergeTree. [#83676](https://github.com/ClickHouse/ClickHouse/pull/83676) ([Igor Nikonov](https://github.com/devcrafter)). +* Fix write with append (in MergeTree used for experimental transactions) with `plain_rewritable`/`plain` metadata types, previously they were simply ignored. [#83695](https://github.com/ClickHouse/ClickHouse/pull/83695) ([Tuan Pham Anh](https://github.com/tuanpach)). +* Mask Avro schema registry authentication details to be not visible to user or in logs. [#83713](https://github.com/ClickHouse/ClickHouse/pull/83713) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). +* Fix the issue where, if a MergeTree table is created with `add_minmax_index_for_numeric_columns=1` or `add_minmax_index_for_string_columns=1`, the index is later materialized during an ALTER operation, and it prevents the Replicated database from initializing correctly on a new replica. [#83751](https://github.com/ClickHouse/ClickHouse/pull/83751) ([Nikolay Degterinsky](https://github.com/evillique)). +* Fixed parquet writer outputting incorrect statistics (min/max) for Decimal types. [#83754](https://github.com/ClickHouse/ClickHouse/pull/83754) ([Michael Kolupaev](https://github.com/al13n321)). +* Fix sort of NaN values in `LowCardinality(Float32|Float64|BFloat16)` type. [#83786](https://github.com/ClickHouse/ClickHouse/pull/83786) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* When restoring from backup, the definer user may not be backed up, which will cause the whole backup to be broken. To fix this, we postpone the permissions check on the target table's creation during restore and only check it during runtime. [#83818](https://github.com/ClickHouse/ClickHouse/pull/83818) ([pufit](https://github.com/pufit)). +* Fix crash in client due to connection left in disconnected state after bad INSERT. [#83842](https://github.com/ClickHouse/ClickHouse/pull/83842) ([Azat Khuzhin](https://github.com/azat)). +* Allow referencing any table in `view(...)` argument of `remote` table function with enabled analyzer. Fixes [#78717](https://github.com/ClickHouse/ClickHouse/issues/78717). Fixes [#79377](https://github.com/ClickHouse/ClickHouse/issues/79377). [#83844](https://github.com/ClickHouse/ClickHouse/pull/83844) ([Dmitry Novik](https://github.com/novikd)). +* Onprogress call in jsoneachrowwithprogress is synchronized with finalization. [#83879](https://github.com/ClickHouse/ClickHouse/pull/83879) ([Sema Checherinda](https://github.com/CheSema)). +* This closes [#81303](https://github.com/ClickHouse/ClickHouse/issues/81303). [#83892](https://github.com/ClickHouse/ClickHouse/pull/83892) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Fix colorSRGBToOKLCH/colorOKLCHToSRGB for mix of const and non-const args. [#83906](https://github.com/ClickHouse/ClickHouse/pull/83906) ([Azat Khuzhin](https://github.com/azat)). +* Fix writing JSON paths with NULL values in RowBinary format. [#83923](https://github.com/ClickHouse/ClickHouse/pull/83923) ([Pavel Kruglov](https://github.com/Avogar)). +* Overflow large values (>2106-02-07) when casting from Date to DateTime64 is fixed. [#83982](https://github.com/ClickHouse/ClickHouse/pull/83982) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Always apply `filesystem_prefetches_limit` (not only from `MergeTreePrefetchedReadPool`). [#83999](https://github.com/ClickHouse/ClickHouse/pull/83999) ([Azat Khuzhin](https://github.com/azat)). +* Fix rare bug when `MATERIALIZE COLUMN` query could lead to unexpected files in `checksums.txt` and eventually detached data parts. [#84007](https://github.com/ClickHouse/ClickHouse/pull/84007) ([alesapin](https://github.com/alesapin)). +* Fix the logical error `Expected single dictionary argument for function` while doing JOIN on an inequality condition when one of the columns is `LowCardinality` and the other is a constant. Closes [#81779](https://github.com/ClickHouse/ClickHouse/issues/81779). [#84019](https://github.com/ClickHouse/ClickHouse/pull/84019) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Fix crash with clickhouse client when used in interactive mode with syntax highlighting. [#84025](https://github.com/ClickHouse/ClickHouse/pull/84025) ([Bharat Nallan](https://github.com/bharatnc)). +* Fixed wrong results when the query condition cache is used in conjunction with recursive CTEs (issue [#81506](https://github.com/ClickHouse/ClickHouse/issues/81506)). [#84026](https://github.com/ClickHouse/ClickHouse/pull/84026) ([zhongyuankai](https://github.com/zhongyuankai)). +* Handle exceptions properly in periodic parts refresh. [#84083](https://github.com/ClickHouse/ClickHouse/pull/84083) ([Azat Khuzhin](https://github.com/azat)). +* Fix filter merging into JOIN condition in cases when equality operands have different types or they reference constants. Fixes [#83432](https://github.com/ClickHouse/ClickHouse/issues/83432). [#84145](https://github.com/ClickHouse/ClickHouse/pull/84145) ([Dmitry Novik](https://github.com/novikd)). +* Fix rare clickhouse crash when table has projection, `lightweight_mutation_projection_mode = 'rebuild'` and user execute lighweight delete which deletes ALL rows from any block in table. [#84158](https://github.com/ClickHouse/ClickHouse/pull/84158) ([alesapin](https://github.com/alesapin)). +* Fix deadlock caused by background cancellation checker thread. [#84203](https://github.com/ClickHouse/ClickHouse/pull/84203) ([Antonio Andelic](https://github.com/antonio2368)). +* Fix infinite recursive analysis of invalid `WINDOW` definitions. Fixes [#83131](https://github.com/ClickHouse/ClickHouse/issues/83131). [#84242](https://github.com/ClickHouse/ClickHouse/pull/84242) ([Dmitry Novik](https://github.com/novikd)). +* Fixed a bug that was causing incorrect Bech32 Encoding and Decoding. The bug wasn't caught originally due to an online implementation of the algorithm used for testing having the same issue. [#84257](https://github.com/ClickHouse/ClickHouse/pull/84257) ([George Larionov](https://github.com/george-larionov)). +* Fixed incorrect construction of empty tuples in the `array()` function. This fixes [#84202](https://github.com/ClickHouse/ClickHouse/issues/84202). [#84297](https://github.com/ClickHouse/ClickHouse/pull/84297) ([Amos Bird](https://github.com/amosbird)). +* Fix `LOGICAL_ERROR` for queries with parallel replicas and multiple INNER joins followed by RIGHT join. Do not use parallel replicas for such queries. [#84299](https://github.com/ClickHouse/ClickHouse/pull/84299) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Previously, `set` indexes didn't consider `Nullable` columns while checking if granules passed the filter (issue [#75485](https://github.com/ClickHouse/ClickHouse/issues/75485)). [#84305](https://github.com/ClickHouse/ClickHouse/pull/84305) ([Elmi Ahmadov](https://github.com/ahmadov)). +* Now ClickHouse read tables from Glue Catalog where table type specified in lower case. [#84316](https://github.com/ClickHouse/ClickHouse/pull/84316) ([alesapin](https://github.com/alesapin)). +* Do not try to substitute table functions to its cluster alternative in presence of a JOIN or subquery. [#84335](https://github.com/ClickHouse/ClickHouse/pull/84335) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Fix logger usage in `IAccessStorage`. [#84365](https://github.com/ClickHouse/ClickHouse/pull/84365) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Fixed a logical error in lightweight updates that update all columns in the table. [#84380](https://github.com/ClickHouse/ClickHouse/pull/84380) ([Anton Popov](https://github.com/CurtizJ)). +* Codec `DoubleDelta` codec can now only be applied to columns of numeric type. In particular `FixedString` columns can no longer be compressed using `DoubleDelta`. (fixes [#80220](https://github.com/ClickHouse/ClickHouse/issues/80220)). [#84383](https://github.com/ClickHouse/ClickHouse/pull/84383) ([Jimmy Aguilar Mena](https://github.com/Ergus)). +* The comparison against nan value was not using the correct ranges during `MinMax` index evaluation. [#84386](https://github.com/ClickHouse/ClickHouse/pull/84386) ([Elmi Ahmadov](https://github.com/ahmadov)). +* Fix reading Variant column with lazy materialization. [#84400](https://github.com/ClickHouse/ClickHouse/pull/84400) ([Pavel Kruglov](https://github.com/Avogar)). +* Make `zoutofmemory` hardware error, otherwise it will throw logical error. see https://github.com/clickhouse/clickhouse-core-incidents/issues/877. [#84420](https://github.com/ClickHouse/ClickHouse/pull/84420) ([Han Fei](https://github.com/hanfei1991)). +* Fixed server crash when a user created with `no_password` attempts to login after the server setting `allow_no_password` was changed to 0. [#84426](https://github.com/ClickHouse/ClickHouse/pull/84426) ([Shankar Iyer](https://github.com/shankar-iyer)). +* Fix out-of-order writes to Keeper changelog. Previously, we could have in-flight writes to changelog, but rollback could cause concurrent change of the destination file. This would lead to inconsistent logs, and possible data loss. [#84434](https://github.com/ClickHouse/ClickHouse/pull/84434) ([Antonio Andelic](https://github.com/antonio2368)). +* Now if all TTL are removed from table MergeTree will do nothing related to TTL. [#84441](https://github.com/ClickHouse/ClickHouse/pull/84441) ([alesapin](https://github.com/alesapin)). +* Parallel distributed INSERT SELECT with LIMIT was allowed which is not correct, it leads to data duplication in target table. [#84477](https://github.com/ClickHouse/ClickHouse/pull/84477) ([Igor Nikonov](https://github.com/devcrafter)). +* Fix pruning files by virtual column in data lakes. [#84520](https://github.com/ClickHouse/ClickHouse/pull/84520) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix leaks for keeper with rocksdb storage (iterators was not destroyed). [#84523](https://github.com/ClickHouse/ClickHouse/pull/84523) ([Azat Khuzhin](https://github.com/azat)). +* Fix ALTER MODIFY ORDER BY not validating TTL columns in sorting keys. TTL columns are now properly rejected when used in ORDER BY clauses during ALTER operations, preventing potential table corruption. [#84536](https://github.com/ClickHouse/ClickHouse/pull/84536) ([xiaohuanlin](https://github.com/xiaohuanlin)). +* Change pre-25.5 value of `allow_experimental_delta_kernel_rs` to `false` for compatibility. [#84587](https://github.com/ClickHouse/ClickHouse/pull/84587) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Stops taking schema from manifest files but stores relevant schemas for each snapshot independently. Infer relevant schema for each data file from its corresponding snapshot. Previous behaviour violated Iceberg specification for manifest files entries with existing status. [#84588](https://github.com/ClickHouse/ClickHouse/pull/84588) ([Daniil Ivanik](https://github.com/divanik)). +* Fixed issue where Keeper setting `rotate_log_storage_interval = 0` would cause ClickHouse to crash. (issue [#83975](https://github.com/ClickHouse/ClickHouse/issues/83975)). [#84637](https://github.com/ClickHouse/ClickHouse/pull/84637) ([George Larionov](https://github.com/george-larionov)). +* Fix logical error from S3Queue "Table is already registered". Closes [#84433](https://github.com/ClickHouse/ClickHouse/issues/84433). Broken after https://github.com/ClickHouse/ClickHouse/pull/83530. [#84677](https://github.com/ClickHouse/ClickHouse/pull/84677) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Lock 'mutex' when getting zookeeper from 'view' in RefreshTask. [#84699](https://github.com/ClickHouse/ClickHouse/pull/84699) ([Tuan Pham Anh](https://github.com/tuanpach)). +* Fix `CORRUPTED_DATA` error when lazy columns are used with external sort. [#84738](https://github.com/ClickHouse/ClickHouse/pull/84738) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). +* Fix column pruning with delta-kernel in storage `DeltaLake`. Closes [#84543](https://github.com/ClickHouse/ClickHouse/issues/84543). [#84745](https://github.com/ClickHouse/ClickHouse/pull/84745) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Refresh credentials in delta-kernel in storage DeltaLake. [#84751](https://github.com/ClickHouse/ClickHouse/pull/84751) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix starting superfluous internal backups after connection problems. [#84755](https://github.com/ClickHouse/ClickHouse/pull/84755) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fixed issue where querying a delayed remote source could result in vector out of bounds. [#84820](https://github.com/ClickHouse/ClickHouse/pull/84820) ([George Larionov](https://github.com/george-larionov)). +* The `ngram` and `no_op` tokenizers no longer crash the (experimental) text index for empty input tokens. [#84849](https://github.com/ClickHouse/ClickHouse/pull/84849) ([Robert Schulze](https://github.com/rschu1ze)). +* Fixed lightweight updates for tables with `ReplacingMergeTree` and `CollapsingMergeTree` engines. [#84851](https://github.com/ClickHouse/ClickHouse/pull/84851) ([Anton Popov](https://github.com/CurtizJ)). +* Correctly store all settings in table metadata for tables using object queue engine. [#84860](https://github.com/ClickHouse/ClickHouse/pull/84860) ([Antonio Andelic](https://github.com/antonio2368)). +* Fix total watches count returned by Keeper. [#84890](https://github.com/ClickHouse/ClickHouse/pull/84890) ([Antonio Andelic](https://github.com/antonio2368)). +* Fixed lightweight updates for tables with `ReplicatedMergeTree` engine created on servers with a version lower than 25.7. [#84933](https://github.com/ClickHouse/ClickHouse/pull/84933) ([Anton Popov](https://github.com/CurtizJ)). +* Fixed lightweight updates for tables with non-replicated `MergeTree` engine after running a `ALTER TABLE ... REPLACE PARTITION` query. [#84941](https://github.com/ClickHouse/ClickHouse/pull/84941) ([Anton Popov](https://github.com/CurtizJ)). +* Fixes column name generation for boolean literals to use "true"/"false" instead of "1"/"0", preventing column name conflicts between boolean and integer literals in queries. [#84945](https://github.com/ClickHouse/ClickHouse/pull/84945) ([xiaohuanlin](https://github.com/xiaohuanlin)). +* Fix memory tracking drift from background schedule pool and executor. [#84946](https://github.com/ClickHouse/ClickHouse/pull/84946) ([Azat Khuzhin](https://github.com/azat)). +* Fix potential inaccurate sorting issues in the Merge table engine. [#85025](https://github.com/ClickHouse/ClickHouse/pull/85025) ([Xiaozhe Yu](https://github.com/wudidapaopao)). +* Implement missing APIs for DiskEncrypted. [#85028](https://github.com/ClickHouse/ClickHouse/pull/85028) ([Azat Khuzhin](https://github.com/azat)). +* Add a check if a correlated subquery is used in a distributed context to avoid a crash. Fixes [#82205](https://github.com/ClickHouse/ClickHouse/issues/82205). [#85030](https://github.com/ClickHouse/ClickHouse/pull/85030) ([Dmitry Novik](https://github.com/novikd)). +* Now Iceberg doesn't try to cache relevant snapshot version between select queries and always try to resolve snapshot honestly. Earlier attempt to cache iceberg snapshot led to problems with usage of Iceberg table with time travel. [#85038](https://github.com/ClickHouse/ClickHouse/pull/85038) ([Daniil Ivanik](https://github.com/divanik)). +* Fixed double-free in `AzureIteratorAsync`. [#85064](https://github.com/ClickHouse/ClickHouse/pull/85064) ([Nikita Taranov](https://github.com/nickitat)). +* Improve error message on attempt to create user identified with JWT. [#85072](https://github.com/ClickHouse/ClickHouse/pull/85072) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Fixed cleanup of patch parts in `ReplicatedMergeTree`. Previously, the result of a lightweight update may temporarily not be visible on the replica until the merged or mutated part that materializes the patch parts is downloaded from another replica. [#85121](https://github.com/ClickHouse/ClickHouse/pull/85121) ([Anton Popov](https://github.com/CurtizJ)). +* Fixing illegal_type_of_argument in mv when types are different. [#85135](https://github.com/ClickHouse/ClickHouse/pull/85135) ([Sema Checherinda](https://github.com/CheSema)). +* Fix segfault in delta-kernel implementation. [#85160](https://github.com/ClickHouse/ClickHouse/pull/85160) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix recovering replicated databases when moving the metadata file takes a long time. [#85177](https://github.com/ClickHouse/ClickHouse/pull/85177) ([Tuan Pham Anh](https://github.com/tuanpach)). +* Fix `Not-ready Set` for `IN (subquery)` inside `additional_table_filters expression` setting. [#85210](https://github.com/ClickHouse/ClickHouse/pull/85210) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Get rid of unnecessary `getStatus()` calls during SYSTEM DROP REPLICA queries. Fixes the case when a table is dropped in the background, and the `Shutdown for storage is called` exception is thrown. [#85220](https://github.com/ClickHouse/ClickHouse/pull/85220) ([Nikolay Degterinsky](https://github.com/evillique)). +* Fix race in `DeltaLake` engine delta-kernel implementation. [#85221](https://github.com/ClickHouse/ClickHouse/pull/85221) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix reading partitioned data with disabled delta-kernel in `DeltaLake` engine. It was broken in 25.7 (https://github.com/ClickHouse/ClickHouse/pull/81136). [#85223](https://github.com/ClickHouse/ClickHouse/pull/85223) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Added missing table name length checks in CREATE OR REPLACE and RENAME queries. [#85326](https://github.com/ClickHouse/ClickHouse/pull/85326) ([Michael Kolupaev](https://github.com/al13n321)). +* Fix the creation of RMV on a new replica of the Replicated database if DEFINER is dropped. [#85327](https://github.com/ClickHouse/ClickHouse/pull/85327) ([Nikolay Degterinsky](https://github.com/evillique)). +* Fix iceberg writes for complex types. [#85330](https://github.com/ClickHouse/ClickHouse/pull/85330) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Writing lower and upper bounds are not supported for complex types. [#85332](https://github.com/ClickHouse/ClickHouse/pull/85332) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Fix logical error while reading from object storage functions through Distributed table or remote table function. Fixes: [#84658](https://github.com/ClickHouse/ClickHouse/issues/84658), Fixes [#85173](https://github.com/ClickHouse/ClickHouse/issues/85173), Fixes [#52022](https://github.com/ClickHouse/ClickHouse/issues/52022). [#85359](https://github.com/ClickHouse/ClickHouse/pull/85359) ([alesapin](https://github.com/alesapin)). +* Fix backup of parts with broken projections. [#85362](https://github.com/ClickHouse/ClickHouse/pull/85362) ([Antonio Andelic](https://github.com/antonio2368)). +* Forbid using `_part_offset` column in projection in releases until it is stabilized. [#85372](https://github.com/ClickHouse/ClickHouse/pull/85372) ([Sema Checherinda](https://github.com/CheSema)). +* Fix crash and data corruption during ALTER UPDATE for JSON. [#85383](https://github.com/ClickHouse/ClickHouse/pull/85383) ([Pavel Kruglov](https://github.com/Avogar)). +* Queries with parallel replicas which uses reading reverse in order optimization can produce incorrect result. [#85406](https://github.com/ClickHouse/ClickHouse/pull/85406) ([Igor Nikonov](https://github.com/devcrafter)). +* Fix possible UB (crashes) in case of MEMORY_LIMIT_EXCEEDED during String deserialization. [#85440](https://github.com/ClickHouse/ClickHouse/pull/85440) ([Azat Khuzhin](https://github.com/azat)). +* Fix incorrect metrics KafkaAssignedPartitions and KafkaConsumersWithAssignment. [#85494](https://github.com/ClickHouse/ClickHouse/pull/85494) ([Ilya Golshtein](https://github.com/ilejn)). +* Fixed processed bytes stat being underestimated when PREWHERE (explicit or automatic) is used. [#85495](https://github.com/ClickHouse/ClickHouse/pull/85495) ([Michael Kolupaev](https://github.com/al13n321)). +* Fix early return condition for S3 request rate slowdown: require either s3_slow_all_threads_after_network_error or backup_slow_all_threads_after_retryable_s3_error to be true to enable slowdown behavior when all threads are paused due to a retryable error, instead of requiring both. [#85505](https://github.com/ClickHouse/ClickHouse/pull/85505) ([Julia Kartseva](https://github.com/jkartseva)). +* This pr fixes the metadata resolution when querying iceberg tables through rest catalog. ... [#85531](https://github.com/ClickHouse/ClickHouse/pull/85531) ([Saurabh Kumar Ojha](https://github.com/saurabhojha)). +* Fixed rare crash in asynchronous inserts that change settings `log_comment` or `insert_deduplication_token`. [#85540](https://github.com/ClickHouse/ClickHouse/pull/85540) ([Anton Popov](https://github.com/CurtizJ)). +* Parameters like date_time_input_format were ignored when using HTTP with multipart/form-data. [#85570](https://github.com/ClickHouse/ClickHouse/pull/85570) ([Sema Checherinda](https://github.com/CheSema)). +* Fix secrets masking in icebergS3Cluster and icebergAzureCluster table functions. [#85658](https://github.com/ClickHouse/ClickHouse/pull/85658) ([MikhailBurdukov](https://github.com/MikhailBurdukov)). +* Fix precision loss in `JSONExtract` when converting JSON numbers to Decimal types. Now numeric JSON values preserve their exact decimal representation, avoiding floating-point rounding errors. [#85665](https://github.com/ClickHouse/ClickHouse/pull/85665) ([ssive7b](https://github.com/ssive7b)). +* Fixed `LOGICAL_ERROR` when using `COMMENT COLUMN IF EXISTS` in the same `ALTER` statement after `DROP COLUMN`. The `IF EXISTS` clause now correctly skips the comment operation when the column has been dropped within the same statement. [#85688](https://github.com/ClickHouse/ClickHouse/pull/85688) ([xiaohuanlin](https://github.com/xiaohuanlin)). +* Fix reading count from cache for delta lake. [#85704](https://github.com/ClickHouse/ClickHouse/pull/85704) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix coalescing merge tree segfault for large strings. This closes [#84582](https://github.com/ClickHouse/ClickHouse/issues/84582). [#85709](https://github.com/ClickHouse/ClickHouse/pull/85709) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Update metadata timestamp in iceberg writes. [#85711](https://github.com/ClickHouse/ClickHouse/pull/85711) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Using `distributed_depth` as an indicator of *Cluster function was incorrect and may lead to data duplication; use `client_info.collaborate_with_initiator` instead. [#85734](https://github.com/ClickHouse/ClickHouse/pull/85734) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Spark can't read position delete files. [#85762](https://github.com/ClickHouse/ClickHouse/pull/85762) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Fix send_logs_source_regexp (after async logging refactoring in [#85105](https://github.com/ClickHouse/ClickHouse/issues/85105)). [#85797](https://github.com/ClickHouse/ClickHouse/pull/85797) ([Azat Khuzhin](https://github.com/azat)). +* Fix possible inconsistency for dictionaries with update_field on MEMORY_LIMIT_EXCEEDED errors. [#85807](https://github.com/ClickHouse/ClickHouse/pull/85807) ([Azat Khuzhin](https://github.com/azat)). +* Support global constants from `WITH` statement for the parallel distributed `INSERT SELECT` with the `Distributed` destination table. Before, the query could throw an `Unknown expression identifier` error. [#85811](https://github.com/ClickHouse/ClickHouse/pull/85811) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Mask credentials for `deltaLakeAzure`, `deltaLakeCluster`, `icebergS3Cluster` and `icebergAzureCluster`. [#85889](https://github.com/ClickHouse/ClickHouse/pull/85889) ([Julian Maicher](https://github.com/jmaicher)). +* Fix logical error on attempt to `CREATE ... AS (SELECT * FROM s3Cluster(...))` with `DatabaseReplicated`. [#85904](https://github.com/ClickHouse/ClickHouse/pull/85904) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Fixes HTTP requests made by the `url()` table function to properly include port numbers in the Host header when accessing non-standard ports. This resolves authentication failures when using presigned URLs with S3-compatible services like MinIO running on custom ports, which is common in development environments. (Fixes [#85898](https://github.com/ClickHouse/ClickHouse/issues/85898)). [#85921](https://github.com/ClickHouse/ClickHouse/pull/85921) ([Tom Quist](https://github.com/tomquist)). +* Now unity catalog will ignore schemas with weird data types in case of non-delta tables. Fixes [#85699](https://github.com/ClickHouse/ClickHouse/issues/85699). [#85950](https://github.com/ClickHouse/ClickHouse/pull/85950) ([alesapin](https://github.com/alesapin)). +* Fix nullability of fields in iceberg. [#85977](https://github.com/ClickHouse/ClickHouse/pull/85977) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Fixed a bug in `Replicated` database recovery: if a table name contains the `%` symbol, it could re-create the table with a different name during recovery. [#85987](https://github.com/ClickHouse/ClickHouse/pull/85987) ([Alexander Tokmakov](https://github.com/tavplubix)). +* Fix backup restores failing due to `BACKUP_ENTRY_NOT_FOUND` error when restoring an empty `Memory` table. [#86012](https://github.com/ClickHouse/ClickHouse/pull/86012) ([Julia Kartseva](https://github.com/jkartseva)). +* Add checks for sharding_key during ALTER of the Distributed table. Previously incorrect ALTER would break the table definition and server restart. [#86015](https://github.com/ClickHouse/ClickHouse/pull/86015) ([Nikolay Degterinsky](https://github.com/evillique)). +* Don't create empty iceberg delete file. [#86061](https://github.com/ClickHouse/ClickHouse/pull/86061) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Fix large setting values breaking S3Queue tables and replica restart. [#86074](https://github.com/ClickHouse/ClickHouse/pull/86074) ([Nikolay Degterinsky](https://github.com/evillique)). + +#### Build/Testing/Packaging Improvement +* Use encrypted disks for tests with S3 by default. [#59898](https://github.com/ClickHouse/ClickHouse/pull/59898) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Use `clickhouse` binary in integrations tests to get unstripped debug symbols. [#83779](https://github.com/ClickHouse/ClickHouse/pull/83779) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). +* Bumped the internal libxml2 from 2.14.4 to 2.14.5. [#84230](https://github.com/ClickHouse/ClickHouse/pull/84230) ([Robert Schulze](https://github.com/rschu1ze)). +* Bumped internal curl from 8.14.0 to 8.15.0. [#84231](https://github.com/ClickHouse/ClickHouse/pull/84231) ([Robert Schulze](https://github.com/rschu1ze)). +* Now we use less memory for caches in CI and have better tests for eviction. [#84676](https://github.com/ClickHouse/ClickHouse/pull/84676) ([alesapin](https://github.com/alesapin)). + + +### ClickHouse release 25.7, 2025-07-24 {#257} + +#### Backward Incompatible Change +* Changes to `extractKeyValuePairs` function: introduce a new argument `unexpected_quoting_character_strategy` that controls what happens when a `quoting_character` is unexpectedly found when reading a non quoted key or value. The value can be one of: `invalid`, `accept` or `promote`. Invalid will discard the key and go back to waiting key state. Accept will treat it as part of the key. Promote will discard previous character and start parsing as a quoted key. In addition, after parsing a quoted value, only parse the next key if a pair delimiter is found. [#80657](https://github.com/ClickHouse/ClickHouse/pull/80657) ([Arthur Passos](https://github.com/arthurpassos)). +* Support zero-byte match in `countMatches` function. Users who like to retain the old behavior can enable setting `count_matches_stop_at_empty_match`. [#81676](https://github.com/ClickHouse/ClickHouse/pull/81676) ([Elmi Ahmadov](https://github.com/ahmadov)). +* Use server-wide throttlers for local (`max_local_read_bandwidth_for_server` and `max_local_write_bandwidth_for_server`) and remote (`max_remote_read_network_bandwidth_for_server` and `max_remote_write_network_bandwidth_for_server`) when generating BACKUPs in addition to their dedicated server settings (`max_backup_bandwidth_for_server`, `max_mutations_bandwidth_for_server` and `max_merges_bandwidth_for_server`). [#81753](https://github.com/ClickHouse/ClickHouse/pull/81753) ([Sergei Trifonov](https://github.com/serxa)). +* Forbid the creation of a table without insertable columns. [#81835](https://github.com/ClickHouse/ClickHouse/pull/81835) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* Parallelize cluster functions by files within archives. In previous versions, the whole archive (such as zip, tar, or 7z) was a unit of work. Added a new setting `cluster_function_process_archive_on_multiple_nodes`, by default equal to `true`. If set to `true`, increases performance of processing archives in cluster functions. Should be set to `false` for compatibility and to avoid errors during upgrade to 25.7+ if using cluster functions with archives on earlier versions. [#82355](https://github.com/ClickHouse/ClickHouse/pull/82355) ([Kseniia Sumarokova](https://github.com/kssenii)). +* `SYSTEM RESTART REPLICAS` query led to the wakeup of tables in the Lazy database, even without access to that database, and it happened while these tables were being concurrently dropped. Note: Now `SYSTEM RESTART REPLICAS` will only restart replicas in the databases where you have permission to `SHOW TABLES`, which is natural. [#83321](https://github.com/ClickHouse/ClickHouse/pull/83321) ([Alexey Milovidov](https://github.com/alexey-milovidov)). + +#### New Feature +* Added support for lightweight updates for `MergeTree`-family tables. Lightweight updates can be used by a new syntax: `UPDATE SET col1 = val1, col2 = val2, ... WHERE `. Added implementation of lightweight deletes via lightweight updates. It can be enabled by setting `lightweight_delete_mode = 'lightweight_update'`. [#82004](https://github.com/ClickHouse/ClickHouse/pull/82004) ([Anton Popov](https://github.com/CurtizJ)). +* Support complex types in Iceberg schema evolution. [#73714](https://github.com/ClickHouse/ClickHouse/pull/73714) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Introduce support for INSERTs into Iceberg tables. [#82692](https://github.com/ClickHouse/ClickHouse/pull/82692) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Read Iceberg data files by field ids. This improves compatibility with Iceberg: the fields can be renamed in the metadata while being mapped to the different names in the underlying Parquet files. This closes [#83065](https://github.com/ClickHouse/ClickHouse/issues/83065). [#83653](https://github.com/ClickHouse/ClickHouse/pull/83653) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Now clickhouse supports compressed `metadata.json` files for Iceberg. Fixes [#70874](https://github.com/ClickHouse/ClickHouse/issues/70874). [#81451](https://github.com/ClickHouse/ClickHouse/pull/81451) ([alesapin](https://github.com/alesapin)). +* Support `TimestampTZ` in Glue catalog. This closes [#81654](https://github.com/ClickHouse/ClickHouse/issues/81654). [#83132](https://github.com/ClickHouse/ClickHouse/pull/83132) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Add AI-powered SQL generation to ClickHouse client. Users can now generate SQL queries from natural language descriptions by prefixing their query with `??`. Supports OpenAI and Anthropic providers with automatic schema discovery. [#83314](https://github.com/ClickHouse/ClickHouse/pull/83314) ([Kaushik Iska](https://github.com/iskakaushik)). +* Add a function to write Geo types into WKB format. [#82935](https://github.com/ClickHouse/ClickHouse/pull/82935) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Introduced two new access types: `READ` and `WRITE` for sources and deprecates all previous access types related to sources. Before `GRANT S3 ON *.* TO user`, now: `GRANT READ, WRITE ON S3 TO user`. This also allows to separate `READ` and `WRITE` permissions for sources, e.g.: `GRANT READ ON * TO user`, `GRANT WRITE ON S3 TO user`. The feature is controlled by a setting `access_control_improvements.enable_read_write_grants` and disabled by default. [#73659](https://github.com/ClickHouse/ClickHouse/pull/73659) ([pufit](https://github.com/pufit)). +* NumericIndexedVector: new vector data-structure backed by bit-sliced, Roaring-bitmap compression, together with more than 20 functions for building, analysing and point-wise arithmetic. Can cut storage and speed up joins, filters and aggregations on sparse data. Implements [#70582](https://github.com/ClickHouse/ClickHouse/issues/70582) and [“Large-Scale Metric Computation in Online Controlled Experiment Platform” paper](https://arxiv.org/abs/2405.08411) by T. Xiong and Y. Wang from VLDB 2024. [#74193](https://github.com/ClickHouse/ClickHouse/pull/74193) ([FriendLey](https://github.com/FriendLey)). +* The workload setting `max_waiting_queries` is now supported. It can be used to limit the size of the query queue. If the limit is reached, all subsequent queries will be terminated with the `SERVER_OVERLOADED` error. [#81250](https://github.com/ClickHouse/ClickHouse/pull/81250) ([Oleg Doronin](https://github.com/dorooleg)). +* Add financial functions: `financialInternalRateOfReturnExtended` (`XIRR`), `financialInternalRateOfReturn` (`IRR`), `financialNetPresentValueExtended` (`XNPV`), `financialNetPresentValue` (`NPV`). [#81599](https://github.com/ClickHouse/ClickHouse/pull/81599) ([Joanna Hulboj](https://github.com/jh0x)). +* Add the geospatial functions `polygonsIntersectCartesian` and `polygonsIntersectSpherical` to check if two polygons intersect. [#81882](https://github.com/ClickHouse/ClickHouse/pull/81882) ([Paul Lamb](https://github.com/plamb)). +* Support `_part_granule_offset` virtual column in MergeTree-family tables. This column indicates the zero-based index of the granule/mark each row belongs to within its data part. This addresses [#79572](https://github.com/ClickHouse/ClickHouse/issues/79572). [#82341](https://github.com/ClickHouse/ClickHouse/pull/82341) ([Amos Bird](https://github.com/amosbird)). [#82341](https://github.com/ClickHouse/ClickHouse/pull/82341) ([Amos Bird](https://github.com/amosbird)) +* Added SQL functions `colorSRGBToOkLCH` and `colorOkLCHToSRGB` for converting colours between the sRGB and OkLCH colour spaces. [#83679](https://github.com/ClickHouse/ClickHouse/pull/83679) ([Fgrtue](https://github.com/Fgrtue)). +* Allow parameters in `CREATE USER` queries for usernames. [#81387](https://github.com/ClickHouse/ClickHouse/pull/81387) ([Diskein](https://github.com/Diskein)). +* The `system.formats` table now contains extended information about formats, such as HTTP content type, the capabilities of schema inference, etc. [#81505](https://github.com/ClickHouse/ClickHouse/pull/81505) ([Alexey Milovidov](https://github.com/alexey-milovidov)). + +#### Experimental Feature +* Added functions `searchAny` and `searchAll` which are general purpose tools to search text indexes. [#80641](https://github.com/ClickHouse/ClickHouse/pull/80641) ([Elmi Ahmadov](https://github.com/ahmadov)). +* The text index now supports the new `split` tokenizer. [#81752](https://github.com/ClickHouse/ClickHouse/pull/81752) ([Elmi Ahmadov](https://github.com/ahmadov)). +* Changed the default index granularity value for `text` indexes to 64. This improves the expected performance for the average test query in internal benchmarks. [#82162](https://github.com/ClickHouse/ClickHouse/pull/82162) ([Jimmy Aguilar Mena](https://github.com/Ergus)). +* The 256-bit bitmap stores the outgoing labels of a state ordered, but outgoing states are saved into disk in the order they appear in the hash table. Therefore, a label would point to a wrong next state while reading from disk. [#82783](https://github.com/ClickHouse/ClickHouse/pull/82783) ([Elmi Ahmadov](https://github.com/ahmadov)). +* Enable zstd compression for FST tree blob in text indices. [#83093](https://github.com/ClickHouse/ClickHouse/pull/83093) ([Elmi Ahmadov](https://github.com/ahmadov)). +* Promote vector similarity index to beta. Introduced alias setting `enable_vector_similarity_index` which must be enabled to use the vector similarity index. [#83459](https://github.com/ClickHouse/ClickHouse/pull/83459) ([Robert Schulze](https://github.com/rschu1ze)). +* Removed experimental `send_metadata` logic related to experimental zero-copy replication. It wasn't ever used and nobody supports this code. Since there were even no tests related to it, there is a high chance that it's broken long time ago. [#82508](https://github.com/ClickHouse/ClickHouse/pull/82508) ([alesapin](https://github.com/alesapin)). +* Integrate `StorageKafka2` to `system.kafka_consumers`. [#82652](https://github.com/ClickHouse/ClickHouse/pull/82652) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). +* Estimate complex CNF/DNF, for example, `(a < 1 and a > 0) or b = 3`, by statistics. [#82663](https://github.com/ClickHouse/ClickHouse/pull/82663) ([Han Fei](https://github.com/hanfei1991)). + +#### Performance Improvement +* Introduce async logging. When logs are output to a slow device, it no longer delays queries. [#82516](https://github.com/ClickHouse/ClickHouse/pull/82516) ([Raúl Marín](https://github.com/Algunenano)). Limit the max number of entries that are hold in the queue. [#83214](https://github.com/ClickHouse/ClickHouse/pull/83214) ([Raúl Marín](https://github.com/Algunenano)). +* Parallel distributed INSERT SELECT is enabled by default in mode where INSERT SELECT executed on each shard independently, see `parallel_distributed_insert_select` setting. [#83040](https://github.com/ClickHouse/ClickHouse/pull/83040) ([Igor Nikonov](https://github.com/devcrafter)). +* When the aggregation query contains only a single `count()` function on a not-`Nullable` column, the aggregation logic is fully inlined during hash table probing. This avoids allocating and maintaining any aggregation state, significantly reducing memory usage and CPU overhead. This partially addresses [#81982](https://github.com/ClickHouse/ClickHouse/issues/81982). [#82104](https://github.com/ClickHouse/ClickHouse/pull/82104) ([Amos Bird](https://github.com/amosbird)). +* Performance of `HashJoin` optimised by removing the additional loop over hash maps in the typical case of only one key column, also `null_map` and `join_mask` checks are eliminated when they're always `true`/`false`. [#82308](https://github.com/ClickHouse/ClickHouse/pull/82308) ([Nikita Taranov](https://github.com/nickitat)). +* Trivial optimization for `-If` combinator. [#78454](https://github.com/ClickHouse/ClickHouse/pull/78454) ([李扬](https://github.com/taiyang-li)). +* Vector search queries using a vector similarity index complete with lower latency due to reduced storage reads and reduced CPU usage. [#79103](https://github.com/ClickHouse/ClickHouse/pull/79103) ([Shankar Iyer](https://github.com/shankar-iyer)). +* Respect `merge_tree_min_{rows,bytes}_for_seek` in `filterPartsByQueryConditionCache` to align it with other methods filtering by indexes. [#80312](https://github.com/ClickHouse/ClickHouse/pull/80312) ([李扬](https://github.com/taiyang-li)). +* Make the pipeline after the `TOTALS` step multithreaded. [#80331](https://github.com/ClickHouse/ClickHouse/pull/80331) ([UnamedRus](https://github.com/UnamedRus)). +* Fix filter by key for `Redis` and `KeeperMap` storages. [#81833](https://github.com/ClickHouse/ClickHouse/pull/81833) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* Add new setting `min_joined_block_size_rows` (analogous to `min_joined_block_size_bytes`; defaults to 65409) to control the minimum block size (in rows) for JOIN input and output blocks (if the join algorithm supports it). Small blocks will be squashed. [#81886](https://github.com/ClickHouse/ClickHouse/pull/81886) ([Nikita Taranov](https://github.com/nickitat)). +* `ATTACH PARTITION` no longer leads to the dropping of all caches. [#82377](https://github.com/ClickHouse/ClickHouse/pull/82377) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Optimize the generated plan for correlated subqueries by removing redundant JOIN operations using equivalence classes. If there are equivalent expressions for all correlated columns, `CROSS JOIN` is not produced if `query_plan_correlated_subqueries_use_substitution` setting is enabled. [#82435](https://github.com/ClickHouse/ClickHouse/pull/82435) ([Dmitry Novik](https://github.com/novikd)). +* Read only required columns in correlated subquery when it appears to be an argument of function `EXISTS`. [#82443](https://github.com/ClickHouse/ClickHouse/pull/82443) ([Dmitry Novik](https://github.com/novikd)). +* Speedup comparisons of query trees during the query analysis a bit. [#82617](https://github.com/ClickHouse/ClickHouse/pull/82617) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Add alignment in the Counter of ProfileEvents to reduce false sharing. [#82697](https://github.com/ClickHouse/ClickHouse/pull/82697) ([Jiebin Sun](https://github.com/jiebinn)). +* The optimizations for `null_map` and `JoinMask` from [#82308](https://github.com/ClickHouse/ClickHouse/issues/82308) were applied to the case of JOIN with multiple disjuncts. Also, the `KnownRowsHolder` data structure was optimized. [#83041](https://github.com/ClickHouse/ClickHouse/pull/83041) ([Nikita Taranov](https://github.com/nickitat)). +* Plain `std::vector` is used for join flags to avoid calculating a hash on each access to flags. [#83043](https://github.com/ClickHouse/ClickHouse/pull/83043) ([Nikita Taranov](https://github.com/nickitat)). +* Don't pre-allocate memory for result columns beforehand when `HashJoin` uses `lazy` output mode. It is suboptimal, especially when the number of matches is low. Moreover, we know the exact amount of matches after joining is done, so we can preallocate more precisely. [#83304](https://github.com/ClickHouse/ClickHouse/pull/83304) ([Nikita Taranov](https://github.com/nickitat)). +* Minimize memory copy in port headers during pipeline construction. Original [PR](https://github.com/ClickHouse/ClickHouse/pull/70105) by [heymind](https://github.com/heymind). [#83381](https://github.com/ClickHouse/ClickHouse/pull/83381) ([Raúl Marín](https://github.com/Algunenano)). +* Improve the startup of clickhouse-keeper when it uses rocksdb storage. [#83390](https://github.com/ClickHouse/ClickHouse/pull/83390) ([Antonio Andelic](https://github.com/antonio2368)). +* Avoid holding the lock while creating storage snapshot data to reduce lock contention with high concurrent load. [#83510](https://github.com/ClickHouse/ClickHouse/pull/83510) ([Duc Canh Le](https://github.com/canhld94)). +* Improved performance of the `ProtobufSingle` input format by reusing the serializer when no parsing errors occur. [#83613](https://github.com/ClickHouse/ClickHouse/pull/83613) ([Eduard Karacharov](https://github.com/korowa)). +* Improve the performance of pipeline building that speeds up short queries. [#83631](https://github.com/ClickHouse/ClickHouse/pull/83631) ([Raúl Marín](https://github.com/Algunenano)). +* Optimize `MergeTreeReadersChain::getSampleBlock` that speeds up short queries. [#83875](https://github.com/ClickHouse/ClickHouse/pull/83875) ([Raúl Marín](https://github.com/Algunenano)). +* Speedup tables listing in data catalogs by asynchronous requests. [#81084](https://github.com/ClickHouse/ClickHouse/pull/81084) ([alesapin](https://github.com/alesapin)). +* Introduce jitter to the S3 retry mechanism when the `s3_slow_all_threads_after_network_error` configuration is enabled. [#81849](https://github.com/ClickHouse/ClickHouse/pull/81849) ([zoomxi](https://github.com/zoomxi)). + +#### Improvement +* Color parenthesis in multiple colors for better readability. [#82538](https://github.com/ClickHouse/ClickHouse/pull/82538) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Highlight metacharacters in LIKE/REGEXP patterns as you type. We already have it in `clickhouse-format` and in the echo in `clickhouse-client`, but now it is done in the command prompt as well. [#82871](https://github.com/ClickHouse/ClickHouse/pull/82871) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Highlighting in `clickhouse-format` and in the client's echo will work in the same way as the highlighting in the command line prompt. [#82874](https://github.com/ClickHouse/ClickHouse/pull/82874) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Now `plain_rewritable` disks are allowed as disks for database metadata. Implement methods `moveFile` and `replaceFile` in `plain_rewritable` to support it as a database disk. [#79424](https://github.com/ClickHouse/ClickHouse/pull/79424) ([Tuan Pham Anh](https://github.com/tuanpach)). +* Allow backups for `PostgreSQL`, `MySQL` and `DataLake` databases. A backup of such a database would only save the definition and not the data inside of it. [#79982](https://github.com/ClickHouse/ClickHouse/pull/79982) ([Nikolay Degterinsky](https://github.com/evillique)). +* Setting `allow_experimental_join_condition` marked as obsolete, because it is now always allowed. [#80566](https://github.com/ClickHouse/ClickHouse/pull/80566) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Add pressure metrics to ClickHouse async metrics. [#80779](https://github.com/ClickHouse/ClickHouse/pull/80779) ([Xander Garbett](https://github.com/Garbett1)). +* Added metrics `MarkCacheEvictedBytes`, `MarkCacheEvictedMarks`, `MarkCacheEvictedFiles` for tracking evictions from the mark cache. (issue [#60989](https://github.com/ClickHouse/ClickHouse/issues/60989)). [#80799](https://github.com/ClickHouse/ClickHouse/pull/80799) ([Shivji Kumar Jha](https://github.com/shiv4289)). +* Support writing Parquet enum as byte array as the [spec](https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#enum) dictates. [#81090](https://github.com/ClickHouse/ClickHouse/pull/81090) ([Arthur Passos](https://github.com/arthurpassos)). +* An improvement for `DeltaLake` table engine: delta-kernel-rs has `ExpressionVisitor` API which is implemented in this PR and is applied to partition column expressions transform (it will replace an old deprecated within the delta-kernel-rs way, which was used before in our code). In the future this `ExpressionVisitor` will also allow to implement statistics based pruning and some delta-lake proprietary features. Additionally the purpose of this change is to support partition pruning in `DeltaLakeCluster` table engine (the result of a parsed expression - ActionsDAG - will be serialized and sent from the initiator along with the data path, because this kind of information, which is needed for pruning, is only available as meta information on data files listing, which is done by initiator only, but it has to be applied to data on each reading server). [#81136](https://github.com/ClickHouse/ClickHouse/pull/81136) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Preserve element names when deriving supertypes for named tuples. [#81345](https://github.com/ClickHouse/ClickHouse/pull/81345) ([lgbo](https://github.com/lgbo-ustc)). +* Count consumed messages manually to avoid depending on previous committed offset in StorageKafka2. [#81662](https://github.com/ClickHouse/ClickHouse/pull/81662) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). +* Added `clickhouse-keeper-utils`, a new command-line tool for managing and analyzing ClickHouse Keeper data. The tool supports dumping state from snapshots and changelogs, analyzing changelog files, and extracting specific log ranges. [#81677](https://github.com/ClickHouse/ClickHouse/pull/81677) ([Antonio Andelic](https://github.com/antonio2368)). +* The total and per-user network throttlers are never reset, which ensures that `max_network_bandwidth_for_all_users` and `max_network_bandwidth_for_all_users` limits are never exceeded. [#81729](https://github.com/ClickHouse/ClickHouse/pull/81729) ([Sergei Trifonov](https://github.com/serxa)). +* Support writing geoparquets as output format. [#81784](https://github.com/ClickHouse/ClickHouse/pull/81784) ([Konstantin Vedernikov](https://github.com/scanhex12)). +* Forbid to start `RENAME COLUMN` alter mutation if it will rename some column that right now affected by incomplete data mutation. [#81823](https://github.com/ClickHouse/ClickHouse/pull/81823) ([Mikhail Artemenko](https://github.com/Michicosun)). +* Header Connection is send at the end of headers. When we know is the connection should be preserved. [#81951](https://github.com/ClickHouse/ClickHouse/pull/81951) ([Sema Checherinda](https://github.com/CheSema)). +* Tune TCP servers queue (64 by default) based on listen_backlog (4096 by default). [#82045](https://github.com/ClickHouse/ClickHouse/pull/82045) ([Azat Khuzhin](https://github.com/azat)). +* Add ability to reload `max_local_read_bandwidth_for_server` and `max_local_write_bandwidth_for_server` on fly without restart server. [#82083](https://github.com/ClickHouse/ClickHouse/pull/82083) ([Kai Zhu](https://github.com/nauu)). +* Add support for clearing all warnings from the `system.warnings` table using `TRUNCATE TABLE system.warnings`. [#82087](https://github.com/ClickHouse/ClickHouse/pull/82087) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Fix partition pruning with data lake cluster functions. [#82131](https://github.com/ClickHouse/ClickHouse/pull/82131) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix reading partitioned data in DeltaLakeCluster table function. In this PR cluster functions protocol version is increased, allowing to send extra info from initiator to replicas. This extra info contains delta-kernel transform expression, which is needed to parse partition columns (and some other staff in the future, like generated columns, etc). [#82132](https://github.com/ClickHouse/ClickHouse/pull/82132) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Function `reinterpret` function now supports conversion to `Array(T)` where `T` is a fixed-size data type (issue [#82621](https://github.com/ClickHouse/ClickHouse/issues/82621)). [#83399](https://github.com/ClickHouse/ClickHouse/pull/83399) ([Shankar Iyer](https://github.com/shankar-iyer)). +* Now database Datalake throw more convenient exception. Fixes [#81211](https://github.com/ClickHouse/ClickHouse/issues/81211). [#82304](https://github.com/ClickHouse/ClickHouse/pull/82304) ([alesapin](https://github.com/alesapin)). +* Improve CROSS JOIN by returning false from `HashJoin::needUsedFlagsForPerRightTableRow`. [#82379](https://github.com/ClickHouse/ClickHouse/pull/82379) ([lgbo](https://github.com/lgbo-ustc)). +* Allow write/read map columns as Array of Tuples. [#82408](https://github.com/ClickHouse/ClickHouse/pull/82408) ([MikhailBurdukov](https://github.com/MikhailBurdukov)). +* List the licenses of [Rust](https://clickhouse.com/blog/rust) crates in `system.licenses`. [#82440](https://github.com/ClickHouse/ClickHouse/pull/82440) ([Raúl Marín](https://github.com/Algunenano)). +* Macros like `{uuid}` can now be used in the `keeper_path` setting of the S3Queue table engine. [#82463](https://github.com/ClickHouse/ClickHouse/pull/82463) ([Nikolay Degterinsky](https://github.com/evillique)). +* Keeper improvement: move changelog files between disk in a background thread. Previously, moving changelog to a different disk would block Keeper globally until the move is finished. This lead to performance degradation if moving is a long operation (e.g. to S3 disk). [#82485](https://github.com/ClickHouse/ClickHouse/pull/82485) ([Antonio Andelic](https://github.com/antonio2368)). +* Keeper improvement: add new config `keeper_server.cleanup_old_and_ignore_new_acl`. If enabled, all nodes will have their ACLs cleared while ACL for new requests will be ignored. If the goal is to completely remove ACL from nodes, it's important to leave the config enabled until a new snapshot is created. [#82496](https://github.com/ClickHouse/ClickHouse/pull/82496) ([Antonio Andelic](https://github.com/antonio2368)). +* Added a new server setting `s3queue_disable_streaming` which disables streaming in tables with S3Queue table engine. This setting is changeable without server restart. [#82515](https://github.com/ClickHouse/ClickHouse/pull/82515) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Refactor dynamic resize feature of filesystem cache. Added more logs for introspection. [#82556](https://github.com/ClickHouse/ClickHouse/pull/82556) ([Kseniia Sumarokova](https://github.com/kssenii)). +* `clickhouse-server` without a configuration file will also listen to the PostgreSQL port 9005, like with the default config. [#82633](https://github.com/ClickHouse/ClickHouse/pull/82633) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* In `ReplicatedMergeTree::executeMetadataAlter`, we get the StorageID, and without taking DDLGuard, we try to call `IDatabase::alterTable`. In between this time we could have technically exchanged the table in question with another table, so when we get the definiton we would get the wrong one. To avoid this we add a separate check for UUIDs to match when we try to call `IDatabase::alterTable`. [#82666](https://github.com/ClickHouse/ClickHouse/pull/82666) ([Nikolay Degterinsky](https://github.com/evillique)). +* When attaching a database with a read-only remote disk, manually add table UUIDs into DatabaseCatalog. [#82670](https://github.com/ClickHouse/ClickHouse/pull/82670) ([Tuan Pham Anh](https://github.com/tuanpach)). +* Prevent user from using `nan` and `inf` with `NumericIndexedVector`. Fixes [#82239](https://github.com/ClickHouse/ClickHouse/issues/82239) and a little more. [#82681](https://github.com/ClickHouse/ClickHouse/pull/82681) ([Raufs Dunamalijevs](https://github.com/rienath)). +* Do not omit zero values in the `X-ClickHouse-Progress` and `X-ClickHouse-Summary` header formats. [#82727](https://github.com/ClickHouse/ClickHouse/pull/82727) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Keeper improvement: support specific permissions for world:anyone ACL. [#82755](https://github.com/ClickHouse/ClickHouse/pull/82755) ([Antonio Andelic](https://github.com/antonio2368)). +* Do not allow `RENAME COLUMN` or `DROP COLUMN` involving explicitly listed columns to sum in SummingMergeTree. Closes [#81836](https://github.com/ClickHouse/ClickHouse/issues/81836). [#82821](https://github.com/ClickHouse/ClickHouse/pull/82821) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Improve the precision of conversion from `Decimal` to `Float32`. Implement conversion from `Decimal` to `BFloat16`. Closes [#82660](https://github.com/ClickHouse/ClickHouse/issues/82660). [#82823](https://github.com/ClickHouse/ClickHouse/pull/82823) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Scrollbars in the Web UI will look slightly better. [#82869](https://github.com/ClickHouse/ClickHouse/pull/82869) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* `clickhouse-server` with embedded configuration will allow using the Web UI by providing an HTTP OPTIONS response. [#82870](https://github.com/ClickHouse/ClickHouse/pull/82870) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Add support for specifying extra Keeper ACL for paths in config. If you want to add extra ACL for a specific path you define it in the config under `zookeeper.path_acls`. [#82898](https://github.com/ClickHouse/ClickHouse/pull/82898) ([Antonio Andelic](https://github.com/antonio2368)). +* Now mutations snapshot will be built from the visible parts snapshot. Also mutation counters used in snapshot will be recalculated from the included mutations. [#82945](https://github.com/ClickHouse/ClickHouse/pull/82945) ([Mikhail Artemenko](https://github.com/Michicosun)). +* Adds ProfileEvent when Keeper rejects a write due to soft memory limit. [#82963](https://github.com/ClickHouse/ClickHouse/pull/82963) ([Xander Garbett](https://github.com/Garbett1)). +* Add columns `commit_time`, `commit_id` to `system.s3queue_log`. [#83016](https://github.com/ClickHouse/ClickHouse/pull/83016) ([Kseniia Sumarokova](https://github.com/kssenii)). +* In some cases, we need to have multiple dimensions to our metrics. For example, counting failed merges or mutations by error codes rather than having a single counter. Introduce `system.dimensional_metrics`, which does precisely that and adds the first dimensional metric called `failed_merges`. [#83030](https://github.com/ClickHouse/ClickHouse/pull/83030) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). +* Consolidate unknown settings warnings in clickhouse client and log them as a summary. [#83042](https://github.com/ClickHouse/ClickHouse/pull/83042) ([Bharat Nallan](https://github.com/bharatnc)). +* Clickhouse client now reports the local port when connection error happens. [#83050](https://github.com/ClickHouse/ClickHouse/pull/83050) ([Jianfei Hu](https://github.com/incfly)). +* Slightly better error handling in `AsynchronousMetrics`. If the `/sys/block` directory exists but is not accessible, the server will start without monitoring the block devices. Closes [#79229](https://github.com/ClickHouse/ClickHouse/issues/79229). [#83115](https://github.com/ClickHouse/ClickHouse/pull/83115) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Shutdown SystemLogs after ordinary tables (and before system tables, instead of before ordinary). [#83134](https://github.com/ClickHouse/ClickHouse/pull/83134) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Add logs for `S3Queue` shutdown process. [#83163](https://github.com/ClickHouse/ClickHouse/pull/83163) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Possibility to parse `Time` and `Time64` as `MM:SS`, `M:SS`, `SS`, or `S`. [#83299](https://github.com/ClickHouse/ClickHouse/pull/83299) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* When `distributed_ddl_output_mode='*_only_active'`, don't wait for new or recovered replicas that have replication lag bigger than `max_replication_lag_to_enqueue`. This should help to avoid `DDL task is not finished on some hosts` when a new replica becomes active after finishing initialization or recovery, but it accumulated huge replication log while initializing. Also, implement `SYSTEM SYNC DATABASE REPLICA STRICT` query that waits for replication log to become below `max_replication_lag_to_enqueue`. [#83302](https://github.com/ClickHouse/ClickHouse/pull/83302) ([Alexander Tokmakov](https://github.com/tavplubix)). +* Do not output too long descriptions of expression actions in exception messages. Closes [#83164](https://github.com/ClickHouse/ClickHouse/issues/83164). [#83350](https://github.com/ClickHouse/ClickHouse/pull/83350) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Add ability to parse part's prefix and suffix and also check coverage for non constant columns. [#83377](https://github.com/ClickHouse/ClickHouse/pull/83377) ([Mikhail Artemenko](https://github.com/Michicosun)). +* Unify parameter names in ODBC and JDBC when using named collections. [#83410](https://github.com/ClickHouse/ClickHouse/pull/83410) ([Andrey Zvonov](https://github.com/zvonand)). +* When the storage is shutting down, `getStatus` throws an `ErrorCodes::ABORTED` exception. Previously, this would fail the select query. Now we catch the `ErrorCodes::ABORTED` exceptions and intentionally ignore them instead. [#83435](https://github.com/ClickHouse/ClickHouse/pull/83435) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). +* Add process resource metrics (such as `UserTimeMicroseconds`, `SystemTimeMicroseconds`, `RealTimeMicroseconds`) to part_log profile events for `MergeParts` entries. [#83460](https://github.com/ClickHouse/ClickHouse/pull/83460) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Enable `create_if_not_exists`, `check_not_exists`, `remove_recursive` feature flags in Keeper by default which enable new types of requests. [#83488](https://github.com/ClickHouse/ClickHouse/pull/83488) ([Antonio Andelic](https://github.com/antonio2368)). +* Shutdown S3(Azure/etc)Queue streaming before shutting down any tables on server shutdown. [#83530](https://github.com/ClickHouse/ClickHouse/pull/83530) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Enable `Date`/`Date32` as integers in `JSON` input formats. [#83597](https://github.com/ClickHouse/ClickHouse/pull/83597) ([MikhailBurdukov](https://github.com/MikhailBurdukov)). +* Made exception messages for certain situations for loading and adding projections easier to read. [#83728](https://github.com/ClickHouse/ClickHouse/pull/83728) ([Robert Schulze](https://github.com/rschu1ze)). +* Introduce a configuration option to skip binary checksum integrity checks for `clickhouse-server`. Resolves [#83637](https://github.com/ClickHouse/ClickHouse/issues/83637). [#83749](https://github.com/ClickHouse/ClickHouse/pull/83749) ([Rafael Roquetto](https://github.com/rafaelroquetto)). + +#### Bug Fix (user-visible misbehavior in an official stable release) +* Fix the wrong default value for the `--reconnect` option in `clickhouse-benchmark`. It was changed by mistake in [#79465](https://github.com/ClickHouse/ClickHouse/issues/79465). [#82677](https://github.com/ClickHouse/ClickHouse/pull/82677) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Fix inconsistent formatting of `CREATE DICTIONARY`. Closes [#82105](https://github.com/ClickHouse/ClickHouse/issues/82105). [#82829](https://github.com/ClickHouse/ClickHouse/pull/82829) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Fix inconsistent formatting of TTL when it contains a `materialize` function. Closes [#82828](https://github.com/ClickHouse/ClickHouse/issues/82828). [#82831](https://github.com/ClickHouse/ClickHouse/pull/82831) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Fix inconsistent formatting of EXPLAIN AST in a subquery when it contains output options such as INTO OUTFILE. Closes [#82826](https://github.com/ClickHouse/ClickHouse/issues/82826). [#82840](https://github.com/ClickHouse/ClickHouse/pull/82840) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Fix inconsistent formatting of parenthesized expressions with aliases in the context when no aliases are allowed. Closes [#82836](https://github.com/ClickHouse/ClickHouse/issues/82836). Closes [#82837](https://github.com/ClickHouse/ClickHouse/issues/82837). [#82867](https://github.com/ClickHouse/ClickHouse/pull/82867) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Use the proper error code when multiplying an aggregate function state with IPv4. Closes [#82817](https://github.com/ClickHouse/ClickHouse/issues/82817). [#82818](https://github.com/ClickHouse/ClickHouse/pull/82818) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Fix logical error in filesystem cache: "Having zero bytes but range is not finished". [#81868](https://github.com/ClickHouse/ClickHouse/pull/81868) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Recalculate the min-max index when TTL reduces rows to ensure the correctness of algorithms relying on it, such as `minmax_count_projection`. This resolves [#77091](https://github.com/ClickHouse/ClickHouse/issues/77091). [#77166](https://github.com/ClickHouse/ClickHouse/pull/77166) ([Amos Bird](https://github.com/amosbird)). +* For queries with combination of `ORDER BY ... LIMIT BY ... LIMIT N`, when ORDER BY is executed as a PartialSorting, the counter `rows_before_limit_at_least` now reflects the number of rows consumed by LIMIT clause instead of number of rows consumed by sorting transform. [#78999](https://github.com/ClickHouse/ClickHouse/pull/78999) ([Eduard Karacharov](https://github.com/korowa)). +* Fix excessive granule skipping for filtering over token/ngram indexes with regexp which contains alternation and non-literal first alternative. [#79373](https://github.com/ClickHouse/ClickHouse/pull/79373) ([Eduard Karacharov](https://github.com/korowa)). +* Fix logical error with `<=>` operator and Join storage, now query returns proper error code. [#80165](https://github.com/ClickHouse/ClickHouse/pull/80165) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Fix a crash in the `loop` function when used with the `remote` function family. Ensure the LIMIT clause is respected in `loop(remote(...))`. [#80299](https://github.com/ClickHouse/ClickHouse/pull/80299) ([Julia Kartseva](https://github.com/jkartseva)). +* Fix incorrect behavior of `to_utc_timestamp` and `from_utc_timestamp` functions when handling dates before Unix epoch (1970-01-01) and after maximum date (2106-02-07 06:28:15). Now these functions properly clamp values to epoch start and maximum date respectively. [#80498](https://github.com/ClickHouse/ClickHouse/pull/80498) ([Surya Kant Ranjan](https://github.com/iit2009046)). +* For some queries executed with parallel replicas, reading in order optimization(s) could be applied on initiator while can't be applied on remote nodes. It leads to different reading modes used by parallel replicas coordinator (on initiator) and on remoted nodes, which is a logical error. [#80652](https://github.com/ClickHouse/ClickHouse/pull/80652) ([Igor Nikonov](https://github.com/devcrafter)). +* Fix logical error during materialize projection when column type was changed to Nullable. [#80741](https://github.com/ClickHouse/ClickHouse/pull/80741) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix incorrect TTL recalculation in TTL GROUP BY when updating TTL. [#81222](https://github.com/ClickHouse/ClickHouse/pull/81222) ([Evgeniy Ulasik](https://github.com/H0uston)). +* Fixed Parquet bloom filter incorrectly applying condition like `WHERE function(key) IN (...)` as if it were `WHERE key IN (...)`. [#81255](https://github.com/ClickHouse/ClickHouse/pull/81255) ([Michael Kolupaev](https://github.com/al13n321)). +* Fixed possible crash in `Aggregator` in case of exception during merge. [#81450](https://github.com/ClickHouse/ClickHouse/pull/81450) ([Nikita Taranov](https://github.com/nickitat)). +* Fixed `InterpreterInsertQuery::extendQueryLogElemImpl` to add backquotes to database and table names when needed (f.g., when names contain special characters like `-`). [#81528](https://github.com/ClickHouse/ClickHouse/pull/81528) ([Ilia Shvyrialkin](https://github.com/Harzu)). +* Fix `IN` execution with `transform_null_in=1` with null in the left argument and non-nullable subquery result. [#81584](https://github.com/ClickHouse/ClickHouse/pull/81584) ([Pavel Kruglov](https://github.com/Avogar)). +* Don't validate experimental/suspicious types in default/materialize expression execution during reading from existing table. [#81618](https://github.com/ClickHouse/ClickHouse/pull/81618) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix "Context has expired" during merges when dict used in TTL expression. [#81690](https://github.com/ClickHouse/ClickHouse/pull/81690) ([Azat Khuzhin](https://github.com/azat)). +* Fix monotonicity of the cast function. [#81722](https://github.com/ClickHouse/ClickHouse/pull/81722) ([zoomxi](https://github.com/zoomxi)). +* Fix the issue where required columns are not read during scalar correlated subquery processing. Fixes [#81716](https://github.com/ClickHouse/ClickHouse/issues/81716). [#81805](https://github.com/ClickHouse/ClickHouse/pull/81805) ([Dmitry Novik](https://github.com/novikd)). +* In previous versions, the server returned excessive content for requests to `/js`. This closes [#61890](https://github.com/ClickHouse/ClickHouse/issues/61890). [#81895](https://github.com/ClickHouse/ClickHouse/pull/81895) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Previously, `MongoDB` table engine definitions could include a path component in the `host:port` argument which was silently ignored. The mongodb integration refuses to load such tables. With this fix *we allow loading such tables and ignore path component* if `MongoDB` engine has five arguments, using the database name from arguments. *Note:* The fix is not applied for newly created tables or queries with `mongo` table function, as well as for dictionary sources and named collections. [#81942](https://github.com/ClickHouse/ClickHouse/pull/81942) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Fixed possible crash in `Aggregator` in case of exception during merge. [#82022](https://github.com/ClickHouse/ClickHouse/pull/82022) ([Nikita Taranov](https://github.com/nickitat)). +* Fix filter analysis when only a constant alias column is used in the query. Fixes [#79448](https://github.com/ClickHouse/ClickHouse/issues/79448). [#82037](https://github.com/ClickHouse/ClickHouse/pull/82037) ([Dmitry Novik](https://github.com/novikd)). +* Fix LOGICAL_ERROR and following crash when using the same column in the TTL for GROUP BY and SET. [#82054](https://github.com/ClickHouse/ClickHouse/pull/82054) ([Pablo Marcos](https://github.com/pamarcos)). +* Fix S3 table function argument validation in secret masking, preventing possible `LOGICAL_ERROR`, close [#80620](https://github.com/ClickHouse/ClickHouse/issues/80620). [#82056](https://github.com/ClickHouse/ClickHouse/pull/82056) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Fix data races in Iceberg. [#82088](https://github.com/ClickHouse/ClickHouse/pull/82088) ([Azat Khuzhin](https://github.com/azat)). +* Fix `DatabaseReplicated::getClusterImpl`. If the first element (or elements) of `hosts` has `id == DROPPED_MARK` and there are no other elements for the same shard, the first element of `shards` will be an empty vector, leading to `std::out_of_range`. [#82093](https://github.com/ClickHouse/ClickHouse/pull/82093) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). +* Fixing copy-paste error in arraySimilarity, disallowing the use of UInt32 and Int32 weights. Update tests and docs. [#82103](https://github.com/ClickHouse/ClickHouse/pull/82103) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). +* Fix the `Not found column` error for queries with `arrayJoin` under `WHERE` condition and `IndexSet`. [#82113](https://github.com/ClickHouse/ClickHouse/pull/82113) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix bug in glue catalog integration. Now clickhouse can read tables with nested data types where some of subcolumns contain decimals, for example: `map`. Fixes [#81301](https://github.com/ClickHouse/ClickHouse/issues/81301). [#82114](https://github.com/ClickHouse/ClickHouse/pull/82114) ([alesapin](https://github.com/alesapin)). +* Fix performance degradation in SummingMergeTree that was intorduced in 25.5 in https://github.com/ClickHouse/ClickHouse/pull/79051. [#82130](https://github.com/ClickHouse/ClickHouse/pull/82130) ([Pavel Kruglov](https://github.com/Avogar)). +* When passing settings over uri the last value is considered. [#82137](https://github.com/ClickHouse/ClickHouse/pull/82137) ([Sema Checherinda](https://github.com/CheSema)). +* Fix "Context has expired" for Iceberg. [#82146](https://github.com/ClickHouse/ClickHouse/pull/82146) ([Azat Khuzhin](https://github.com/azat)). +* Fix possible deadlock for remote queries when server is under memory pressure. [#82160](https://github.com/ClickHouse/ClickHouse/pull/82160) ([Kirill](https://github.com/kirillgarbar)). +* Fixes overflow in `numericIndexedVectorPointwiseAdd`, `numericIndexedVectorPointwiseSubtract`, `numericIndexedVectorPointwiseMultiply`, `numericIndexedVectorPointwiseDivide` functions that happened when we applied them to large numbers. [#82165](https://github.com/ClickHouse/ClickHouse/pull/82165) ([Raufs Dunamalijevs](https://github.com/rienath)). +* Fix a bug in table dependencies causing Materialized Views to miss INSERT queries. [#82222](https://github.com/ClickHouse/ClickHouse/pull/82222) ([Nikolay Degterinsky](https://github.com/evillique)). +* Fix possible data-race between suggestion thread and main client thread. [#82233](https://github.com/ClickHouse/ClickHouse/pull/82233) ([Azat Khuzhin](https://github.com/azat)). +* Now ClickHouse can read iceberg tables from Glue catalog after schema evolution. Fixes [#81272](https://github.com/ClickHouse/ClickHouse/issues/81272). [#82301](https://github.com/ClickHouse/ClickHouse/pull/82301) ([alesapin](https://github.com/alesapin)). +* Fix the validation of async metrics settings `asynchronous_metrics_update_period_s` and `asynchronous_heavy_metrics_update_period_s`. [#82310](https://github.com/ClickHouse/ClickHouse/pull/82310) ([Bharat Nallan](https://github.com/bharatnc)). +* Fix logical error when resolving matcher in query with multiple JOINs, close [#81969](https://github.com/ClickHouse/ClickHouse/issues/81969). [#82421](https://github.com/ClickHouse/ClickHouse/pull/82421) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Add expiration to AWS ECS token so it can be reloaded. [#82422](https://github.com/ClickHouse/ClickHouse/pull/82422) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Fixes a bug for `NULL` arguments in `CASE` function. [#82436](https://github.com/ClickHouse/ClickHouse/pull/82436) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Fix data-races in client (by not using global context) and `session_timezone` overrides (previously in case of `session_timezone` was set in i.e. `users.xml`/client options to non empty and in query context to empty, then, value from `users.xml` was used, while this is wrong, now query context will always have a priority over global context). [#82444](https://github.com/ClickHouse/ClickHouse/pull/82444) ([Azat Khuzhin](https://github.com/azat)). +* Fix disabling boundary alignment for cached buffer in external table engines. It was broken in https://github.com/ClickHouse/ClickHouse/pull/81868. [#82493](https://github.com/ClickHouse/ClickHouse/pull/82493) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix the crash if key-value storage is joined with a type-casted key. [#82497](https://github.com/ClickHouse/ClickHouse/pull/82497) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* Fix hiding named collection values in logs/query_log. Closes [#82405](https://github.com/ClickHouse/ClickHouse/issues/82405). [#82510](https://github.com/ClickHouse/ClickHouse/pull/82510) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix a possible crash in logging while terminating a session as the user_id might sometimes be empty. [#82513](https://github.com/ClickHouse/ClickHouse/pull/82513) ([Bharat Nallan](https://github.com/bharatnc)). +* Fixes cases where parsing of Time could cause msan issues. This fixes: [#82477](https://github.com/ClickHouse/ClickHouse/issues/82477). [#82514](https://github.com/ClickHouse/ClickHouse/pull/82514) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Disallow setting `threadpool_writer_pool_size` to zero to ensure that server operations don't get stuck. [#82532](https://github.com/ClickHouse/ClickHouse/pull/82532) ([Bharat Nallan](https://github.com/bharatnc)). +* Fix `LOGICAL_ERROR` during row policy expression analysis for correlated columns. [#82618](https://github.com/ClickHouse/ClickHouse/pull/82618) ([Dmitry Novik](https://github.com/novikd)). +* Fix incorrect usage of parent metadata in `mergeTreeProjection` table function when `enable_shared_storage_snapshot_in_query = 1`. This is for [#82634](https://github.com/ClickHouse/ClickHouse/issues/82634). [#82638](https://github.com/ClickHouse/ClickHouse/pull/82638) ([Amos Bird](https://github.com/amosbird)). +* Functions `trim{Left,Right,Both}` now support input strings of type "FixedString(N)". For example, `SELECT trimBoth(toFixedString('abc', 3), 'ac')` now works. [#82691](https://github.com/ClickHouse/ClickHouse/pull/82691) ([Robert Schulze](https://github.com/rschu1ze)). +* In AzureBlobStorage, for native copy we compare authentication methods, during which if we get an exception, updated the code to fallback to read and copy (i.e. non native copy). [#82693](https://github.com/ClickHouse/ClickHouse/pull/82693) ([Smita Kulkarni](https://github.com/SmitaRKulkarni)). +* Fix deserialization of `groupArraySample`/`groupArrayLast` in case of empty elements (deserialization could skip part of the binary if the input was empty, this can lead to corruption during data read and UNKNOWN_PACKET_FROM_SERVER in TCP protocol). This does not affect numbers and date time types. [#82763](https://github.com/ClickHouse/ClickHouse/pull/82763) ([Pedro Ferreira](https://github.com/PedroTadim)). +* Fix backup of an empty `Memory` table, causing the backup restore to fail with with `BACKUP_ENTRY_NOT_FOUND` error. [#82791](https://github.com/ClickHouse/ClickHouse/pull/82791) ([Julia Kartseva](https://github.com/jkartseva)). +* Fix exception safety in union/intersect/except_default_mode rewrite. Closes [#82664](https://github.com/ClickHouse/ClickHouse/issues/82664). [#82820](https://github.com/ClickHouse/ClickHouse/pull/82820) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Keep track of the number of async tables loading jobs. If there are some running jobs, do not update `tail_ptr` in `TransactionLog::removeOldEntries`. [#82824](https://github.com/ClickHouse/ClickHouse/pull/82824) ([Tuan Pham Anh](https://github.com/tuanpach)). +* Fix data races in Iceberg. [#82841](https://github.com/ClickHouse/ClickHouse/pull/82841) ([Azat Khuzhin](https://github.com/azat)). +* Setting `use_skip_indexes_if_final_exact_mode` optimization (introduced in 25.6) could fail to select a relevant candidate range depending upon `MergeTree` engine settings / data distribution. That has been resolved now. [#82879](https://github.com/ClickHouse/ClickHouse/pull/82879) ([Shankar Iyer](https://github.com/shankar-iyer)). +* Set salt for auth data when parsing from AST with type SCRAM_SHA256_PASSWORD. [#82888](https://github.com/ClickHouse/ClickHouse/pull/82888) ([Tuan Pham Anh](https://github.com/tuanpach)). +* When using a non-caching Database implementation, the metadata of the corresponding table is deleted after the columns are returned and the reference is invalidated. [#82939](https://github.com/ClickHouse/ClickHouse/pull/82939) ([buyval01](https://github.com/buyval01)). +* Fix filter modification for queries with a JOIN expression with a table with storage `Merge`. Fixes [#82092](https://github.com/ClickHouse/ClickHouse/issues/82092). [#82950](https://github.com/ClickHouse/ClickHouse/pull/82950) ([Dmitry Novik](https://github.com/novikd)). +* Fix LOGICAL_ERROR in QueryMetricLog: Mutex cannot be NULL. [#82979](https://github.com/ClickHouse/ClickHouse/pull/82979) ([Pablo Marcos](https://github.com/pamarcos)). +* Fixed incorrect output of function `formatDateTime` when formatter `%f` is used together with variable-size formatters (e.g. `%M`). [#83020](https://github.com/ClickHouse/ClickHouse/pull/83020) ([Robert Schulze](https://github.com/rschu1ze)). +* Fix performance degradation with the enabled analyzer when secondary queries always read all columns from the VIEWs. Fixes [#81718](https://github.com/ClickHouse/ClickHouse/issues/81718). [#83036](https://github.com/ClickHouse/ClickHouse/pull/83036) ([Dmitry Novik](https://github.com/novikd)). +* Fix misleading error message when restoring a backup on a read-only disk. [#83051](https://github.com/ClickHouse/ClickHouse/pull/83051) ([Julia Kartseva](https://github.com/jkartseva)). +* Do not check for cyclic dependencies on create table with no dependencies. It fixes performance degradation of the use cases with creation of thousands of tables that was introduced in https://github.com/ClickHouse/ClickHouse/pull/65405. [#83077](https://github.com/ClickHouse/ClickHouse/pull/83077) ([Pavel Kruglov](https://github.com/Avogar)). +* Fixes issue with implicit reading of negative Time values into the table and make the docs not confusing. [#83091](https://github.com/ClickHouse/ClickHouse/pull/83091) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Do not use unrelated parts of a shared dictionary in the `lowCardinalityKeys` function. [#83118](https://github.com/ClickHouse/ClickHouse/pull/83118) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Fix the regression in use of subcolumns with Materialized Views. This fixes: [#82784](https://github.com/ClickHouse/ClickHouse/issues/82784). [#83221](https://github.com/ClickHouse/ClickHouse/pull/83221) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Fix crash in client due to connection left in disconnected state after bad INSERT. [#83253](https://github.com/ClickHouse/ClickHouse/pull/83253) ([Azat Khuzhin](https://github.com/azat)). +* Fix crash when calculating the size of a block with empty columns. [#83271](https://github.com/ClickHouse/ClickHouse/pull/83271) ([Raúl Marín](https://github.com/Algunenano)). +* Fix possible crash in Variant type in UNION. [#83295](https://github.com/ClickHouse/ClickHouse/pull/83295) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix LOGICAL_ERROR in clickhouse-local for unsupported SYSTEM queries. [#83333](https://github.com/ClickHouse/ClickHouse/pull/83333) ([Surya Kant Ranjan](https://github.com/iit2009046)). +* Fix `no_sign_request` for S3 client. It can be used to explicitly avoid signing S3 requests. It can also be defined for specific endpoints using endpoint-based settings. [#83379](https://github.com/ClickHouse/ClickHouse/pull/83379) ([Antonio Andelic](https://github.com/antonio2368)). +* Fixes a crash that may happen for a query with a setting 'max_threads=1' when executed under load with CPU scheduling enabled. [#83387](https://github.com/ClickHouse/ClickHouse/pull/83387) ([Fan Ziqi](https://github.com/f2quantum)). +* Fix `TOO_DEEP_SUBQUERIES` exception when CTE definition references another table expression with the same name. [#83413](https://github.com/ClickHouse/ClickHouse/pull/83413) ([Dmitry Novik](https://github.com/novikd)). +* Fix incorrect behavior when executing `REVOKE S3 ON system.*` revokes S3 permissions for `*.*`. This fixes [#83417](https://github.com/ClickHouse/ClickHouse/issues/83417). [#83420](https://github.com/ClickHouse/ClickHouse/pull/83420) ([pufit](https://github.com/pufit)). +* Do not share async_read_counters between queries. [#83423](https://github.com/ClickHouse/ClickHouse/pull/83423) ([Azat Khuzhin](https://github.com/azat)). +* Disable parallel replicas when a subquery contains the FINAL. [#83455](https://github.com/ClickHouse/ClickHouse/pull/83455) ([zoomxi](https://github.com/zoomxi)). +* Resolve minor integer overflow in configuration of setting `role_cache_expiration_time_seconds` (issue [#83374](https://github.com/ClickHouse/ClickHouse/issues/83374)). [#83461](https://github.com/ClickHouse/ClickHouse/pull/83461) ([wushap](https://github.com/wushap)). +* Fix a bug introduced in https://github.com/ClickHouse/ClickHouse/pull/79963. When inserting into an MV with a definer, the permission check should use the definer's grants. This fixes [#79951](https://github.com/ClickHouse/ClickHouse/issues/79951). [#83502](https://github.com/ClickHouse/ClickHouse/pull/83502) ([pufit](https://github.com/pufit)). +* Disable bounds-based file pruning for iceberg array element and iceberg map values, including all their nested subfields. [#83520](https://github.com/ClickHouse/ClickHouse/pull/83520) ([Daniil Ivanik](https://github.com/divanik)). +* Fix possible file cache not initialized errors when it's used as a temporary data storage. [#83539](https://github.com/ClickHouse/ClickHouse/pull/83539) ([Bharat Nallan](https://github.com/bharatnc)). +* Keeper fix: update total watch count correctly when ephemeral nodes are deleted on session close. [#83583](https://github.com/ClickHouse/ClickHouse/pull/83583) ([Antonio Andelic](https://github.com/antonio2368)). +* Fix incorrect memory around max_untracked_memory. [#83607](https://github.com/ClickHouse/ClickHouse/pull/83607) ([Azat Khuzhin](https://github.com/azat)). +* INSERT SELECT with UNION ALL could lead to a null pointer dereference in a corner case. This closes [#83618](https://github.com/ClickHouse/ClickHouse/issues/83618). [#83643](https://github.com/ClickHouse/ClickHouse/pull/83643) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Disallow zero value for max_insert_block_size as it could cause logical error. [#83688](https://github.com/ClickHouse/ClickHouse/pull/83688) ([Bharat Nallan](https://github.com/bharatnc)). +* Fix endless loop in estimateCompressionRatio() with block_size_bytes=0. [#83704](https://github.com/ClickHouse/ClickHouse/pull/83704) ([Azat Khuzhin](https://github.com/azat)). +* Fix `IndexUncompressedCacheBytes`/`IndexUncompressedCacheCells`/`IndexMarkCacheBytes`/`IndexMarkCacheFiles` metrics (previously they were included into metric w/o `Cache` prefix). [#83730](https://github.com/ClickHouse/ClickHouse/pull/83730) ([Azat Khuzhin](https://github.com/azat)). +* Fix possible abort (due to joining threads from the task) and hopefully hungs (in unit tests) during `BackgroundSchedulePool` shutdown. [#83769](https://github.com/ClickHouse/ClickHouse/pull/83769) ([Azat Khuzhin](https://github.com/azat)). +* Introduce backward compatibility setting to allow new analyzer to reference outer alias in WITH clause in the case of name clashes. Fixes [#82700](https://github.com/ClickHouse/ClickHouse/issues/82700). [#83797](https://github.com/ClickHouse/ClickHouse/pull/83797) ([Dmitry Novik](https://github.com/novikd)). +* Fix deadlock on shutdown due to recursive context locking during library bridge cleanup. [#83824](https://github.com/ClickHouse/ClickHouse/pull/83824) ([Azat Khuzhin](https://github.com/azat)). + +#### Build/Testing/Packaging Improvement +* Build a minimal C library (10 KB) for the ClickHouse lexer. This is needed for [#80977](https://github.com/ClickHouse/ClickHouse/issues/80977). [#81347](https://github.com/ClickHouse/ClickHouse/pull/81347) ([Alexey Milovidov](https://github.com/alexey-milovidov)). Add test for standalone lexer, add test tag `fasttest-only`. [#82472](https://github.com/ClickHouse/ClickHouse/pull/82472) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* Add a check for Nix submodule inputs. [#81691](https://github.com/ClickHouse/ClickHouse/pull/81691) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Fix a list of problems that can occur when trying to run integration tests on a localhost. [#82135](https://github.com/ClickHouse/ClickHouse/pull/82135) ([Oleg Doronin](https://github.com/dorooleg)). +* Compile SymbolIndex on Mac and FreeBSD. (But it will work only on ELF systems, Linux and FreeBSD). [#82347](https://github.com/ClickHouse/ClickHouse/pull/82347) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Bumped Azure SDK to v1.15.0. [#82747](https://github.com/ClickHouse/ClickHouse/pull/82747) ([Smita Kulkarni](https://github.com/SmitaRKulkarni)). +* Add storage module from google-cloud-cpp to build system. [#82881](https://github.com/ClickHouse/ClickHouse/pull/82881) ([Pablo Marcos](https://github.com/pamarcos)). +* Change `Dockerfile.ubuntu` for clickhouse-server to fit requirements in Docker Official Library. [#83039](https://github.com/ClickHouse/ClickHouse/pull/83039) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). +* A follow-up for [#83158](https://github.com/ClickHouse/ClickHouse/issues/83158) to fix uploading builds to `curl clickhouse.com`. [#83463](https://github.com/ClickHouse/ClickHouse/pull/83463) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). +* Adding `busybox` binary and install tools in `clickhouse/clickhouse-server` and official `clickhouse` images. [#83735](https://github.com/ClickHouse/ClickHouse/pull/83735) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). +* Added support for the `CLICKHOUSE_HOST` environment variable to specify the ClickHouse server host, aligning with existing `CLICKHOUSE_USER` and `CLICKHOUSE_PASSWORD` environment variables. This allows for easier configuration without modifying client or configuration files directly. [#83659](https://github.com/ClickHouse/ClickHouse/pull/83659) ([Doron David](https://github.com/dorki)). + + +### ClickHouse release 25.6, 2025-06-26 {#256} + +#### Backward Incompatible Change +* Previously, function `countMatches` would stop counting at the first empty match even if the pattern accepts it. To overcome this issue, `countMatches` now continues execution by advancing by a single character when an empty match occurs. Users who like to retain the old behavior can enable setting `count_matches_stop_at_empty_match`. [#81676](https://github.com/ClickHouse/ClickHouse/pull/81676) ([Elmi Ahmadov](https://github.com/ahmadov)). +* Minor: Force `backup_threads` and `restore_threads` server settings to be non zero. [#80224](https://github.com/ClickHouse/ClickHouse/pull/80224) ([Raúl Marín](https://github.com/Algunenano)). +* Minor: Fix `bitNot` for `String` will return a zero-terminated string in the internal memory representation. This should not affect any user visible behavior, however the author wanted to highlight this change. [#80791](https://github.com/ClickHouse/ClickHouse/pull/80791) ([Azat Khuzhin](https://github.com/azat)). + +#### New Feature +* New data types: `Time` (\[H\]HH:MM:SS) and `Time64` (\[H\]HH:MM:SS\[.fractional\]), and some basic cast functions and functions to interact with other data types. Added settings for compatibility with the existing function `toTime`. The setting `use_legacy_to_time` is set to keep the old behavior for now. [#81217](https://github.com/ClickHouse/ClickHouse/pull/81217) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). Support comparison between Time/Time64. [#80327](https://github.com/ClickHouse/ClickHouse/pull/80327) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* A new CLI tool, [`chdig`](https://github.com/azat/chdig/) - TUI interface for ClickHouse (top like) as part of ClickHouse. [#79666](https://github.com/ClickHouse/ClickHouse/pull/79666) ([Azat Khuzhin](https://github.com/azat)). +* Support `disk` setting for `Atomic` and `Ordinary` database engines, specifying the disk to store table metadata files. [#80546](https://github.com/ClickHouse/ClickHouse/pull/80546) ([Tuan Pham Anh](https://github.com/tuanpach)). This allows attaching databases from external sources. +* A new type of MergeTree, `CoalescingMergeTree` - the engine takes the first non-Null value during background merges. This closes [#78869](https://github.com/ClickHouse/ClickHouse/issues/78869). [#79344](https://github.com/ClickHouse/ClickHouse/pull/79344) ([scanhex12](https://github.com/scanhex12)). +* Support functions to read WKB ("Well-Known Binary" is a format for binary encoding of various geometry types, used in GIS applications). See [#43941](https://github.com/ClickHouse/ClickHouse/issues/43941). [#80139](https://github.com/ClickHouse/ClickHouse/pull/80139) ([scanhex12](https://github.com/scanhex12)). +* Added query slot scheduling for workloads, see [workload scheduling](https://clickhouse.com/docs/operations/workload-scheduling#query_scheduling) for details. [#78415](https://github.com/ClickHouse/ClickHouse/pull/78415) ([Sergei Trifonov](https://github.com/serxa)). +* `timeSeries*` helper functions to speedup some scenarios when working with time series data: - re-sample the data to the time grid with specified start timestamp, end timestamp and step - calculate PromQL-like `delta`, `rate`, `idelta` and `irate`. [#80590](https://github.com/ClickHouse/ClickHouse/pull/80590) ([Alexander Gololobov](https://github.com/davenger)). +* Add `mapContainsValuesLike`/`mapContainsValues`/`mapExtractValuesLike` functions to filter on map values and their support in bloom filter based indexes. [#78171](https://github.com/ClickHouse/ClickHouse/pull/78171) ([UnamedRus](https://github.com/UnamedRus)). +* Now settings constraints can specify a set of disallowed values. [#78499](https://github.com/ClickHouse/ClickHouse/pull/78499) ([Bharat Nallan](https://github.com/bharatnc)). +* Added a setting `enable_shared_storage_snapshot_in_query` to enable sharing the same storage snapshot across all subqueries in a single query. This ensures consistent reads from the same table, even when the table is referenced multiple times within a query. [#79471](https://github.com/ClickHouse/ClickHouse/pull/79471) ([Amos Bird](https://github.com/amosbird)). +* Support writing `JSON` columns to `Parquet` and reading `JSON` columns from `Parquet` directly. [#79649](https://github.com/ClickHouse/ClickHouse/pull/79649) ([Nihal Z. Miaji](https://github.com/nihalzp)). +* Add `MultiPolygon` support for `pointInPolygon`. [#79773](https://github.com/ClickHouse/ClickHouse/pull/79773) ([Nihal Z. Miaji](https://github.com/nihalzp)). +* Add support for querying local filesystem-mounted Delta tables via `deltaLakeLocal` table function. [#79781](https://github.com/ClickHouse/ClickHouse/pull/79781) ([roykim98](https://github.com/roykim98)). +* Add new setting `cast_string_to_date_time_mode` that allows to choose DateTime parsing mode during cast from String. [#80210](https://github.com/ClickHouse/ClickHouse/pull/80210) ([Pavel Kruglov](https://github.com/Avogar)). For example, you can set it to the best effort mode. +* Added `bech32Encode` and `bech32Decode` functions for working with Bitcoin's Bech algorithm (issue [#40381](https://github.com/ClickHouse/ClickHouse/issues/40381)). [#80239](https://github.com/ClickHouse/ClickHouse/pull/80239) ([George Larionov](https://github.com/glarik)). +* Add SQL functions to analyse the names of MergeTree parts. [#80573](https://github.com/ClickHouse/ClickHouse/pull/80573) ([Mikhail Artemenko](https://github.com/Michicosun)). +* Allow filtering parts selected for query by the disk they reside on by introducing a new virtual column, `_disk_name`. [#80650](https://github.com/ClickHouse/ClickHouse/pull/80650) ([tanner-bruce](https://github.com/tanner-bruce)). +* Add a landing page with the list of embedded web tools. It will open when requested by a browser-like user agent. [#81129](https://github.com/ClickHouse/ClickHouse/pull/81129) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Function `arrayFirst`, `arrayFirstIndex`, `arrayLast` and `arrayLastIndex` will filter away NULL values returned by the filter expression. In previous versions, Nullable filter results were not supported. Fixes [#81113](https://github.com/ClickHouse/ClickHouse/issues/81113). [#81197](https://github.com/ClickHouse/ClickHouse/pull/81197) ([Lennard Eijsackers](https://github.com/Blokje5)). +* It's now possible to write `USE DATABASE name` instead of `USE name`. [#81307](https://github.com/ClickHouse/ClickHouse/pull/81307) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Added a new system table `system.codecs` to introspect the available codecs. (issue [#81525](https://github.com/ClickHouse/ClickHouse/issues/81525)). [#81600](https://github.com/ClickHouse/ClickHouse/pull/81600) ([Jimmy Aguilar Mena](https://github.com/Ergus)). +* Support `lag` and `lead` window functions. Closes [#9887](https://github.com/ClickHouse/ClickHouse/issues/9887). [#82108](https://github.com/ClickHouse/ClickHouse/pull/82108) ([Dmitry Novik](https://github.com/novikd)). +* Function `tokens` now supports a new tokenizer, named `split`, which is good for logs. [#80195](https://github.com/ClickHouse/ClickHouse/pull/80195) ([Robert Schulze](https://github.com/rschu1ze)). +* Add support for the `--database` argument in `clickhouse-local`. You can switch to a previously created database. This closes [#44115](https://github.com/ClickHouse/ClickHouse/issues/44115). [#81465](https://github.com/ClickHouse/ClickHouse/pull/81465) ([Alexey Milovidov](https://github.com/alexey-milovidov)). + +#### Experimental Feature +* Implement Kafka rebalance like logic for `Kafka2` using ClickHouse Keeper For each replica we support two types of partition locks: permanent locks and temporary locks. The replica tries to hold permanent locks as long as possible, at any given time there are no more than `all_topic_partitions / active_replicas_count` (here `all_topic_partitions` is the number of all partitions, `active_replicas_count` is the number of active replicas) permanent locks on the replica, if there are more, then the replica releases some partitions. Some partitions are temporarily held by the replica. The maximum number of temporary locks on a replica changes dynamically to give other replicas a chance to take some partitions into permanent locks. When updating temporary locks, the replica releases them all and tries to take some others again. [#78726](https://github.com/ClickHouse/ClickHouse/pull/78726) ([Daria Fomina](https://github.com/sinfillo)). +* An improvement for the experimental text index: explicit parameters are supported via key-value pairs. Currently, supported parameters are a mandatory `tokenizer` and two optional `max_rows_per_postings_list` and `ngram_size`. [#80262](https://github.com/ClickHouse/ClickHouse/pull/80262) ([Elmi Ahmadov](https://github.com/ahmadov)). +* Previously, `packed` storage was not supported for the full-text index, because the segment id was updated on-fly by reading and writing (`.gin_sid`) file on disk. In case of packed storage, reading a value from the uncommited file is not supported and this led to an issue. Now it is alright. [#80852](https://github.com/ClickHouse/ClickHouse/pull/80852) ([Elmi Ahmadov](https://github.com/ahmadov)). +* Experimental indexes of type `gin` (which I don't like because it is an inside joke of PostgreSQL hackers) were renamed to `text`. Existing indexes of type `gin` remain loadable but they will throw an exception (suggesting `text` indexes instead) when one tries to use them in searches. [#80855](https://github.com/ClickHouse/ClickHouse/pull/80855) ([Robert Schulze](https://github.com/rschu1ze)). + +#### Performance Improvement +* Enable multiple-projection filtering support, allowing to use more than one projection for part-level filtering. This addresses [#55525](https://github.com/ClickHouse/ClickHouse/issues/55525). This is the second step to implement projection index, following [#78429](https://github.com/ClickHouse/ClickHouse/issues/78429). [#80343](https://github.com/ClickHouse/ClickHouse/pull/80343) ([Amos Bird](https://github.com/amosbird)). +* Use `SLRU` cache policy in filesystem cache by default. [#75072](https://github.com/ClickHouse/ClickHouse/pull/75072) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Remove contention in the Resize step in query pipeline. [#77562](https://github.com/ClickHouse/ClickHouse/pull/77562) ([Zhiguo Zhou](https://github.com/ZhiguoZh)). +* Introduced an option to offload (de)compression and (de)serialization of blocks into pipeline threads instead of a single thread associated with a network connection. Controlled by the setting `enable_parallel_blocks_marshalling`. It should speed up distributed queries that transfer significant amounts of data between the initiator and remote nodes. [#78694](https://github.com/ClickHouse/ClickHouse/pull/78694) ([Nikita Taranov](https://github.com/nickitat)). +* Performance improvements to all bloom filter types. [Video from the OpenHouse conference](https://www.youtube.com/watch?v=yIVz0NKwQvA&pp=ygUQb3BlbmhvdXNlIG9wZW5haQ%3D%3D) [#79800](https://github.com/ClickHouse/ClickHouse/pull/79800) ([Delyan Kratunov](https://github.com/dkratunov)). +* Introduced a happy path in `UniqExactSet::merge` when one of the sets is empty. Also, now if the LHS set is two-level and the RHS is single-level, we won't do the conversion to two-level for the RHS. [#79971](https://github.com/ClickHouse/ClickHouse/pull/79971) ([Nikita Taranov](https://github.com/nickitat)). +* Improve memory reuse efficiency and reduce page faults when using the two-level hash tables. This meant to speed-up GROUP BY. [#80245](https://github.com/ClickHouse/ClickHouse/pull/80245) ([Jiebin Sun](https://github.com/jiebinn)). +* Avoid unnecessary update and reduce lock contention in query condition cache. [#80247](https://github.com/ClickHouse/ClickHouse/pull/80247) ([Jiebin Sun](https://github.com/jiebinn)). +* Trivial optimization for `concatenateBlocks`. Chances are it's good for parallel hash join. [#80328](https://github.com/ClickHouse/ClickHouse/pull/80328) ([李扬](https://github.com/taiyang-li)). +* When selecting mark ranges from the primary key range, binary search cannot be used if the primary key is wrapped with functions. This PR improves this limitation: binary search can still be applied when the primary key is wrapped with an always monotonic function chain, or when the RPN contains an element that is always true. Closes [#45536](https://github.com/ClickHouse/ClickHouse/issues/45536). [#80597](https://github.com/ClickHouse/ClickHouse/pull/80597) ([zoomxi](https://github.com/zoomxi)). +* Improve shutdown speed of `Kafka` engine (remove extra 3 seconds delay in case of multiple `Kafka` tables). [#80796](https://github.com/ClickHouse/ClickHouse/pull/80796) ([Azat Khuzhin](https://github.com/azat)). +* Async inserts: reduce memory usage and improve performance of insert queries. [#80972](https://github.com/ClickHouse/ClickHouse/pull/80972) ([Raúl Marín](https://github.com/Algunenano)). +* Don't profile processors if the log table is disabled. [#81256](https://github.com/ClickHouse/ClickHouse/pull/81256) ([Raúl Marín](https://github.com/Algunenano)). This speeds up very short queries. +* Speed up `toFixedString` when the source is exactly what's requested. [#81257](https://github.com/ClickHouse/ClickHouse/pull/81257) ([Raúl Marín](https://github.com/Algunenano)). +* Don't process quota values if the user is not limited. [#81549](https://github.com/ClickHouse/ClickHouse/pull/81549) ([Raúl Marín](https://github.com/Algunenano)). This speeds up very short queries. +* Fixed performance regression in memory tracking. [#81694](https://github.com/ClickHouse/ClickHouse/pull/81694) ([Michael Kolupaev](https://github.com/al13n321)). +* Improve sharding key optimization on distributed query. [#78452](https://github.com/ClickHouse/ClickHouse/pull/78452) ([fhw12345](https://github.com/fhw12345)). +* Parallel replicas: avoid waiting for slow unused replicas if all read tasks have been assigned to other replicas. [#80199](https://github.com/ClickHouse/ClickHouse/pull/80199) ([Igor Nikonov](https://github.com/devcrafter)). +* Parallel replicas uses separate connection timeout, see `parallel_replicas_connect_timeout_ms` setting. Before `connect_timeout_with_failover_ms`/`connect_timeout_with_failover_secure_ms` settings were used as connection timeout values for parallel replicas queries (1 second by default). [#80421](https://github.com/ClickHouse/ClickHouse/pull/80421) ([Igor Nikonov](https://github.com/devcrafter)). +* In filesystem with journal `mkdir` is written to the journal of filesystem which is persisted to disk. In case of slow disk this can take long time. Move it out from reserve lock scope. [#81371](https://github.com/ClickHouse/ClickHouse/pull/81371) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Postpone reading of Iceberg manifest files until first reading query. [#81619](https://github.com/ClickHouse/ClickHouse/pull/81619) ([Daniil Ivanik](https://github.com/divanik)). +* Allow moving `GLOBAL [NOT] IN` predicate to `PREWHERE` clause if applicable. [#79996](https://github.com/ClickHouse/ClickHouse/pull/79996) ([Eduard Karacharov](https://github.com/korowa)). + +#### Improvement +* `EXPLAIN SYNTAX` now uses a new analyzer. It returns AST built from the query tree. Added option `query_tree_passes` to control the number of passes to executed before converting query tree to the AST. [#74536](https://github.com/ClickHouse/ClickHouse/pull/74536) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Implement flattened serialization for Dynamic and JSON in Native format that allows to serialize/deserialize Dynamic and JSON data without special structures like shared variant for Dynamic and shared data for JSON. This serialization can be enabled by setting `output_format_native_use_flattened_dynamic_and_json_serialization`. This serialization can be used for easier support for Dynamic and JSON in TCP protocol in clients in different languages. [#80499](https://github.com/ClickHouse/ClickHouse/pull/80499) ([Pavel Kruglov](https://github.com/Avogar)). +* Refresh `S3` credentials after error `AuthenticationRequired`. [#77353](https://github.com/ClickHouse/ClickHouse/pull/77353) ([Vitaly Baranov](https://github.com/vitlibar)). +* Added dictionary metrics to `system.asynchronous_metrics` - `DictionaryMaxUpdateDelay` - the maximum delay (in seconds) of dictionary update. - `DictionaryTotalFailedUpdates` - the number of errors since last successful loading in all dictionaries. [#78175](https://github.com/ClickHouse/ClickHouse/pull/78175) ([Vlad](https://github.com/codeworse)). +* Add a warning about databases that were potentially created to save broken tables. [#78841](https://github.com/ClickHouse/ClickHouse/pull/78841) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). +* Add `_time` virtual column in `S3Queue`, `AzureQueue` engine. [#78926](https://github.com/ClickHouse/ClickHouse/pull/78926) ([Anton Ivashkin](https://github.com/ianton-ru)). +* Make settings controlling connection drop on overloaded CPU hot-reloadable. [#79052](https://github.com/ClickHouse/ClickHouse/pull/79052) ([Alexey Katsman](https://github.com/alexkats)). +* Add container prefix to data paths reported in system.tables for plain disks in Azure blob storage, making reporting consistent with S3 and GCP. [#79241](https://github.com/ClickHouse/ClickHouse/pull/79241) ([Julia Kartseva](https://github.com/jkartseva)). +* Now, clickhouse-client and local also accept query parameters as `param-` (dash) along with `param_` (underscore). This closes [#63093](https://github.com/ClickHouse/ClickHouse/issues/63093). [#79429](https://github.com/ClickHouse/ClickHouse/pull/79429) ([Engel Danila](https://github.com/aaaengel)). +* Detailed warning msg for bandwidth discount when copying data from local to remote S3 with checksum enabled. [#79464](https://github.com/ClickHouse/ClickHouse/pull/79464) ([VicoWu](https://github.com/VicoWu)). +* Previously, when `input_format_parquet_max_block_size = 0` (an invalid value) ClickHouse would stuck. Now this behaviour is fixed. This closes [#79394](https://github.com/ClickHouse/ClickHouse/issues/79394). [#79601](https://github.com/ClickHouse/ClickHouse/pull/79601) ([abashkeev](https://github.com/abashkeev)). +* Add `throw_on_error` setting for `startup_scripts`: when `throw_on_error` is true, the server will not start unless all queries complete successfully. By default, `throw_on_error` is false, preserving the previous behavior. [#79732](https://github.com/ClickHouse/ClickHouse/pull/79732) ([Aleksandr Musorin](https://github.com/AVMusorin)). +* Allow to add `http_response_headers` in `http_handlers` of any kind. [#79975](https://github.com/ClickHouse/ClickHouse/pull/79975) ([Andrey Zvonov](https://github.com/zvonand)). +* Function `reverse` now supports `Tuple` data type. Closes [#80053](https://github.com/ClickHouse/ClickHouse/issues/80053). [#80083](https://github.com/ClickHouse/ClickHouse/pull/80083) ([flynn](https://github.com/ucasfl)). +* Resolve [#75817](https://github.com/ClickHouse/ClickHouse/issues/75817): allow getting `auxiliary_zookeepers` data from `system.zookeeper` table. [#80146](https://github.com/ClickHouse/ClickHouse/pull/80146) ([Nikolay Govorov](https://github.com/mrdimidium)). +* Add asynchronous metrics about the server's TCP sockets. This improves the observability. Closes [#80187](https://github.com/ClickHouse/ClickHouse/issues/80187). [#80188](https://github.com/ClickHouse/ClickHouse/pull/80188) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Support `anyLast_respect_nulls` and `any_respect_nulls` as a `SimpleAggregateFunction`. [#80219](https://github.com/ClickHouse/ClickHouse/pull/80219) ([Diskein](https://github.com/Diskein)). +* Remove unnecessary call `adjustCreateQueryForBackup` for replicated databases. [#80282](https://github.com/ClickHouse/ClickHouse/pull/80282) ([Vitaly Baranov](https://github.com/vitlibar)). +* Allow extra options (that go after `--` like `-- --config.value='abc'`) in `clickhouse-local` without the equality sign. Closes [#80292](https://github.com/ClickHouse/ClickHouse/issues/80292). [#80293](https://github.com/ClickHouse/ClickHouse/pull/80293) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Highlight metacharacters in `SHOW ... LIKE` queries. This closes [#80275](https://github.com/ClickHouse/ClickHouse/issues/80275). [#80297](https://github.com/ClickHouse/ClickHouse/pull/80297) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Make SQL UDF persistent in `clickhouse-local`. The previously created function will be loaded at startup. This closes [#80085](https://github.com/ClickHouse/ClickHouse/issues/80085). [#80300](https://github.com/ClickHouse/ClickHouse/pull/80300) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Fix description in explain plan for preliminary DISTINCT step. [#80330](https://github.com/ClickHouse/ClickHouse/pull/80330) ([UnamedRus](https://github.com/UnamedRus)). +* Allow to use named collections in ODBC/JDBC. [#80334](https://github.com/ClickHouse/ClickHouse/pull/80334) ([Andrey Zvonov](https://github.com/zvonand)). +* Metrics for number of readonly and broken disks. Indicator logs when DiskLocalCheckThread is started. [#80391](https://github.com/ClickHouse/ClickHouse/pull/80391) ([VicoWu](https://github.com/VicoWu)). +* Implement support for `s3_plain_rewritable` storage with projections. In previous versions, metadata objects in S3 referencing projections would not get updated when moved. Closes [#70258](https://github.com/ClickHouse/ClickHouse/issues/70258). [#80393](https://github.com/ClickHouse/ClickHouse/pull/80393) ([Sav](https://github.com/sberss)). +* The `SYSTEM UNFREEZE` command will not try to look up parts in readonly and write-once disks. This closes [#80430](https://github.com/ClickHouse/ClickHouse/issues/80430). [#80432](https://github.com/ClickHouse/ClickHouse/pull/80432) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Lowered the log level of merged parts messages. [#80476](https://github.com/ClickHouse/ClickHouse/pull/80476) ([Hans Krutzer](https://github.com/hkrutzer)). +* Change the default behavior of partition pruning for Iceberg tables. [#80583](https://github.com/ClickHouse/ClickHouse/pull/80583) ([Melvyn Peignon](https://github.com/melvynator)). +* Add two new ProfileEvents for index search algorithm observability: `IndexBinarySearchAlgorithm` and `IndexGenericExclusionSearchAlgorithm`. [#80679](https://github.com/ClickHouse/ClickHouse/pull/80679) ([Pablo Marcos](https://github.com/pamarcos)). +* Do not complain about unsupported `MADV_POPULATE_WRITE` for older kernels in logs (to avoid logs polluting). [#80704](https://github.com/ClickHouse/ClickHouse/pull/80704) ([Robert Schulze](https://github.com/rschu1ze)). +* Added support for `Date32` and `DateTime64` in `TTL` expressions. [#80710](https://github.com/ClickHouse/ClickHouse/pull/80710) ([Andrey Zvonov](https://github.com/zvonand)). +* Adjust compatibility values for `max_merge_delayed_streams_for_parallel_write`. [#80760](https://github.com/ClickHouse/ClickHouse/pull/80760) ([Azat Khuzhin](https://github.com/azat)). +* Fix a crash: if an exception is thrown in an attempt to remove a temporary file (they are used for spilling temporary data on disk) in the destructor, the program can terminate. [#80776](https://github.com/ClickHouse/ClickHouse/pull/80776) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Add `IF EXISTS` modifier to `SYSTEM SYNC REPLICA`. [#80810](https://github.com/ClickHouse/ClickHouse/pull/80810) ([Raúl Marín](https://github.com/Algunenano)). +* Extend exception message about "Having zero bytes, but read range is not finished...", add finished_download_time column to `system.filesystem_cache`. [#80849](https://github.com/ClickHouse/ClickHouse/pull/80849) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Add search algorithm section to `EXPLAIN` output when using it with indexes = 1. If shows either "binary search" or "generic exclusion search". [#80881](https://github.com/ClickHouse/ClickHouse/pull/80881) ([Pablo Marcos](https://github.com/pamarcos)). +* At the beginning of 2024, `prefer_column_name_to_alias` was hardcoded to true for MySQL handler because the new analyzer was not enabled by default. Now, it can be unhardcoded. [#80916](https://github.com/ClickHouse/ClickHouse/pull/80916) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Now `system.iceberg_history` shows history for catalogs databases like glue or iceberg rest. Also renamed `table_name` and `database_name` columns to `table` and `database` in `system.iceberg_history` for consistency. [#80975](https://github.com/ClickHouse/ClickHouse/pull/80975) ([alesapin](https://github.com/alesapin)). +* Allow read-only mode for the `merge` table function, so the `CREATE TEMPORARY TABLE` grant is not required for using it. [#80981](https://github.com/ClickHouse/ClickHouse/pull/80981) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). +* Better introspection of in-memory caches (expose information about caches in `system.metrics` over incomplete `system.asynchronouse_metrics`). Add in-memory caches size (in bytes) into `dashboard.html`. `VectorSimilarityIndexCacheSize`/`IcebergMetadataFilesCacheSize` has been renamed to `VectorSimilarityIndexCacheBytes`/`IcebergMetadataFilesCacheBytes`. [#81023](https://github.com/ClickHouse/ClickHouse/pull/81023) ([Azat Khuzhin](https://github.com/azat)). +* Ignore databases with engines that can't contain `RocksDB` tables while reading from `system.rocksdb`. [#81083](https://github.com/ClickHouse/ClickHouse/pull/81083) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* Allow `filesystem_caches` and `named_collections` in the `clickhouse-local` configuration file. [#81105](https://github.com/ClickHouse/ClickHouse/pull/81105) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Fix highlighting of `PARTITION BY` in `INSERT` queries. In previous versions, `PARTITION BY` was not highlighted as a keyword. [#81106](https://github.com/ClickHouse/ClickHouse/pull/81106) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Two mini improvements in Web UI: - correctly handle queries without output, such as `CREATE`, `INSERT` (until recently, these queries resulted in an infinite spinner); - when double clicking on a table, scroll to the top. [#81131](https://github.com/ClickHouse/ClickHouse/pull/81131) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* `MemoryResidentWithoutPageCache` metric provides the amount of physical memory used by the server process, excluding userspace page cache, in bytes. This provides a more accurate view of actual memory usage when userspace page cache is utilized. When userspace page cache is disabled, this value equals `MemoryResident`. [#81233](https://github.com/ClickHouse/ClickHouse/pull/81233) ([Jayme Bird](https://github.com/jaymebrd)). +* Mark manually logged exceptions in client, local server, keeper client and disks app as logged, so that they are not logged twice. [#81271](https://github.com/ClickHouse/ClickHouse/pull/81271) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). +* Setting `use_skip_indexes_if_final` and `use_skip_indexes_if_final_exact_mode` now default to `True`. Queries with `FINAL` clause will now use skip indexes (if applicable) to shortlist granules and also read any additional granules corresponding to matching primary key ranges. Users needing earlier behaviour of approximate/imprecise results can set `use_skip_indexes_if_final_exact_mode` to FALSE after careful evaluation. [#81331](https://github.com/ClickHouse/ClickHouse/pull/81331) ([Shankar Iyer](https://github.com/shankar-iyer)). +* When you have multiple queries in the web UI, it will run the one under the cursor. Continuation of [#80977](https://github.com/ClickHouse/ClickHouse/issues/80977). [#81354](https://github.com/ClickHouse/ClickHouse/pull/81354) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* This PR addresses issues with the implementation of `is_strict` in the monotonicity checks for conversion functions. Currently, some conversion functions, such as `toFloat64(UInt32)` and `toDate(UInt8)`, incorrectly return `is_strict` as false when they should return true. [#81359](https://github.com/ClickHouse/ClickHouse/pull/81359) ([zoomxi](https://github.com/zoomxi)). +* When checking if a `KeyCondition` matches a continuous range, if the key is wrapped with a non-strict function chain, a `Constraint::POINT` may needs to be converted to a`Constraint::RANGE`. For example: `toDate(event_time) = '2025-06-03'` implies a range for `event_time`: ['2025-06-03 00:00:00', '2025-06-04 00:00:00'). This PR fixes this behavior. [#81400](https://github.com/ClickHouse/ClickHouse/pull/81400) ([zoomxi](https://github.com/zoomxi)). +* `clickhouse`/`ch` aliases will invoke `clickhouse-client` instead of `clickhouse-local` if `--host` or `--port` are specified. Continuation of [#79422](https://github.com/ClickHouse/ClickHouse/issues/79422). Closes [#65252](https://github.com/ClickHouse/ClickHouse/issues/65252). [#81509](https://github.com/ClickHouse/ClickHouse/pull/81509) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Now that we have the keeper response time distribution data, we can tune the histogram buckets for metrics. [#81516](https://github.com/ClickHouse/ClickHouse/pull/81516) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). +* Add profile event `PageCacheReadBytes`. [#81742](https://github.com/ClickHouse/ClickHouse/pull/81742) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix logical error in filesystem cache: "Having zero bytes but range is not finished". [#81868](https://github.com/ClickHouse/ClickHouse/pull/81868) ([Kseniia Sumarokova](https://github.com/kssenii)). + +#### Bug Fix (user-visible misbehavior in an official stable release) +* Fix parameterized view with SELECT EXCEPT query. Closes [#49447](https://github.com/ClickHouse/ClickHouse/issues/49447). [#57380](https://github.com/ClickHouse/ClickHouse/pull/57380) ([Nikolay Degterinsky](https://github.com/evillique)). +* Analyzer: Fix column projection name after column type promotion in join. Closes [#63345](https://github.com/ClickHouse/ClickHouse/issues/63345). [#63519](https://github.com/ClickHouse/ClickHouse/pull/63519) ([Dmitry Novik](https://github.com/novikd)). +* Fixed a logical error in cases of column name clashes when analyzer_compatibility_join_using_top_level_identifier is enabled. [#75676](https://github.com/ClickHouse/ClickHouse/pull/75676) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Fix CTE usage in pushed-down predicates when `allow_push_predicate_ast_for_distributed_subqueries` is enabled. Fixes [#75647](https://github.com/ClickHouse/ClickHouse/issues/75647). Fixes [#79672](https://github.com/ClickHouse/ClickHouse/issues/79672). [#77316](https://github.com/ClickHouse/ClickHouse/pull/77316) ([Dmitry Novik](https://github.com/novikd)). +* Fixes an issue where SYSTEM SYNC REPLICA LIGHTWEIGHT 'foo' would report success even when the specified replica didn't exist. The command now properly validates that the replica exists in Keeper before attempting synchronization. [#78405](https://github.com/ClickHouse/ClickHouse/pull/78405) ([Jayme Bird](https://github.com/jaymebrd)). +* Fix crash for a very specific situation when the `currentDatabase` function was used in `CONSTRAINT` sections for `ON CLUSTER` queries Closes [#78100](https://github.com/ClickHouse/ClickHouse/issues/78100). [#79070](https://github.com/ClickHouse/ClickHouse/pull/79070) ([pufit](https://github.com/pufit)). +* Fix passing of external roles in interserver queries. [#79099](https://github.com/ClickHouse/ClickHouse/pull/79099) ([Andrey Zvonov](https://github.com/zvonand)). +* Try to use IColumn instead of Field in SingleValueDataGeneric. It fixes the incorrect return values for some aggregate functions like `argMax` for types `Dynamic/Variant/JSON`. [#79166](https://github.com/ClickHouse/ClickHouse/pull/79166) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix applying use_native_copy and allow_azure_native_copy setting for azure blob storage and updated to use native copy only when credentials match resolves [#78964](https://github.com/ClickHouse/ClickHouse/issues/78964). [#79561](https://github.com/ClickHouse/ClickHouse/pull/79561) ([Smita Kulkarni](https://github.com/SmitaRKulkarni)). +* Fix logical errors about a column's unknown origin scope produced while checking if this column is correlated. Fixes [#78183](https://github.com/ClickHouse/ClickHouse/issues/78183). Fixes [#79451](https://github.com/ClickHouse/ClickHouse/issues/79451). [#79727](https://github.com/ClickHouse/ClickHouse/pull/79727) ([Dmitry Novik](https://github.com/novikd)). +* Fix wrong results for grouping sets with ColumnConst and Analyzer. [#79743](https://github.com/ClickHouse/ClickHouse/pull/79743) ([Andrey Zvonov](https://github.com/zvonand)). +* Fix local shard result duplication when reading from distributed table with local replica being stale. [#79761](https://github.com/ClickHouse/ClickHouse/pull/79761) ([Eduard Karacharov](https://github.com/korowa)). +* Fix the sorting order of the NaNs with a negative sign bit. [#79847](https://github.com/ClickHouse/ClickHouse/pull/79847) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* Now GROUP BY ALL doesn't take into account the GROUPING part. [#79915](https://github.com/ClickHouse/ClickHouse/pull/79915) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Fixed incorrect state merging for `TopK` / `TopKWeighted` functions that would cause excessive error values even when capacity was not exhausted. [#79939](https://github.com/ClickHouse/ClickHouse/pull/79939) ([Joel Höner](https://github.com/athre0z)). +* Respect `readonly` setting in `azure_blob_storage` object storage. [#79954](https://github.com/ClickHouse/ClickHouse/pull/79954) ([Julia Kartseva](https://github.com/jkartseva)). +* Fixed incorrect query results and out-of-memory crashes when using `match(column, '^…')` with backslash-escaped characters. [#79969](https://github.com/ClickHouse/ClickHouse/pull/79969) ([filimonov](https://github.com/filimonov)). +* Disabling hive partitioning for datalakes Partially addresses https://github.com/issues/assigned?issue=ClickHouse%7CClickHouse%7C79937. [#80005](https://github.com/ClickHouse/ClickHouse/pull/80005) ([Daniil Ivanik](https://github.com/divanik)). +* Skip indexes with lambda expressions could not be applied. Fix the case when high-level functions in the index definition exactly match the one in the query. [#80025](https://github.com/ClickHouse/ClickHouse/pull/80025) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix metadata version during attach part on the replica executing ATTACH_PART command from replication log. [#80038](https://github.com/ClickHouse/ClickHouse/pull/80038) ([Aleksei Filatov](https://github.com/aalexfvk)). +* Executable User Defined Functions (eUDF) names are not added to the `used_functions` column of the `system.query_log` table, unlike other functions. This PR implements the addition of the eUDF name if the eUDF was used in the request. [#80073](https://github.com/ClickHouse/ClickHouse/pull/80073) ([Kyamran](https://github.com/nibblerenush)). +* Fix logical error in Arrow format with LowCardinality(FixedString). [#80156](https://github.com/ClickHouse/ClickHouse/pull/80156) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix reading subcolumns from Merge engine. [#80158](https://github.com/ClickHouse/ClickHouse/pull/80158) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix a bug about the comparison between numeric types in `KeyCondition`. [#80207](https://github.com/ClickHouse/ClickHouse/pull/80207) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Fix AMBIGUOUS_COLUMN_NAME when lazy materialization applied to table with projections. [#80251](https://github.com/ClickHouse/ClickHouse/pull/80251) ([Igor Nikonov](https://github.com/devcrafter)). +* Fix incorrect count optimization for string prefix filters like LIKE 'ab_c%' when using implicit projections. This fixes [#80250](https://github.com/ClickHouse/ClickHouse/issues/80250). [#80261](https://github.com/ClickHouse/ClickHouse/pull/80261) ([Amos Bird](https://github.com/amosbird)). +* Fix improper serialization of nested numeric fields as strings in MongoDB documents. Remove maximum depth limit for documents from MongoDB. [#80289](https://github.com/ClickHouse/ClickHouse/pull/80289) ([Kirill Nikiforov](https://github.com/allmazz)). +* Perform less strict metadata checks for RMT in the Replicated database. Closes [#80296](https://github.com/ClickHouse/ClickHouse/issues/80296). [#80298](https://github.com/ClickHouse/ClickHouse/pull/80298) ([Nikolay Degterinsky](https://github.com/evillique)). +* Fix text representation of DateTime and DateTime64 for PostgreSQL storage. [#80301](https://github.com/ClickHouse/ClickHouse/pull/80301) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* Allow `DateTime` with timezone in `StripeLog` tables. This closes [#44120](https://github.com/ClickHouse/ClickHouse/issues/44120). [#80304](https://github.com/ClickHouse/ClickHouse/pull/80304) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Disable filter-push-down for the predicate with a non-deterministic function in case the query plan step changes the number of rows. Fixes [#40273](https://github.com/ClickHouse/ClickHouse/issues/40273). [#80329](https://github.com/ClickHouse/ClickHouse/pull/80329) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix possible logical errors and crashes in projections with subcolumns. [#80333](https://github.com/ClickHouse/ClickHouse/pull/80333) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix `NOT_FOUND_COLUMN_IN_BLOCK` error caused by filter-push-down optimization of the logical JOIN sep in case `ON` expression is not a trivial equality. Fixes [#79647](https://github.com/ClickHouse/ClickHouse/issues/79647) Fixes [#77848](https://github.com/ClickHouse/ClickHouse/issues/77848). [#80360](https://github.com/ClickHouse/ClickHouse/pull/80360) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix incorrect result when reading reverse-ordered keys in partitioned tables. This fixes [#79987](https://github.com/ClickHouse/ClickHouse/issues/79987). [#80448](https://github.com/ClickHouse/ClickHouse/pull/80448) ([Amos Bird](https://github.com/amosbird)). +* Fixed wrong sorting in tables with a nullable key and enabled optimize_read_in_order. [#80515](https://github.com/ClickHouse/ClickHouse/pull/80515) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* Fixed refreshable materialized view DROP getting stuck if the view was paused using SYSTEM STOP REPLICATED VIEW. [#80543](https://github.com/ClickHouse/ClickHouse/pull/80543) ([Michael Kolupaev](https://github.com/al13n321)). +* Fix 'Cannot find column' with constant tuple in distributed query. [#80596](https://github.com/ClickHouse/ClickHouse/pull/80596) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* Fix `shardNum` function in Distributed tables with `join_use_nulls`. [#80612](https://github.com/ClickHouse/ClickHouse/pull/80612) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). +* Fix incorrect result during reading column that exists in subset of tables in Merge engine. [#80643](https://github.com/ClickHouse/ClickHouse/pull/80643) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix possible SSH protocol (due to hang in replxx). [#80688](https://github.com/ClickHouse/ClickHouse/pull/80688) ([Azat Khuzhin](https://github.com/azat)). +* The timestamp in the iceberg_history table should now be correct. [#80711](https://github.com/ClickHouse/ClickHouse/pull/80711) ([Melvyn Peignon](https://github.com/melvynator)). +* Fix possible crash in case of dictionary registration failed (when `CREATE DICTIONARY` failed with `CANNOT_SCHEDULE_TASK` it is possible to leave dangling pointer in the dictionary registry, which later lead to crash). [#80714](https://github.com/ClickHouse/ClickHouse/pull/80714) ([Azat Khuzhin](https://github.com/azat)). +* Fix handling of enum globs of a single element in object storage table functions. [#80716](https://github.com/ClickHouse/ClickHouse/pull/80716) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Fix wrong result type of comparison functions with Tuple(Dynamic) and String that led to logical error. [#80728](https://github.com/ClickHouse/ClickHouse/pull/80728) ([Pavel Kruglov](https://github.com/Avogar)). +* Add missing support data type `timestamp_ntz` for unity catalog. Fixes [#79535](https://github.com/ClickHouse/ClickHouse/issues/79535), Fixes [#79875](https://github.com/ClickHouse/ClickHouse/issues/79875). [#80740](https://github.com/ClickHouse/ClickHouse/pull/80740) ([alesapin](https://github.com/alesapin)). +* Fix `THERE_IS_NO_COLUMN` error for distributed queries with `IN cte`. Fixes [#75032](https://github.com/ClickHouse/ClickHouse/issues/75032). [#80757](https://github.com/ClickHouse/ClickHouse/pull/80757) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix excessive number of files (leads to excessive memory usage) for external ORDER BY. [#80777](https://github.com/ClickHouse/ClickHouse/pull/80777) ([Azat Khuzhin](https://github.com/azat)). +* This PR might close [#80742](https://github.com/ClickHouse/ClickHouse/issues/80742). [#80783](https://github.com/ClickHouse/ClickHouse/pull/80783) ([zoomxi](https://github.com/zoomxi)). +* Fix crash in Kafka due to get_member_id() was creating std::string from NULL (it was likely an issue only in case of connection to broker had been failed). [#80793](https://github.com/ClickHouse/ClickHouse/pull/80793) ([Azat Khuzhin](https://github.com/azat)). +* Properly wait consumers before shutting down Kafka engine (active consumers after shutdown can trigger various debug assertions and also may read data from brokers in background after table has been dropped/detached). [#80795](https://github.com/ClickHouse/ClickHouse/pull/80795) ([Azat Khuzhin](https://github.com/azat)). +* Fix `NOT_FOUND_COLUMN_IN_BLOCK`, which is caused by `predicate-push-down` optimization. Fixes [#80443](https://github.com/ClickHouse/ClickHouse/issues/80443). [#80834](https://github.com/ClickHouse/ClickHouse/pull/80834) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix logical error when resolving star (*) matcher in table function in JOIN with USING. [#80894](https://github.com/ClickHouse/ClickHouse/pull/80894) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Fix memory accounting for Iceberg metadata files cache. [#80904](https://github.com/ClickHouse/ClickHouse/pull/80904) ([Azat Khuzhin](https://github.com/azat)). +* Fix wrong partitioning with nullable partition key. [#80913](https://github.com/ClickHouse/ClickHouse/pull/80913) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* Fix `Table does not exist` error for distributed queries with pushed-down predicate (`allow_push_predicate_ast_for_distributed_subqueries=1`) when the source table does not exist on the initialtor. Fixes [#77281](https://github.com/ClickHouse/ClickHouse/issues/77281). [#80915](https://github.com/ClickHouse/ClickHouse/pull/80915) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix the logical error in the nested functions with named windows. [#80926](https://github.com/ClickHouse/ClickHouse/pull/80926) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* Fix extremes for nullable and floating-point columns. [#80970](https://github.com/ClickHouse/ClickHouse/pull/80970) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* Fix possible crash while querying from system.tables (likely the case under memory pressure). [#80976](https://github.com/ClickHouse/ClickHouse/pull/80976) ([Azat Khuzhin](https://github.com/azat)). +* Fix atomic rename with truncate for files which compression is inferred from their file extension. [#80979](https://github.com/ClickHouse/ClickHouse/pull/80979) ([Pablo Marcos](https://github.com/pamarcos)). +* Fix ErrorCodes::getName. [#81032](https://github.com/ClickHouse/ClickHouse/pull/81032) ([RinChanNOW](https://github.com/RinChanNOWWW)). +* Fix bug when user cannot list tables in Unity Catalog without permissions for all of them. Now all tables are listed properly, attempt to read from restricted table will throw an exception. [#81044](https://github.com/ClickHouse/ClickHouse/pull/81044) ([alesapin](https://github.com/alesapin)). +* Now ClickHouse will ignore errors and unexpected responses from data lake catalogs in `SHOW TABLES` query. Fixes [#79725](https://github.com/ClickHouse/ClickHouse/issues/79725). [#81046](https://github.com/ClickHouse/ClickHouse/pull/81046) ([alesapin](https://github.com/alesapin)). +* Fix parsing of DateTime64 from integers in JSONExtract and JSON type parsing. [#81050](https://github.com/ClickHouse/ClickHouse/pull/81050) ([Pavel Kruglov](https://github.com/Avogar)). +* Reflect date_time_input_format setting in schema inference cache. [#81052](https://github.com/ClickHouse/ClickHouse/pull/81052) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix crash on INSERT if table was DROPed after query started but before columns sent. [#81053](https://github.com/ClickHouse/ClickHouse/pull/81053) ([Azat Khuzhin](https://github.com/azat)). +* Fix use-of-uninitialized-value in quantileDeterministic. [#81062](https://github.com/ClickHouse/ClickHouse/pull/81062) ([Azat Khuzhin](https://github.com/azat)). +* Fix hardlinks count management for metadatastoragefromdisk disk transactions. add tests. [#81066](https://github.com/ClickHouse/ClickHouse/pull/81066) ([Sema Checherinda](https://github.com/CheSema)). +* User Defined Functions (UDF) names are not added to the `system.query_log` table, unlike other functions. This PR implements the addition of the UDF name to one of the two columns `used_executable_user_defined_functions` or `used_sql_user_defined_functions` if the UDF was used in the request. [#81101](https://github.com/ClickHouse/ClickHouse/pull/81101) ([Kyamran](https://github.com/nibblerenush)). +* Fixed `Too large size ... passed to allocator` errors or possible crashes on inserts via http protocol with text formats (`JSON`, `Values`, ...) and omitted `Enum` fields. [#81145](https://github.com/ClickHouse/ClickHouse/pull/81145) ([Anton Popov](https://github.com/CurtizJ)). +* Fix LOGICAL_ERROR in case of Sparse column in INSERT block pushed to non-MT MV. [#81161](https://github.com/ClickHouse/ClickHouse/pull/81161) ([Azat Khuzhin](https://github.com/azat)). +* Fix `Unknown table expression identifier` for `distributed_product_mode_local=local` with cross-replication. [#81162](https://github.com/ClickHouse/ClickHouse/pull/81162) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fixed incorrectly caching number of rows in parquet files after filtering. [#81184](https://github.com/ClickHouse/ClickHouse/pull/81184) ([Michael Kolupaev](https://github.com/al13n321)). +* Fix fs cache max_size_to_total_space setting when used with relative cache path. [#81237](https://github.com/ClickHouse/ClickHouse/pull/81237) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fixed clickhouse-local crashing when outputting const tuples or maps in Parquet format. [#81249](https://github.com/ClickHouse/ClickHouse/pull/81249) ([Michael Kolupaev](https://github.com/al13n321)). +* Verify array offsets received over network. [#81269](https://github.com/ClickHouse/ClickHouse/pull/81269) ([Azat Khuzhin](https://github.com/azat)). +* Fix some corner case in query that joins empty tables and uses window functions. The bug leads to exploding number of parallel streams which leads to OOMs. [#81299](https://github.com/ClickHouse/ClickHouse/pull/81299) ([Alexander Gololobov](https://github.com/davenger)). +* Fixes for datalake Cluster functions (`deltaLakeCluster`, `icebergCluster`, etc): (1) fix potential segfault in `DataLakeConfiguration` when using `Cluster` function with old analyzer; (2) remove duplicating data lake metadata updates (extra object storage requests); (3) fix redundant listing in object storage when format is not explicitly specified (which was already done for non-cluster data lake engines). [#81300](https://github.com/ClickHouse/ClickHouse/pull/81300) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Make force_restore_data flag recover lost keeper metadata. [#81324](https://github.com/ClickHouse/ClickHouse/pull/81324) ([Raúl Marín](https://github.com/Algunenano)). +* Fix region error in delta-kernel. Fixes [#79914](https://github.com/ClickHouse/ClickHouse/issues/79914). [#81353](https://github.com/ClickHouse/ClickHouse/pull/81353) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Disable incorrect JIT for divideOrNull. [#81370](https://github.com/ClickHouse/ClickHouse/pull/81370) ([Raúl Marín](https://github.com/Algunenano)). +* Fix insert error when MergeTree table has a long partition column name. [#81390](https://github.com/ClickHouse/ClickHouse/pull/81390) ([hy123q](https://github.com/haoyangqian)). +* Backported in [#81957](https://github.com/ClickHouse/ClickHouse/issues/81957): Fixed possible crash in `Aggregator` in case of exception during merge. [#81450](https://github.com/ClickHouse/ClickHouse/pull/81450) ([Nikita Taranov](https://github.com/nickitat)). +* Don't store content of several manifest files in memory. [#81470](https://github.com/ClickHouse/ClickHouse/pull/81470) ([Daniil Ivanik](https://github.com/divanik)). +* Fix possible crash during shutting down background pools (`background_.*pool_size`). [#81473](https://github.com/ClickHouse/ClickHouse/pull/81473) ([Azat Khuzhin](https://github.com/azat)). +* Fix out-of-bounds read in the `Npy` format happening when writing to a table with the `URL` engine. This closes [#81356](https://github.com/ClickHouse/ClickHouse/issues/81356). [#81502](https://github.com/ClickHouse/ClickHouse/pull/81502) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* There is a chance that Web UI displays `NaN%` (typical JavaScript problems). [#81507](https://github.com/ClickHouse/ClickHouse/pull/81507) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Fix `DatabaseReplicated` for `database_replicated_enforce_synchronous_settings=1`. [#81564](https://github.com/ClickHouse/ClickHouse/pull/81564) ([Azat Khuzhin](https://github.com/azat)). +* Fix sorting order for LowCardinality(Nullable(...)) types. [#81583](https://github.com/ClickHouse/ClickHouse/pull/81583) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* Server should not preserve a HTTP connection if the request has not been fully read from the socket. [#81595](https://github.com/ClickHouse/ClickHouse/pull/81595) ([Sema Checherinda](https://github.com/CheSema)). +* Make scalar correlated subqueries return a nullable result of the projection expression. Fix the case when a correlated subquery produces an empty result set. [#81632](https://github.com/ClickHouse/ClickHouse/pull/81632) ([Dmitry Novik](https://github.com/novikd)). +* Fix `Unexpected relative path for a deduplicated part` during `ATTACH` to `ReplicatedMergeTree`. [#81647](https://github.com/ClickHouse/ClickHouse/pull/81647) ([Azat Khuzhin](https://github.com/azat)). +* Query settings `use_iceberg_partition_pruning` will not take effect for iceberg storage, because it uses global context rather than query context. it's not critical because its default value is true. this pr can fix it. [#81673](https://github.com/ClickHouse/ClickHouse/pull/81673) ([Han Fei](https://github.com/hanfei1991)). +* Backported in [#82128](https://github.com/ClickHouse/ClickHouse/issues/82128): Fix "Context has expired" during merges when dict used in TTL expression. [#81690](https://github.com/ClickHouse/ClickHouse/pull/81690) ([Azat Khuzhin](https://github.com/azat)). +* Add validation for mergetree setting `merge_max_block_size` to ensure that it's non zero. [#81693](https://github.com/ClickHouse/ClickHouse/pull/81693) ([Bharat Nallan](https://github.com/bharatnc)). +* Fix issues with `clickhouse-local` involving stuck `DROP VIEW ` queries. [#81705](https://github.com/ClickHouse/ClickHouse/pull/81705) ([Bharat Nallan](https://github.com/bharatnc)). +* Fix StorageRedis join in some cases. [#81736](https://github.com/ClickHouse/ClickHouse/pull/81736) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* Fix crash in `ConcurrentHashJoin` with empty `USING ()` and old analyzer enabled. [#81754](https://github.com/ClickHouse/ClickHouse/pull/81754) ([Nikita Taranov](https://github.com/nickitat)). +* Keeper fix: block commits of new logs if there is invalid entry in the logs. Previously, if leader applied some logs incorrectly, it would continue to commit new logs, even though the follower would detect digest mismatch and abort. [#81780](https://github.com/ClickHouse/ClickHouse/pull/81780) ([Antonio Andelic](https://github.com/antonio2368)). +* Fix the issue where required columns are not read during scalar correlated subquery processing. Fixes [#81716](https://github.com/ClickHouse/ClickHouse/issues/81716). [#81805](https://github.com/ClickHouse/ClickHouse/pull/81805) ([Dmitry Novik](https://github.com/novikd)). +* Someone littered our code with Kusto. Cleaned it up. This closes [#81643](https://github.com/ClickHouse/ClickHouse/issues/81643). [#81885](https://github.com/ClickHouse/ClickHouse/pull/81885) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* In previous versions, the server returned excessive content for requests to `/js`. This closes [#61890](https://github.com/ClickHouse/ClickHouse/issues/61890). [#81895](https://github.com/ClickHouse/ClickHouse/pull/81895) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Previously, `MongoDB` table engine definitions could include a path component in the `host:port` argument which was silently ignored. The mongodb integration refuses to load such tables. With this fix *we allow loading such tables and ignore path component* if `MongoDB` engine has five arguments, using the database name from arguments. *Note:* The fix is not applied for newly created tables or queries with `mongo` table function, as well as for dictionary sources and named collections. [#81942](https://github.com/ClickHouse/ClickHouse/pull/81942) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Fixed possible crash in `Aggregator` in case of exception during merge. [#82022](https://github.com/ClickHouse/ClickHouse/pull/82022) ([Nikita Taranov](https://github.com/nickitat)). +* Fixing copy-paste error in `arraySimilarity`, disallowing the use of `UInt32` and `Int32` weights. Update tests and docs. [#82103](https://github.com/ClickHouse/ClickHouse/pull/82103) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). +* Fix possible data-race between suggestion thread and main client thread. [#82233](https://github.com/ClickHouse/ClickHouse/pull/82233) ([Azat Khuzhin](https://github.com/azat)). + +#### Build/Testing/Packaging Improvement +* Use `postgres` 16.9. [#81437](https://github.com/ClickHouse/ClickHouse/pull/81437) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Use `openssl` 3.2.4. [#81438](https://github.com/ClickHouse/ClickHouse/pull/81438) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Use `abseil-cpp` 2025-01-27. [#81440](https://github.com/ClickHouse/ClickHouse/pull/81440) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Use `mongo-c-driver` 1.30.4. [#81449](https://github.com/ClickHouse/ClickHouse/pull/81449) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Use `krb5` 1.21.3-final. [#81453](https://github.com/ClickHouse/ClickHouse/pull/81453) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Use `orc` 2.1.2. [#81455](https://github.com/ClickHouse/ClickHouse/pull/81455) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Use `grpc` 1.73.0. [#81629](https://github.com/ClickHouse/ClickHouse/pull/81629) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Use `delta-kernel-rs` v0.12.1. [#81707](https://github.com/ClickHouse/ClickHouse/pull/81707) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Update `c-ares` to `v1.34.5`. [#81159](https://github.com/ClickHouse/ClickHouse/pull/81159) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Upgrade `curl` to 8.14 to address CVE-2025-5025 and CVE-2025-4947. [#81171](https://github.com/ClickHouse/ClickHouse/pull/81171) ([larryluogit](https://github.com/larryluogit)). +* Upgrade `libarchive` to 3.7.9 to address: CVE-2024-20696 CVE-2025-25724 CVE-2024-48958 CVE-2024-57970 CVE-2025-1632 CVE-2024-48957 CVE-2024-48615. [#81174](https://github.com/ClickHouse/ClickHouse/pull/81174) ([larryluogit](https://github.com/larryluogit)). +* Upgrade `libxml2` to 2.14.3. [#81187](https://github.com/ClickHouse/ClickHouse/pull/81187) ([larryluogit](https://github.com/larryluogit)). +* Avoid copying vendored Rust sources to `CARGO_HOME`. [#79560](https://github.com/ClickHouse/ClickHouse/pull/79560) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Remove dependency on the Sentry library by replacing it with our own endpoint. [#80236](https://github.com/ClickHouse/ClickHouse/pull/80236) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Update python dependencies in CI images to address Dependabot alerts. [#80658](https://github.com/ClickHouse/ClickHouse/pull/80658) ([Raúl Marín](https://github.com/Algunenano)). +* Retry reading of replicated DDL stop flag from Keeper at startup to make tests more robust when fault injection is enabled for Keeper. [#80964](https://github.com/ClickHouse/ClickHouse/pull/80964) ([Alexander Gololobov](https://github.com/davenger)). +* Use https for ubuntu archive url. [#81016](https://github.com/ClickHouse/ClickHouse/pull/81016) ([Raúl Marín](https://github.com/Algunenano)). +* Update python dependencies in test images. [#81042](https://github.com/ClickHouse/ClickHouse/pull/81042) ([dependabot[bot]](https://github.com/apps/dependabot)). +* Introduce `flake.nix` for Nix builds. [#81463](https://github.com/ClickHouse/ClickHouse/pull/81463) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Fix `delta-kernel-rs` requiring network access during build. Closes [#80609](https://github.com/ClickHouse/ClickHouse/issues/80609). [#81602](https://github.com/ClickHouse/ClickHouse/pull/81602) ([Konstantin Bogdanov](https://github.com/thevar1able)). Read the article [A Year of Rust in ClickHouse](https://clickhouse.com/blog/rust). + + +### ClickHouse release 25.5, 2025-05-22 {#255} + +#### Backward Incompatible Change +* Function `geoToH3` now accepts the input in the order (lat, lon, res) (which is common for other geometric functions). Users who wish to retain the previous result order (lon, lat, res) can set setting `geotoh3_argument_order = 'lon_lat'`. [#78852](https://github.com/ClickHouse/ClickHouse/pull/78852) ([Pratima Patel](https://github.com/pratimapatel2008)). +* Add a filesystem cache setting `allow_dynamic_cache_resize`, by default `false`, to allow dynamic resize of filesystem cache. Why: in certain environments (ClickHouse Cloud) all the scaling events happen through the restart of the process and we would love this feature to be explicitly disabled to have more control over the behaviour + as a safety measure. This PR is marked as backward incompatible, because in older versions dynamic cache resize worked by default without special setting. [#79148](https://github.com/ClickHouse/ClickHouse/pull/79148) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Removed support for legacy index types `annoy` and `usearch`. Both have been stubs for a long time, i.e. every attempt to use the legacy indexes returned an error anyways. If you still have `annoy` and `usearch` indexes, please drop them. [#79802](https://github.com/ClickHouse/ClickHouse/pull/79802) ([Robert Schulze](https://github.com/rschu1ze)). +* Remove `format_alter_commands_with_parentheses` server setting. The setting was introduced and disabled by default in 24.2. It was enabled by default in 25.2. As there are no LTS versions that don't support the new format, we can remove the setting. [#79970](https://github.com/ClickHouse/ClickHouse/pull/79970) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). +* Enable `DeltaLake` storage `delta-kernel-rs` implementation by default. [#79541](https://github.com/ClickHouse/ClickHouse/pull/79541) ([Kseniia Sumarokova](https://github.com/kssenii)). +* If reading from an `URL` involves multiple redirects, setting `enable_url_encoding` is correctly applied across all redirects in the chain. [#79563](https://github.com/ClickHouse/ClickHouse/pull/79563) ([Shankar Iyer](https://github.com/shankar-iyer)). Setting `enble_url_encoding` default value is now set to `false`. [#80088](https://github.com/ClickHouse/ClickHouse/pull/80088) ([Shankar Iyer](https://github.com/shankar-iyer)). + +#### New Feature +* Support scalar correlated subqueries in the WHERE clause. Closes [#6697](https://github.com/ClickHouse/ClickHouse/issues/6697). [#79600](https://github.com/ClickHouse/ClickHouse/pull/79600) ([Dmitry Novik](https://github.com/novikd)). Support correlated subqueries in the projection list in simple cases. [#79925](https://github.com/ClickHouse/ClickHouse/pull/79925) ([Dmitry Novik](https://github.com/novikd)). [#76078](https://github.com/ClickHouse/ClickHouse/pull/76078) ([Dmitry Novik](https://github.com/novikd)). Now it covers 100% of TPC-H test suite. +* Vector search using the vector similarity index is now beta (from previously experimental). [#80164](https://github.com/ClickHouse/ClickHouse/pull/80164) ([Robert Schulze](https://github.com/rschu1ze)). +* Support geo types in `Parquet` format. This closes [#75317](https://github.com/ClickHouse/ClickHouse/issues/75317). [#79777](https://github.com/ClickHouse/ClickHouse/pull/79777) ([scanhex12](https://github.com/scanhex12)). +* New functions `sparseGrams`, `sparseGramsHashes`, `sparseGramsHashesUTF8`, `sparseGramsUTF8` for calculating "sparse-ngrams" - a robust algorithm for extracting substrings for indexing and search. [#79517](https://github.com/ClickHouse/ClickHouse/pull/79517) ([scanhex12](https://github.com/scanhex12)). +* `clickhouse-local` (and its shorthand alias, `ch`) now use an implicit `FROM table` when there is input data for processing. This closes [#65023](https://github.com/ClickHouse/ClickHouse/issues/65023). Also enabled format inference in clickhouse-local if `--input-format` is not specified and it processes a regular file. [#79085](https://github.com/ClickHouse/ClickHouse/pull/79085) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Add `stringBytesUniq` and `stringBytesEntropy` functions to search for possibly random or encrypted data. [#79350](https://github.com/ClickHouse/ClickHouse/pull/79350) ([Sachin Kumar Singh](https://github.com/sachinkumarsingh092)). +* Added functions for encoding and decoding base32. [#79809](https://github.com/ClickHouse/ClickHouse/pull/79809) ([Joanna Hulboj](https://github.com/jh0x)). +* Add `getServerSetting` and `getMergeTreeSetting` function. Closes #78318. [#78439](https://github.com/ClickHouse/ClickHouse/pull/78439) ([NamNguyenHoai](https://github.com/NamHoaiNguyen)). +* Add new `iceberg_enable_version_hint` setting to leverage `version-hint.text` file. [#78594](https://github.com/ClickHouse/ClickHouse/pull/78594) ([Arnaud Briche](https://github.com/arnaudbriche)). +* Gives the possibility to truncate specific tables from a database, filtered with the `LIKE` keyword. [#78597](https://github.com/ClickHouse/ClickHouse/pull/78597) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Support `_part_starting_offset` virtual column in `MergeTree`-family tables. This column represents the cumulative row count of all preceding parts, calculated at query time based on the current part list. The cumulative values are retained throughout query execution and remain effective even after part pruning. Related internal logic has been refactored to support this behavior. [#79417](https://github.com/ClickHouse/ClickHouse/pull/79417) ([Amos Bird](https://github.com/amosbird)). +* Add functions `divideOrNull`,`moduloOrNull`, `intDivOrNull`,`positiveModuloOrNull` to return NULL when right argument is zero. [#78276](https://github.com/ClickHouse/ClickHouse/pull/78276) ([kevinyhzou](https://github.com/KevinyhZou)). +* Clickhouse vector search now supports both pre-filtering and post-filtering and provides related settings for finer control. (issue [#78161](https://github.com/ClickHouse/ClickHouse/issues/78161)). [#79854](https://github.com/ClickHouse/ClickHouse/pull/79854) ([Shankar Iyer](https://github.com/shankar-iyer)). +* Add [`icebergHash`](https://iceberg.apache.org/spec/#appendix-b-32-bit-hash-requirements) and [`icebergBucket`](https://iceberg.apache.org/spec/#bucket-transform-details) functions. Support data files pruning in `Iceberg` tables partitioned with [`bucket transfom`](https://iceberg.apache.org/spec/#partitioning). [#79262](https://github.com/ClickHouse/ClickHouse/pull/79262) ([Daniil Ivanik](https://github.com/divanik)). + +#### Experimental Feature +* New `Time`/`Time64` data types: `Time` (HHH:MM:SS) and `Time64` (HHH:MM:SS.``) and some basic cast functions and functions to interact with other data types. Also, changed the existing function's name toTime to toTimeWithFixedDate because the function toTime is required for the cast function. [#75735](https://github.com/ClickHouse/ClickHouse/pull/75735) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +72459). +* Hive metastore catalog for Iceberg datalake. [#77677](https://github.com/ClickHouse/ClickHouse/pull/77677) ([scanhex12](https://github.com/scanhex12)). +* Indexes of type `full_text` were renamed to `gin`. This follows the more familiar terminology of PostgreSQL and other databases. Existing indexes of type `full_text` remain loadable but they will throw an exception (suggesting `gin` indexes instead) when one tries to use them in searches. [#79024](https://github.com/ClickHouse/ClickHouse/pull/79024) ([Robert Schulze](https://github.com/rschu1ze)). + +#### Performance Improvement +* Change the Compact part format to save marks for each substream to be able to read individual subcolumns. Old Compact format is still supported for reads and can be enabled for writes using MergeTree setting `write_marks_for_substreams_in_compact_parts`. It's disabled by default for safer upgrades as it changes the compact parts storage. It will be enabled by default in one of the next releases. [#77940](https://github.com/ClickHouse/ClickHouse/pull/77940) ([Pavel Kruglov](https://github.com/Avogar)). +* Allow moving conditions with subcolumns to prewhere. [#79489](https://github.com/ClickHouse/ClickHouse/pull/79489) ([Pavel Kruglov](https://github.com/Avogar)). +* Speed up secondary indices by evaluating their expressions on multiple granules at once. [#64109](https://github.com/ClickHouse/ClickHouse/pull/64109) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Enable `compile_expressions` (JIT compiler for fragments of ordinary expressions) by default. This closes [#51264](https://github.com/ClickHouse/ClickHouse/issues/51264) and [#56386](https://github.com/ClickHouse/ClickHouse/issues/56386) and [#66486](https://github.com/ClickHouse/ClickHouse/issues/66486). [#79907](https://github.com/ClickHouse/ClickHouse/pull/79907) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* New setting introduced: `use_skip_indexes_in_final_exact_mode`. If a query on a `ReplacingMergeTree` table has FINAL clause, reading only table ranges based on skip indexes may produce incorrect result. This setting can ensure that correct results are returned by scanning newer parts that have overlap with primary key ranges returned by the skip index. Set to 0 to disable, 1 to enable. [#78350](https://github.com/ClickHouse/ClickHouse/pull/78350) ([Shankar Iyer](https://github.com/shankar-iyer)). +* Object storage cluster table functions (e.g. `s3Cluster`) will now assign files to replicas for reading based on consistent hash to improve cache locality. [#77326](https://github.com/ClickHouse/ClickHouse/pull/77326) ([Andrej Hoos](https://github.com/adikus)). +* Improve performance of `S3Queue`/`AzureQueue` by allowing INSERTs data in parallel (can be enabled with `parallel_inserts=true` queue setting). Previously S3Queue/AzureQueue can only do first part of pipeline in parallel (downloading, parsing), INSERT was single-threaded. And `INSERT`s are almost always the bottleneck. Now it will scale almost linear with `processing_threads_num`. [#77671](https://github.com/ClickHouse/ClickHouse/pull/77671) ([Azat Khuzhin](https://github.com/azat)). More fair max_processed_files_before_commit in S3Queue/AzureQueue. [#79363](https://github.com/ClickHouse/ClickHouse/pull/79363) ([Azat Khuzhin](https://github.com/azat)). +* Introduced threshold (regulated by setting `parallel_hash_join_threshold`) to fall back to the `hash` algorithm when the size of the right table is below the threshold. [#76185](https://github.com/ClickHouse/ClickHouse/pull/76185) ([Nikita Taranov](https://github.com/nickitat)). +* Now we use number of replicas to determine task size for reading with parallel replicas enabled. This provides better work distribution between replicas when the amount of data to read is not really big. [#78695](https://github.com/ClickHouse/ClickHouse/pull/78695) ([Nikita Taranov](https://github.com/nickitat)). +* Allow parallel merging of `uniqExact` states during the final stage of distributed aggregation. [#78703](https://github.com/ClickHouse/ClickHouse/pull/78703) ([Nikita Taranov](https://github.com/nickitat)). +* Fix possible performance degradation of the parallel merging of `uniqExact` states for aggregation with key. [#78724](https://github.com/ClickHouse/ClickHouse/pull/78724) ([Nikita Taranov](https://github.com/nickitat)). +* Reduce the number of List Blobs API calls to Azure storage. [#78860](https://github.com/ClickHouse/ClickHouse/pull/78860) ([Julia Kartseva](https://github.com/jkartseva)). +* Fix performance of the distributed INSERT SELECT with parallel replicas. [#79441](https://github.com/ClickHouse/ClickHouse/pull/79441) ([Azat Khuzhin](https://github.com/azat)). +* Prevent `LogSeriesLimiter` from doing cleanup on every construction, avoiding lock contention and performance regressions in high-concurrency scenarios. [#79864](https://github.com/ClickHouse/ClickHouse/pull/79864) ([filimonov](https://github.com/filimonov)). +* Speedup queries with trivial count optimization. [#79945](https://github.com/ClickHouse/ClickHouse/pull/79945) ([Raúl Marín](https://github.com/Algunenano)). +* Better inlining for some operations with `Decimal`. [#79999](https://github.com/ClickHouse/ClickHouse/pull/79999) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Set `input_format_parquet_bloom_filter_push_down` to true by default. Also, fix a mistake in the settings changes history. [#80058](https://github.com/ClickHouse/ClickHouse/pull/80058) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Optimized `ALTER ... DELETE` mutations for parts in which all rows should be deleted. Now, in such cases an empty part is created instead of original without executing a mutation. [#79307](https://github.com/ClickHouse/ClickHouse/pull/79307) ([Anton Popov](https://github.com/CurtizJ)). +* Avoid extra copying of the block during insertion into Compact part when possible. [#79536](https://github.com/ClickHouse/ClickHouse/pull/79536) ([Pavel Kruglov](https://github.com/Avogar)). +* Add setting `input_format_max_block_size_bytes` to limit blocks created in input formats in bytes. It can help to avoid high memory usage during data import when rows contains large values. [#79495](https://github.com/ClickHouse/ClickHouse/pull/79495) ([Pavel Kruglov](https://github.com/Avogar)). +* Remove guard pages for threads and async_socket_for_remote/use_hedge_requests. Change the allocation method in `FiberStack` from `mmap` to `aligned_alloc`. Since this splits VMAs and under heavy load vm.max_map_count can be reached. [#79147](https://github.com/ClickHouse/ClickHouse/pull/79147) ([Sema Checherinda](https://github.com/CheSema)). +* Lazy Materialization with parallel replicas. [#79401](https://github.com/ClickHouse/ClickHouse/pull/79401) ([Igor Nikonov](https://github.com/devcrafter)). + +#### Improvement +* Added an ability to apply lightweight deletes on the fly (with settings `lightweight_deletes_sync = 0`, `apply_mutations_on_fly = 1`. [#79281](https://github.com/ClickHouse/ClickHouse/pull/79281) ([Anton Popov](https://github.com/CurtizJ)). +* If data in the pretty format is displayed in the terminal, and a subsequent block has the same column widths, it can continue from the previous block, glue it to the previous block by moving the cursor up. This closes [#79333](https://github.com/ClickHouse/ClickHouse/issues/79333). The feature is controlled by the new setting, `output_format_pretty_glue_chunks`. [#79339](https://github.com/ClickHouse/ClickHouse/pull/79339) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Extend the `isIPAddressInRange` function to `String`, `IPv4`, `IPv6`, `Nullable(String)`, `Nullable(IPv4)`, and `Nullable(IPv6)` data types. [#78364](https://github.com/ClickHouse/ClickHouse/pull/78364) ([YjyJeff](https://github.com/YjyJeff)). +* Allow changing `PostgreSQL` engine connection pooler settings dynamically. [#78414](https://github.com/ClickHouse/ClickHouse/pull/78414) ([Samay Sharma](https://github.com/samay-sharma)). +* Allow to specify `_part_offset` in normal projection. This is the first step to build projection index. It can be used with [#58224](https://github.com/ClickHouse/ClickHouse/issues/58224) and can help improve #63207. [#78429](https://github.com/ClickHouse/ClickHouse/pull/78429) ([Amos Bird](https://github.com/amosbird)). +* Add new columns (`create_query` and `source`) for `system.named_collections`. Closes [#78179](https://github.com/ClickHouse/ClickHouse/issues/78179). [#78582](https://github.com/ClickHouse/ClickHouse/pull/78582) ([MikhailBurdukov](https://github.com/MikhailBurdukov)). +* Added a new field `condition` to system table `system.query_condition_cache`. It stores the plaintext condition whose hash is used as a key in the query condition cache. [#78671](https://github.com/ClickHouse/ClickHouse/pull/78671) ([Robert Schulze](https://github.com/rschu1ze)). +* Vector similarity indexes can now be created on top of `BFloat16` columns. [#78850](https://github.com/ClickHouse/ClickHouse/pull/78850) ([Robert Schulze](https://github.com/rschu1ze)). +* Support unix timestapms with fractional part in best effort `DateTime64` parsing. [#78908](https://github.com/ClickHouse/ClickHouse/pull/78908) ([Pavel Kruglov](https://github.com/Avogar)). +* In the storage `DeltaLake` delta-kernel implementation, fix for column mapping mode, add tests for schema evolution. [#78921](https://github.com/ClickHouse/ClickHouse/pull/78921) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Improve insert into `Variant` column in Values format by better conversion of values. [#78923](https://github.com/ClickHouse/ClickHouse/pull/78923) ([Pavel Kruglov](https://github.com/Avogar)). +* The `tokens` function was extended to accept an additional "tokenizer" argument plus further tokenizer-specific arguments. [#79001](https://github.com/ClickHouse/ClickHouse/pull/79001) ([Elmi Ahmadov](https://github.com/ahmadov)). +* The `SHOW CLUSTER` statement now expands macros (if any) in its argument. [#79006](https://github.com/ClickHouse/ClickHouse/pull/79006) ([arf42](https://github.com/arf42)). +* Hash functions now support `NULL`s inside arrays, tuples, and maps. (issues [#48365](https://github.com/ClickHouse/ClickHouse/issues/48365) and [#48623](https://github.com/ClickHouse/ClickHouse/issues/48623)). [#79008](https://github.com/ClickHouse/ClickHouse/pull/79008) ([Michael Kolupaev](https://github.com/al13n321)). +* Update cctz to 2025a. [#79043](https://github.com/ClickHouse/ClickHouse/pull/79043) ([Raúl Marín](https://github.com/Algunenano)). +* Change the default stderr processing for UDFs to "log_last". It's better for usability. [#79066](https://github.com/ClickHouse/ClickHouse/pull/79066) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Make tabs undo-able in the Web UI. This closes [#71284](https://github.com/ClickHouse/ClickHouse/issues/71284). [#79084](https://github.com/ClickHouse/ClickHouse/pull/79084) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Remove settings during `recoverLostReplica` same as it was done in: https://github.com/ClickHouse/ClickHouse/pull/78637. [#79113](https://github.com/ClickHouse/ClickHouse/pull/79113) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Add profile events: `ParquetReadRowGroups` and `ParquetPrunedRowGroups` to profile parquet index prune. [#79180](https://github.com/ClickHouse/ClickHouse/pull/79180) ([flynn](https://github.com/ucasfl)). +* Support `ALTER`ing database on cluster. [#79242](https://github.com/ClickHouse/ClickHouse/pull/79242) ([Tuan Pham Anh](https://github.com/tuanpach)). +* Explicitly skip missed runs of statistics collection for QueryMetricLog, otherwise the log will take a long time to catch up with the current time. [#79257](https://github.com/ClickHouse/ClickHouse/pull/79257) ([Mikhail Artemenko](https://github.com/Michicosun)). +* Some small optimizations for reading `Arrow`-based formats. [#79308](https://github.com/ClickHouse/ClickHouse/pull/79308) ([Bharat Nallan](https://github.com/bharatnc)). +* The setting `allow_archive_path_syntax` was marked as experimental by mistake. Add a test to prevent having experimental settings enabled by default. [#79320](https://github.com/ClickHouse/ClickHouse/pull/79320) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Made page cache settings adjustable on a per-query level. This is needed for faster experimentation and for the possibility of fine-tuning for high-throughput and low-latency queries. [#79337](https://github.com/ClickHouse/ClickHouse/pull/79337) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Do not print number tips in pretty formats for numbers that look like most of the 64-bit hashes. This closes [#79334](https://github.com/ClickHouse/ClickHouse/issues/79334). [#79338](https://github.com/ClickHouse/ClickHouse/pull/79338) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Colors of graphs on the advanced dashboards will be calculated from the hash of the corresponding query. This makes it easier to remember and locate a graph while scrolling the dashboard. [#79341](https://github.com/ClickHouse/ClickHouse/pull/79341) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Add asynchronous metric, `FilesystemCacheCapacity` - total capacity in the `cache` virtual filesystem. This is useful for global infrastructure monitoring. [#79348](https://github.com/ClickHouse/ClickHouse/pull/79348) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Optimize access to system.parts (read columns/indexes size only when requested). [#79352](https://github.com/ClickHouse/ClickHouse/pull/79352) ([Azat Khuzhin](https://github.com/azat)). +* Calculate the relevant fields for query `'SHOW CLUSTER '` instead of all fields. [#79368](https://github.com/ClickHouse/ClickHouse/pull/79368) ([Tuan Pham Anh](https://github.com/tuanpach)). +* Allow to specify storage settings for `DatabaseCatalog`. [#79407](https://github.com/ClickHouse/ClickHouse/pull/79407) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Support local storage in `DeltaLake`. [#79416](https://github.com/ClickHouse/ClickHouse/pull/79416) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Add a query level setting to enable delta-kernel-rs: `allow_experimental_delta_kernel_rs`. [#79418](https://github.com/ClickHouse/ClickHouse/pull/79418) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix possible endless loop when listing blobs from Azure/S3 blob storage. [#79425](https://github.com/ClickHouse/ClickHouse/pull/79425) ([Alexander Gololobov](https://github.com/davenger)). +* Add filesystem cache setting `max_size_ratio_to_total_space`. [#79460](https://github.com/ClickHouse/ClickHouse/pull/79460) ([Kseniia Sumarokova](https://github.com/kssenii)). +* For `clickhouse-benchmark` reconfigure `reconnect` option to take 0, 1 or N as values for reconnecting accordingly. [#79465](https://github.com/ClickHouse/ClickHouse/pull/79465) ([Sachin Kumar Singh](https://github.com/sachinkumarsingh092)). +* Allow `ALTER TABLE ... MOVE|REPLACE PARTITION` for tables on different `plain_rewritable` disks. [#79566](https://github.com/ClickHouse/ClickHouse/pull/79566) ([Julia Kartseva](https://github.com/jkartseva)). +* The vector similarity index is now also used if the reference vector is of type `Array(BFloat16)`. [#79745](https://github.com/ClickHouse/ClickHouse/pull/79745) ([Shankar Iyer](https://github.com/shankar-iyer)). +* Add last_error_message, last_error_trace and query_id to the system.error_log table. Related ticket [#75816](https://github.com/ClickHouse/ClickHouse/issues/75816). [#79836](https://github.com/ClickHouse/ClickHouse/pull/79836) ([Andrei Tinikov](https://github.com/Dolso)). +* Enable sending crash reports by default. This can be turned off in the server's configuration file. [#79838](https://github.com/ClickHouse/ClickHouse/pull/79838) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* System table `system.functions` now shows in which ClickHouse version functions were first introduced. [#79839](https://github.com/ClickHouse/ClickHouse/pull/79839) ([Robert Schulze](https://github.com/rschu1ze)). +* Added `access_control_improvements.enable_user_name_access_type` setting. This setting allows enabling/disabling of precise grants for users/roles, introduced in https://github.com/ClickHouse/ClickHouse/pull/72246. You may want to turn this setting off in case you have a cluster with the replicas older than 25.1. [#79842](https://github.com/ClickHouse/ClickHouse/pull/79842) ([pufit](https://github.com/pufit)). +* Proper implementation of `ASTSelectWithUnionQuery::clone()` method now takes into account `is_normalized` field as well. This might help with [#77569](https://github.com/ClickHouse/ClickHouse/issues/77569). [#79909](https://github.com/ClickHouse/ClickHouse/pull/79909) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Fix the inconsistent formatting of certain queries with the EXCEPT operator. If the left-hand side of the EXCEPT operator ends with `*`, the formatted query loses parentheses and is then parsed as a `*` with the `EXCEPT` modifier. These queries are found by the fuzzer and are unlikely to be found in practice. This closes [#79950](https://github.com/ClickHouse/ClickHouse/issues/79950). [#79952](https://github.com/ClickHouse/ClickHouse/pull/79952) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Small improvement in `JSON` type parsing by using cache of variants deserialization order. [#79984](https://github.com/ClickHouse/ClickHouse/pull/79984) ([Pavel Kruglov](https://github.com/Avogar)). +* Add setting `s3_slow_all_threads_after_network_error`. [#80035](https://github.com/ClickHouse/ClickHouse/pull/80035) ([Vitaly Baranov](https://github.com/vitlibar)). +* The logging level about the selected parts to merge was wrong (Information). Closes [#80061](https://github.com/ClickHouse/ClickHouse/issues/80061). [#80062](https://github.com/ClickHouse/ClickHouse/pull/80062) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* trace-visualizer: add runtime/share in tooltips and status messages. [#79040](https://github.com/ClickHouse/ClickHouse/pull/79040) ([Sergei Trifonov](https://github.com/serxa)). +* trace-visualizer: load data from clickhouse server. [#79042](https://github.com/ClickHouse/ClickHouse/pull/79042) ([Sergei Trifonov](https://github.com/serxa)). +* Add metrics on failing merges. [#79228](https://github.com/ClickHouse/ClickHouse/pull/79228) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). +* `clickhouse-benchmark` will display percentage based on the max iterations if specified. [#79346](https://github.com/ClickHouse/ClickHouse/pull/79346) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Add system.parts table visualizer. [#79437](https://github.com/ClickHouse/ClickHouse/pull/79437) ([Sergei Trifonov](https://github.com/serxa)). +* Add tool for query latency analyzing. [#79978](https://github.com/ClickHouse/ClickHouse/pull/79978) ([Sergei Trifonov](https://github.com/serxa)). + +#### Bug Fix (user-visible misbehavior in an official stable release) +* Fix renames of columns missing in part. [#76346](https://github.com/ClickHouse/ClickHouse/pull/76346) ([Anton Popov](https://github.com/CurtizJ)). +* A materialized view can start too late, e.g. after the Kafka table that streams to it. [#72123](https://github.com/ClickHouse/ClickHouse/pull/72123) ([Ilya Golshtein](https://github.com/ilejn)). +* Fix `SELECT` query rewriting during `VIEW` creation with enabled analyzer. closes [#75956](https://github.com/ClickHouse/ClickHouse/issues/75956). [#76356](https://github.com/ClickHouse/ClickHouse/pull/76356) ([Dmitry Novik](https://github.com/novikd)). +* Fix applying `async_insert` from server (via `apply_settings_from_server`) (previously leads to `Unknown packet 11 from server` errors on the client). [#77578](https://github.com/ClickHouse/ClickHouse/pull/77578) ([Azat Khuzhin](https://github.com/azat)). +* Fixed refreshable materialized view in Replicated database not working on newly added replicas. [#77774](https://github.com/ClickHouse/ClickHouse/pull/77774) ([Michael Kolupaev](https://github.com/al13n321)). +* Fixed refreshable materialized views breaking backups. [#77893](https://github.com/ClickHouse/ClickHouse/pull/77893) ([Michael Kolupaev](https://github.com/al13n321)). +* Fix old firing logical error for `transform`. [#78247](https://github.com/ClickHouse/ClickHouse/pull/78247) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Fix some cases where secondary index was not applied with analyzer. Fixes [#65607](https://github.com/ClickHouse/ClickHouse/issues/65607) , fixes [#69373](https://github.com/ClickHouse/ClickHouse/issues/69373). [#78485](https://github.com/ClickHouse/ClickHouse/pull/78485) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix dumping profile events (`NetworkSendElapsedMicroseconds`/`NetworkSendBytes`) for HTTP protocol with compression enabled (the error should not be more then the buffer size, usually around 1MiB). [#78516](https://github.com/ClickHouse/ClickHouse/pull/78516) ([Azat Khuzhin](https://github.com/azat)). +* Fix analyzer producing LOGICAL_ERROR when JOIN ... USING involves ALIAS column - should produce appropriate error. [#78618](https://github.com/ClickHouse/ClickHouse/pull/78618) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* Fix analyzer: CREATE VIEW ... ON CLUSTER fails if SELECT contains positional arguments. [#78663](https://github.com/ClickHouse/ClickHouse/pull/78663) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* Fix `Block structure mismatch` error in case of `INSERT SELECT` into table a function with schema inference if `SELECT` has scalar subqueries. [#78677](https://github.com/ClickHouse/ClickHouse/pull/78677) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* Fix analyzer: with prefer_global_in_and_join=1 for Distributed table in SELECT query `in` function should be replaced by `globalIn`. [#78749](https://github.com/ClickHouse/ClickHouse/pull/78749) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* Fixed several types of `SELECT` queries that read from tables with `MongoDB` engine or `mongodb` table function: queries with implicit conversion of const value in `WHERE` clause (e.g. `WHERE datetime = '2025-03-10 00:00:00'`) ; queries with `LIMIT` and `GROUP BY`. Previously, they could return the wrong result. [#78777](https://github.com/ClickHouse/ClickHouse/pull/78777) ([Anton Popov](https://github.com/CurtizJ)). +* Fix conversion between different JSON types. Not it's performed by simple cast through convertion to/from String. It's less effective but 100% accurate. [#78807](https://github.com/ClickHouse/ClickHouse/pull/78807) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix logical error during convertion of Dynamic type to Interval. [#78813](https://github.com/ClickHouse/ClickHouse/pull/78813) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix column rollback on JSON parsing error. [#78836](https://github.com/ClickHouse/ClickHouse/pull/78836) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix 'bad cast' error when join using constant alias column. [#78848](https://github.com/ClickHouse/ClickHouse/pull/78848) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Don't allow prewhere in materialized view on columns with different types in view and target table. [#78889](https://github.com/ClickHouse/ClickHouse/pull/78889) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix logical error during parsing of bad binary data of Variant column. [#78982](https://github.com/ClickHouse/ClickHouse/pull/78982) ([Pavel Kruglov](https://github.com/Avogar)). +* Throw an exception when the parquet batch size is set to 0. Previously when output_format_parquet_batch_size = 0 ClickHouse would hang. Now this behavior is fixed. [#78991](https://github.com/ClickHouse/ClickHouse/pull/78991) ([daryawessely](https://github.com/daryawessely)). +* Fix deserialization of variant discriminators with basic format in compact parts. It was introduced in https://github.com/ClickHouse/ClickHouse/pull/55518. [#79000](https://github.com/ClickHouse/ClickHouse/pull/79000) ([Pavel Kruglov](https://github.com/Avogar)). +* Dictionaries of type `complex_key_ssd_cache` now reject zero or negative `block_size` and `write_buffer_size` parameters (issue [#78314](https://github.com/ClickHouse/ClickHouse/issues/78314)). [#79028](https://github.com/ClickHouse/ClickHouse/pull/79028) ([Elmi Ahmadov](https://github.com/ahmadov)). +* Avoid using Field for non-aggregated columns in SummingMergeTree. It could lead to unexpected errors with Dynamic/Variant types used in SummingMergeTree. [#79051](https://github.com/ClickHouse/ClickHouse/pull/79051) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix read from Materialized View with Distributed destination table and different header in analyzer. [#79059](https://github.com/ClickHouse/ClickHouse/pull/79059) ([Pavel Kruglov](https://github.com/Avogar)). +* Fixes a bug where `arrayUnion()` returned extra (incorrect) values on tables that had batch inserts. Fixes [#75057](https://github.com/ClickHouse/ClickHouse/issues/75057). [#79079](https://github.com/ClickHouse/ClickHouse/pull/79079) ([Peter Nguyen](https://github.com/petern48)). +* Fix segfault in `OpenSSLInitializer`. Closes [#79092](https://github.com/ClickHouse/ClickHouse/issues/79092). [#79097](https://github.com/ClickHouse/ClickHouse/pull/79097) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Always set prefix for S3 ListObject. [#79114](https://github.com/ClickHouse/ClickHouse/pull/79114) ([Azat Khuzhin](https://github.com/azat)). +* Fixes a bug where arrayUnion() returned extra (incorrect) values on tables that had batch inserts. Fixes [#79157](https://github.com/ClickHouse/ClickHouse/issues/79157). [#79158](https://github.com/ClickHouse/ClickHouse/pull/79158) ([Peter Nguyen](https://github.com/petern48)). +* Fix logical error after filter pushdown. [#79164](https://github.com/ClickHouse/ClickHouse/pull/79164) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* Fix DeltaLake table engine with delta-kernel implementation being used with http based endpoints, fix NOSIGN. Closes [#78124](https://github.com/ClickHouse/ClickHouse/issues/78124). [#79203](https://github.com/ClickHouse/ClickHouse/pull/79203) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Keeper fix: Avoid triggering watches on failed multi requests. [#79247](https://github.com/ClickHouse/ClickHouse/pull/79247) ([Antonio Andelic](https://github.com/antonio2368)). +* Forbid Dynamic and JSON types in IN. With current implementation of `IN` it can lead to incorrect results. Proper support of this types in `IN` is complicated and can be done in future. [#79282](https://github.com/ClickHouse/ClickHouse/pull/79282) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix check for duplicate paths in JSON type parsing. [#79317](https://github.com/ClickHouse/ClickHouse/pull/79317) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix SecureStreamSocket connection issues. [#79383](https://github.com/ClickHouse/ClickHouse/pull/79383) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Fix loading of plain_rewritable disks containing data. [#79439](https://github.com/ClickHouse/ClickHouse/pull/79439) ([Julia Kartseva](https://github.com/jkartseva)). +* Fix crash in dynamic subcolumns discovery in Wide parts in MergeTree. [#79466](https://github.com/ClickHouse/ClickHouse/pull/79466) ([Pavel Kruglov](https://github.com/Avogar)). +* Verify the table name's length only for initial create queries. Do not verify this for secondary creates to avoid backward compatibility issues. [#79488](https://github.com/ClickHouse/ClickHouse/pull/79488) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). +* Fixed error `Block structure mismatch` in several cases with tables with sparse columns. [#79491](https://github.com/ClickHouse/ClickHouse/pull/79491) ([Anton Popov](https://github.com/CurtizJ)). +* Fix two cases of "Logical Error: Can't set alias of * of Asterisk on alias". [#79505](https://github.com/ClickHouse/ClickHouse/pull/79505) ([Raúl Marín](https://github.com/Algunenano)). +* Fix using incorrect paths when renaming an Atomic database. [#79569](https://github.com/ClickHouse/ClickHouse/pull/79569) ([Tuan Pham Anh](https://github.com/tuanpach)). +* Fix order by JSON column with other columns. [#79591](https://github.com/ClickHouse/ClickHouse/pull/79591) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix result duplication when reading from remote with both `use_hedged_requests` and `allow_experimental_parallel_reading_from_replicas` disabled. [#79599](https://github.com/ClickHouse/ClickHouse/pull/79599) ([Eduard Karacharov](https://github.com/korowa)). +* Fix crash in delta-kernel implementation when using unity catalog. [#79677](https://github.com/ClickHouse/ClickHouse/pull/79677) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Resolve macros for autodiscovery clusters. [#79696](https://github.com/ClickHouse/ClickHouse/pull/79696) ([Anton Ivashkin](https://github.com/ianton-ru)). +* Handle incorrectly configured page_cache_limits suitably. [#79805](https://github.com/ClickHouse/ClickHouse/pull/79805) ([Bharat Nallan](https://github.com/bharatnc)). +* Fixes the result of SQL function `formatDateTime` if a variable-size formatter (e.g. `%W` aka. weekday `Monday` `Tuesday`, etc.) is followed by a compound formatter (a formatter that prints multiple components at once, e.g. `%D` aka. the American date `05/04/25`). [#79835](https://github.com/ClickHouse/ClickHouse/pull/79835) ([Robert Schulze](https://github.com/rschu1ze)). +* IcebergS3 supports count optimization, but IcebergS3Cluster does not. As a result, the count() result returned in cluster mode may be a multiple of the number of replicas. [#79844](https://github.com/ClickHouse/ClickHouse/pull/79844) ([wxybear](https://github.com/wxybear)). +* Fixes AMBIGUOUS_COLUMN_NAME error with lazy materialization when no columns are used for query execution until projection. Example, SELECT * FROM t ORDER BY rand() LIMIT 5. [#79926](https://github.com/ClickHouse/ClickHouse/pull/79926) ([Igor Nikonov](https://github.com/devcrafter)). +* Hide password for query `CREATE DATABASE datalake ENGINE = DataLakeCatalog(\'http://catalog:8181\', \'admin\', \'password\')`. [#79941](https://github.com/ClickHouse/ClickHouse/pull/79941) ([Han Fei](https://github.com/hanfei1991)). +* Allow to specify an alias in JOIN USING. Specify this alias in case the column was renamed (e.g., because of ARRAY JOIN). Fixes [#73707](https://github.com/ClickHouse/ClickHouse/issues/73707). [#79942](https://github.com/ClickHouse/ClickHouse/pull/79942) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Allow materialized views with UNIONs to work correctly on new replicas. [#80037](https://github.com/ClickHouse/ClickHouse/pull/80037) ([Samay Sharma](https://github.com/samay-sharma)). +* Format specifier `%e` in SQL function `parseDateTime` now recognizes single-digit days (e.g. `3`), whereas it previously required space padding (e.g. ` 3`). This makes its behavior compatible with MySQL. To retain the previous behaviour, set setting `parsedatetime_e_requires_space_padding = 1`. (issue [#78243](https://github.com/ClickHouse/ClickHouse/issues/78243)). [#80057](https://github.com/ClickHouse/ClickHouse/pull/80057) ([Robert Schulze](https://github.com/rschu1ze)). +* Fix warnings `Cannot find 'kernel' in '[...]/memory.stat'` in ClickHouse's log (issue [#77410](https://github.com/ClickHouse/ClickHouse/issues/77410)). [#80129](https://github.com/ClickHouse/ClickHouse/pull/80129) ([Robert Schulze](https://github.com/rschu1ze)). +* Check stack size in FunctionComparison to avoid stack overflow crash. [#78208](https://github.com/ClickHouse/ClickHouse/pull/78208) ([Julia Kartseva](https://github.com/jkartseva)). +* Fix race during SELECT from `system.workloads`. [#78743](https://github.com/ClickHouse/ClickHouse/pull/78743) ([Sergei Trifonov](https://github.com/serxa)). +* Fix: lazy materialization in distributed queries. [#78815](https://github.com/ClickHouse/ClickHouse/pull/78815) ([Igor Nikonov](https://github.com/devcrafter)). +* Fix `Array(Bool)` to `Array(FixedString)` conversion. [#78863](https://github.com/ClickHouse/ClickHouse/pull/78863) ([Nikita Taranov](https://github.com/nickitat)). +* Make parquet version selection less confusing. [#78818](https://github.com/ClickHouse/ClickHouse/pull/78818) ([Michael Kolupaev](https://github.com/al13n321)). +* Fix `ReservoirSampler` self-merging. [#79031](https://github.com/ClickHouse/ClickHouse/pull/79031) ([Nikita Taranov](https://github.com/nickitat)). +* Fix storage of insertion table in client context. [#79046](https://github.com/ClickHouse/ClickHouse/pull/79046) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* Fix the destruction order of data members of `AggregatingSortedAlgorithm` and `SummingSortedAlgorithm`. [#79056](https://github.com/ClickHouse/ClickHouse/pull/79056) ([Nikita Taranov](https://github.com/nickitat)). +* `enable_user_name_access_type` must not affect `DEFINER` access type. [#80026](https://github.com/ClickHouse/ClickHouse/pull/80026) ([pufit](https://github.com/pufit)). +* Query to system database can hang if system database metadata located in keeper. [#79304](https://github.com/ClickHouse/ClickHouse/pull/79304) ([Mikhail Artemenko](https://github.com/Michicosun)). + +#### Build/Testing/Packaging Improvement +* Make it possible to reuse the built `chcache` binary instead of always rebuilding it. [#78851](https://github.com/ClickHouse/ClickHouse/pull/78851) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). +* Add NATS pause waiting. [#78987](https://github.com/ClickHouse/ClickHouse/pull/78987) ([Dmitry Novikov](https://github.com/dmitry-sles-novikov)). +* Fix for incorrectly publishing ARM build as amd64compat. [#79122](https://github.com/ClickHouse/ClickHouse/pull/79122) ([Alexander Gololobov](https://github.com/davenger)). +* Use generated ahead of time assembly for OpenSSL. [#79386](https://github.com/ClickHouse/ClickHouse/pull/79386) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Fixes to allow building with `clang20`. [#79588](https://github.com/ClickHouse/ClickHouse/pull/79588) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* `chcache`: Rust caching support. [#78691](https://github.com/ClickHouse/ClickHouse/pull/78691) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Add unwind information for `zstd` assembly files. [#79288](https://github.com/ClickHouse/ClickHouse/pull/79288) ([Michael Kolupaev](https://github.com/al13n321)). + + +### ClickHouse release 25.4, 2025-04-22 {#254} + +#### Backward Incompatible Change +* Check if all columns in a materialized view match the target table when `allow_materialized_view_with_bad_select` is `false`. [#74481](https://github.com/ClickHouse/ClickHouse/pull/74481) ([Christoph Wurm](https://github.com/cwurm)). +* Fix cases where `dateTrunc` is used with negative Date/DateTime arguments. [#77622](https://github.com/ClickHouse/ClickHouse/pull/77622) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* The legacy `MongoDB` integration has been removed. Server setting `use_legacy_mongodb_integration` became obsolete and now does nothing. [#77895](https://github.com/ClickHouse/ClickHouse/pull/77895) ([Robert Schulze](https://github.com/rschu1ze)). +* Enhance `SummingMergeTree` validation to skip aggregation for columns used in partition or sort keys. [#78022](https://github.com/ClickHouse/ClickHouse/pull/78022) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). + +#### New Feature +* Added CPU slot scheduling for workloads, see [the docs](https://clickhouse.com/docs/operations/workload-scheduling#cpu_scheduling) for details. [#77595](https://github.com/ClickHouse/ClickHouse/pull/77595) ([Sergei Trifonov](https://github.com/serxa)). +* `clickhouse-local` will retain its databases after restart if you specify the `--path` command line argument. This closes [#50647](https://github.com/ClickHouse/ClickHouse/issues/50647). This closes [#49947](https://github.com/ClickHouse/ClickHouse/issues/49947). [#71722](https://github.com/ClickHouse/ClickHouse/pull/71722) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Reject queries when the server is overloaded. The decision is made based on the ratio of wait time (`OSCPUWaitMicroseconds`) to busy time (`OSCPUVirtualTimeMicroseconds`). The query is dropped with some probability, when this ratio is between `min_os_cpu_wait_time_ratio_to_throw` and `max_os_cpu_wait_time_ratio_to_throw` (those are query level settings). [#63206](https://github.com/ClickHouse/ClickHouse/pull/63206) ([Alexey Katsman](https://github.com/alexkats)). +* Time travel in `Iceberg`: add setting to query `Iceberg` tables as of a specific timestamp. [#71072](https://github.com/ClickHouse/ClickHouse/pull/71072) ([Brett Hoerner](https://github.com/bretthoerner)). [#77439](https://github.com/ClickHouse/ClickHouse/pull/77439) ([Daniil Ivanik](https://github.com/divanik)). +* An in-memory cache for `Iceberg` metadata, which stores manifest files/list and `metadata.json` to speed up queries. [#77156](https://github.com/ClickHouse/ClickHouse/pull/77156) ([Han Fei](https://github.com/hanfei1991)). +* Support `DeltaLake` table engine for Azure Blob Storage. Fixes [#68043](https://github.com/ClickHouse/ClickHouse/issues/68043). [#74541](https://github.com/ClickHouse/ClickHouse/pull/74541) ([Smita Kulkarni](https://github.com/SmitaRKulkarni)). +* Added an in-memory cache for deserialized vector similarity indexes. This should make repeated approximate nearest neighbor (ANN) search queries faster. The size of the new cache is controlled by server settings `vector_similarity_index_cache_size` and `vector_similarity_index_cache_max_entries`. This feature supersedes the skipping index cache feature of earlier releases. [#77905](https://github.com/ClickHouse/ClickHouse/pull/77905) ([Shankar Iyer](https://github.com/shankar-iyer)). +* Support partition pruning in DeltaLake. [#78486](https://github.com/ClickHouse/ClickHouse/pull/78486) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Support for a background refresh in readonly `MergeTree` tables which allows querying updateable tables with an infinite amount of distributed readers (ClickHouse-native data lake). [#76467](https://github.com/ClickHouse/ClickHouse/pull/76467) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Support using custom disks to store databases metadata files. Currently it can be configured only on a global server level. [#77365](https://github.com/ClickHouse/ClickHouse/pull/77365) ([Tuan Pham Anh](https://github.com/tuanpach)). +* Support ALTER TABLE ... ATTACH|DETACH|MOVE|REPLACE PARTITION for the plain_rewritable disk. [#77406](https://github.com/ClickHouse/ClickHouse/pull/77406) ([Julia Kartseva](https://github.com/jkartseva)). +* Add table settings for `SASL` configuration and credentials to the `Kafka` table engine. This allows configuring SASL-based authentication to Kafka and Kafka-compatible systems directly in the CREATE TABLE statement rather than having to use configuration files or named collections. [#78810](https://github.com/ClickHouse/ClickHouse/pull/78810) ([Christoph Wurm](https://github.com/cwurm)). +* Allow setting `default_compression_codec` for MergeTree tables: it is used when the CREATE query does not explicitly define one for the given columns. This closes [#42005](https://github.com/ClickHouse/ClickHouse/issues/42005). [#66394](https://github.com/ClickHouse/ClickHouse/pull/66394) ([gvoelfin](https://github.com/gvoelfin)). +* Add `bind_host` setting in the clusters configuration so that ClickHouse can use a specific network for distributed connections. [#74741](https://github.com/ClickHouse/ClickHouse/pull/74741) ([Todd Yocum](https://github.com/toddyocum)). +* Introduce a new column, `parametrized_view_parameters` in `system.tables`. Closes https://github.com/clickhouse/clickhouse/issues/66756. [#75112](https://github.com/ClickHouse/ClickHouse/pull/75112) ([NamNguyenHoai](https://github.com/NamHoaiNguyen)). +* Allow changing a database comment. Closes [#73351](https://github.com/ClickHouse/ClickHouse/issues/73351) ### Documentation entry for user-facing changes. [#75622](https://github.com/ClickHouse/ClickHouse/pull/75622) ([NamNguyenHoai](https://github.com/NamHoaiNguyen)). +* Support `SCRAM-SHA-256` authentication in the PostgreSQL compatibility protocol. [#76839](https://github.com/ClickHouse/ClickHouse/pull/76839) ([scanhex12](https://github.com/scanhex12)). +* Add functions `arrayLevenshteinDistance`, `arrayLevenshteinDistanceWeighted`, and `arraySimilarity`. [#77187](https://github.com/ClickHouse/ClickHouse/pull/77187) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). +* The setting `parallel_distributed_insert_select` makes effect for `INSERT SELECT` into `ReplicatedMergeTree` (previously it required Distribued tables). [#78041](https://github.com/ClickHouse/ClickHouse/pull/78041) ([Igor Nikonov](https://github.com/devcrafter)). +* Introduce `toInterval` function. This function accepts 2 arguments (value and unit), and converts the value to a specific `Interval` type. [#78723](https://github.com/ClickHouse/ClickHouse/pull/78723) ([Andrew Davis](https://github.com/pulpdrew)). +* Add several convenient ways to resolve root `metadata.json` file in an iceberg table function and engine. Closes [#78455](https://github.com/ClickHouse/ClickHouse/issues/78455). [#78475](https://github.com/ClickHouse/ClickHouse/pull/78475) ([Daniil Ivanik](https://github.com/divanik)). +* Support password based auth in SSH protocol in ClickHouse. [#78586](https://github.com/ClickHouse/ClickHouse/pull/78586) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). + +#### Experimental Feature +* Support correlated subqueries as an argument of `EXISTS` expression in the `WHERE` clause. Closes [#72459](https://github.com/ClickHouse/ClickHouse/issues/72459). [#76078](https://github.com/ClickHouse/ClickHouse/pull/76078) ([Dmitry Novik](https://github.com/novikd)). +* Functions `sparseGrams` and `sparseGramsHashes` with ASCII and UTF8 versions added. Author: [scanhex12](https://github.com/scanhex12). [#78176](https://github.com/ClickHouse/ClickHouse/pull/78176) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). Do not use it: the implementation will change in the next versions. + +#### Performance Improvement +* Optimize performance with lazy columns, that read the data after ORDER BY and LIMIT. [#55518](https://github.com/ClickHouse/ClickHouse/pull/55518) ([Xiaozhe Yu](https://github.com/wudidapaopao)). +* Enabled the query condition cache by default. [#79080](https://github.com/ClickHouse/ClickHouse/pull/79080) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Speed-up building JOIN result by de-virtualizing calls to `col->insertFrom()`. [#77350](https://github.com/ClickHouse/ClickHouse/pull/77350) ([Alexander Gololobov](https://github.com/davenger)). +* Merge equality conditions from filter query plan step into JOIN condition if possible to allow using them as hash table keys. [#78877](https://github.com/ClickHouse/ClickHouse/pull/78877) ([Dmitry Novik](https://github.com/novikd)). +* Use dynamic sharding for JOIN if the JOIN key is a prefix of PK for both parts. This optimization is enabled with `query_plan_join_shard_by_pk_ranges` setting (disabled by default). [#74733](https://github.com/ClickHouse/ClickHouse/pull/74733) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Support `Iceberg` data pruning based on lower and upper boundary values for columns. Fixes [#77638](https://github.com/ClickHouse/ClickHouse/issues/77638). [#78242](https://github.com/ClickHouse/ClickHouse/pull/78242) ([alesapin](https://github.com/alesapin)). +* Implement trivial count optimization for `Iceberg`. Now queries with `count()` and without any filters should be faster. Closes [#77639](https://github.com/ClickHouse/ClickHouse/issues/77639). [#78090](https://github.com/ClickHouse/ClickHouse/pull/78090) ([alesapin](https://github.com/alesapin)). +* Add ability to configure the number of columns that merges can flush in parallel using `max_merge_delayed_streams_for_parallel_write` (this should reduce memory usage for vertical merges to S3 about 25x times). [#77922](https://github.com/ClickHouse/ClickHouse/pull/77922) ([Azat Khuzhin](https://github.com/azat)). +* Disable `filesystem_cache_prefer_bigger_buffer_size` when the cache is used passively, such as for merges. This lowers memory consumption on merges. [#77898](https://github.com/ClickHouse/ClickHouse/pull/77898) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Now we use number of replicas to determine task size for reading with parallel replicas enabled. This provides better work distribution between replicas when the amount of data to read is not really big. [#78695](https://github.com/ClickHouse/ClickHouse/pull/78695) ([Nikita Taranov](https://github.com/nickitat)). +* Support asynchronous IO prefetch for the `ORC` format, which improves overall performance by hiding remote IO latency. [#70534](https://github.com/ClickHouse/ClickHouse/pull/70534) ([李扬](https://github.com/taiyang-li)). +* Preallocate memory used by asynchronous inserts to improve performance. [#74945](https://github.com/ClickHouse/ClickHouse/pull/74945) ([Ilya Golshtein](https://github.com/ilejn)). +* Decrease the amount of Keeper requests by eliminating the use of single `get` requests, which could have caused a significant load on Keeper with the increased number of replicas, in places where `multiRead` is available. [#56862](https://github.com/ClickHouse/ClickHouse/pull/56862) ([Nikolay Degterinsky](https://github.com/evillique)). +* A marginal optimization for running functions on Nullable arguments. [#76489](https://github.com/ClickHouse/ClickHouse/pull/76489) ([李扬](https://github.com/taiyang-li)). +* Optimize `arraySort`. [#76850](https://github.com/ClickHouse/ClickHouse/pull/76850) ([李扬](https://github.com/taiyang-li)). +* Merge marks of the same part and write them to the query condition cache at one time to reduce the consumption of locks. [#77377](https://github.com/ClickHouse/ClickHouse/pull/77377) ([zhongyuankai](https://github.com/zhongyuankai)). +* Optimize `s3Cluster` performance for queries with one bracket expansion. [#77686](https://github.com/ClickHouse/ClickHouse/pull/77686) ([Tomáš Hromada](https://github.com/gyfis)). +* Optimize order by single Nullable or LowCardinality columns. [#77789](https://github.com/ClickHouse/ClickHouse/pull/77789) ([李扬](https://github.com/taiyang-li)). +* Optimize memory usage of the `Native` format. [#78442](https://github.com/ClickHouse/ClickHouse/pull/78442) ([Azat Khuzhin](https://github.com/azat)). +* Trivial optimization: do not rewrite `count(if(...))` to `countIf` if a type cast is required. Close [#78564](https://github.com/ClickHouse/ClickHouse/issues/78564). [#78565](https://github.com/ClickHouse/ClickHouse/pull/78565) ([李扬](https://github.com/taiyang-li)). +* The `hasAll` function can now take advantage of the `tokenbf_v1`, `ngrambf_v1` full-text skipping indices. [#77662](https://github.com/ClickHouse/ClickHouse/pull/77662) ([UnamedRus](https://github.com/UnamedRus)). +* Vector similarity index could over-allocate main memory by up to 2x. This fix reworks the memory allocation strategy, reducing the memory consumption and improving the effectiveness of the vector similarity index cache. (issue [#78056](https://github.com/ClickHouse/ClickHouse/issues/78056)). [#78394](https://github.com/ClickHouse/ClickHouse/pull/78394) ([Shankar Iyer](https://github.com/shankar-iyer)). +* Introduce a setting `schema_type` for `system.metric_log` table with schema type. There are three allowed schemas: `wide` -- current schema, each metric/event in a separate column (most effective for reads of separate columns), `transposed` -- similar to `system.asynchronous_metric_log`, metrics/events are stored as rows, and the most interesting `transposed_with_wide_view` -- create underlying table with `transposed` schema, but also introduce a view with `wide` schema which translates queries to underlying table. In `transposed_with_wide_view` subsecond resolution for view is not supported, `event_time_microseconds` is just an alias for backward compatibility. [#78412](https://github.com/ClickHouse/ClickHouse/pull/78412) ([alesapin](https://github.com/alesapin)). + +#### Improvement +* Serialize query plan for `Distributed` queries. A new setting `serialize_query_plan` is added. When enabled, queries from `Distributed` table will use a serialized query plan for remote query execution. This introduces a new packet type to TCP protocol, `true` should be added to the server config to allow processing this packet. [#69652](https://github.com/ClickHouse/ClickHouse/pull/69652) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Support `JSON` type and subcolumns reading from views. [#76903](https://github.com/ClickHouse/ClickHouse/pull/76903) ([Pavel Kruglov](https://github.com/Avogar)). +* Support ALTER DATABASE ... ON CLUSTER. [#79242](https://github.com/ClickHouse/ClickHouse/pull/79242) ([Tuan Pham Anh](https://github.com/tuanpach)). +* Refreshes of refreshable materialized views now appear in `system.query_log`. [#71333](https://github.com/ClickHouse/ClickHouse/pull/71333) ([Michael Kolupaev](https://github.com/al13n321)). +* User-defined functions (UDFs) can now be marked as deterministic via a new setting in their configuration. Also, the query cache now checks if UDFs called within a query are deterministic. If this is the case, it caches the query result. (Issue [#59988](https://github.com/ClickHouse/ClickHouse/issues/59988)). [#77769](https://github.com/ClickHouse/ClickHouse/pull/77769) ([Jimmy Aguilar Mena](https://github.com/Ergus)). +* Enabled a backoff logic for all types of replicated tasks. It will provide the ability to reduce CPU usage, memory usage, and log file sizes. Added new settings `max_postpone_time_for_failed_replicated_fetches_ms`, `max_postpone_time_for_failed_replicated_merges_ms` and `max_postpone_time_for_failed_replicated_tasks_ms` which are similar to `max_postpone_time_for_failed_mutations_ms`. [#74576](https://github.com/ClickHouse/ClickHouse/pull/74576) ([MikhailBurdukov](https://github.com/MikhailBurdukov)). +* Add `query_id` to `system.errors`. Closes [#75815](https://github.com/ClickHouse/ClickHouse/issues/75815). [#76581](https://github.com/ClickHouse/ClickHouse/pull/76581) ([Vladimir Baikov](https://github.com/bkvvldmr)). +* Adding support for converting `UInt128` to `IPv6`. This allows the `bitAnd` operation and arithmatics for `IPv6` and conversion back to `IPv6`. Closes [#76752](https://github.com/ClickHouse/ClickHouse/issues/76752). This allows the result from `bitAnd` operation on `IPv6` to be converted back to `IPv6`, as well. See also [#57707](https://github.com/ClickHouse/ClickHouse/pull/57707). [#76928](https://github.com/ClickHouse/ClickHouse/pull/76928) ([Muzammil Abdul Rehman](https://github.com/muzammilar)). +* Don't parse special `Bool` values in text formats inside `Variant` type by default. It can be enabled using setting `allow_special_bool_values_inside_variant`. [#76974](https://github.com/ClickHouse/ClickHouse/pull/76974) ([Pavel Kruglov](https://github.com/Avogar)). +* Support configurable per task waiting time of low `priority` query in session level and in server level. [#77013](https://github.com/ClickHouse/ClickHouse/pull/77013) ([VicoWu](https://github.com/VicoWu)). +* Implement comparison for values of JSON data type. Now JSON objects can be compared similarly to Maps. [#77397](https://github.com/ClickHouse/ClickHouse/pull/77397) ([Pavel Kruglov](https://github.com/Avogar)). +* Better permission support by `system.kafka_consumers`. Forward internal `librdkafka` errors (worth noting that this library is a crap). [#77700](https://github.com/ClickHouse/ClickHouse/pull/77700) ([Ilya Golshtein](https://github.com/ilejn)). +* Added validation for the settings of the Buffer table engine. [#77840](https://github.com/ClickHouse/ClickHouse/pull/77840) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* Add config `enable_hdfs_pread` to enable or disable pread in `HDFS`. [#77885](https://github.com/ClickHouse/ClickHouse/pull/77885) ([kevinyhzou](https://github.com/KevinyhZou)). +* Add profile events for number of zookeeper `multi` read and write requests. [#77888](https://github.com/ClickHouse/ClickHouse/pull/77888) ([JackyWoo](https://github.com/JackyWoo)). +* Allow creating and inserting into temporary tables when `disable_insertion_and_mutation` is on. [#77901](https://github.com/ClickHouse/ClickHouse/pull/77901) ([Xu Jia](https://github.com/XuJia0210)). +* Decrease `max_insert_delayed_streams_for_parallel_write` (to 100). [#77919](https://github.com/ClickHouse/ClickHouse/pull/77919) ([Azat Khuzhin](https://github.com/azat)). +* Fix year parsing in Joda syntax (this is from the Java world if you're wondering) like `yyy`. [#77973](https://github.com/ClickHouse/ClickHouse/pull/77973) ([李扬](https://github.com/taiyang-li)). +* Attaching parts of `MergeTree` tables will be performed in their block order, which is important for special merging algorithms, such as `ReplacingMergeTree`. This closes [#71009](https://github.com/ClickHouse/ClickHouse/issues/71009). [#77976](https://github.com/ClickHouse/ClickHouse/pull/77976) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Query masking rules are now able to throw a `LOGICAL_ERROR` in case if the match happened. This will help to check if pre-defined password is leaking anywhere in logs. [#78094](https://github.com/ClickHouse/ClickHouse/pull/78094) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Added column `index_length_column` to `information_schema.tables` for better compatibility with MySQL. [#78119](https://github.com/ClickHouse/ClickHouse/pull/78119) ([Paweł Zakrzewski](https://github.com/KrzaQ)). +* Introduce two new metrics: `TotalMergeFailures` and `NonAbortedMergeFailures`. These metrics are needed to detect the cases where too many merges fail within a short period. [#78150](https://github.com/ClickHouse/ClickHouse/pull/78150) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). +* Fix incorrect S3 URL parsing when key is not specified on path style. [#78185](https://github.com/ClickHouse/ClickHouse/pull/78185) ([Arthur Passos](https://github.com/arthurpassos)). +* Fix incorrect values of `BlockActiveTime`, `BlockDiscardTime`, `BlockWriteTime`, `BlockQueueTime`, and `BlockReadTime` asynchronous metrics (before the change 1 second was incorrectly reported as 0.001). [#78211](https://github.com/ClickHouse/ClickHouse/pull/78211) ([filimonov](https://github.com/filimonov)). +* Respect `loading_retries` limit for errors during push to materialized view for StorageS3(Azure)Queue. Before that such errors were retried indefinitely. [#78313](https://github.com/ClickHouse/ClickHouse/pull/78313) ([Kseniia Sumarokova](https://github.com/kssenii)). +* In DeltaLake with `delta-kernel-rs` implementation, fix performance and progress bar. [#78368](https://github.com/ClickHouse/ClickHouse/pull/78368) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Support `include`, `from_env`, `from_zk` for runtime disks. Closes [#78177](https://github.com/ClickHouse/ClickHouse/issues/78177). [#78470](https://github.com/ClickHouse/ClickHouse/pull/78470) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Add a dynamic warning to the `system.warnings` table for long running mutations. [#78658](https://github.com/ClickHouse/ClickHouse/pull/78658) ([Bharat Nallan](https://github.com/bharatnc)). +* Added field `condition` to system table `system.query_condition_cache`. It stores the plaintext condition whose hash is used as a key in the query condition cache. [#78671](https://github.com/ClickHouse/ClickHouse/pull/78671) ([Robert Schulze](https://github.com/rschu1ze)). +* Allow an empty value for Hive partitioning. [#78816](https://github.com/ClickHouse/ClickHouse/pull/78816) ([Arthur Passos](https://github.com/arthurpassos)). +* Fix `IN` clause type coercion for `BFloat16` (i.e. `SELECT toBFloat16(1) IN [1, 2, 3];` now returns `1`). Closes [#78754](https://github.com/ClickHouse/ClickHouse/issues/78754). [#78839](https://github.com/ClickHouse/ClickHouse/pull/78839) ([Raufs Dunamalijevs](https://github.com/rienath)). +* Do not check parts on other disks for `MergeTree` if `disk = ...` is set. [#78855](https://github.com/ClickHouse/ClickHouse/pull/78855) ([Azat Khuzhin](https://github.com/azat)). +* Make data types in `used_data_type_families` in `system.query_log` to be recorded with canonical names. [#78972](https://github.com/ClickHouse/ClickHouse/pull/78972) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Cleanup settings during `recoverLostReplica` same as it was done in: [#78637](https://github.com/ClickHouse/ClickHouse/pull/78637). [#79113](https://github.com/ClickHouse/ClickHouse/pull/79113) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Use insertion columns for INFILE schema inference. [#78490](https://github.com/ClickHouse/ClickHouse/pull/78490) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). + +#### Bug Fix (user-visible misbehavior in an official stable release) +* Fix incorrect projection analysis when `count(Nullable)` is used in aggregate projections. This fixes [#74495](https://github.com/ClickHouse/ClickHouse/issues/74495) . This PR also adds some logs around projection analysis to clarify why a projection is used or why not. [#74498](https://github.com/ClickHouse/ClickHouse/pull/74498) ([Amos Bird](https://github.com/amosbird)). +* Fix `Part <...> does not contain in snapshot of previous virtual parts. (PART_IS_TEMPORARILY_LOCKED)` during `DETACH PART`. [#76039](https://github.com/ClickHouse/ClickHouse/pull/76039) ([Aleksei Filatov](https://github.com/aalexfvk)). +* Fix not working skip indexes with expression with literals in analyzer and remove trivial casts during indexes analysis. [#77229](https://github.com/ClickHouse/ClickHouse/pull/77229) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix a bug when `close_session` query parameter didn't have any effect leading to named sessions being closed only after `session_timeout`. [#77336](https://github.com/ClickHouse/ClickHouse/pull/77336) ([Alexey Katsman](https://github.com/alexkats)). +* Fixed receiving messages from NATS server without attached Materialized Views. [#77392](https://github.com/ClickHouse/ClickHouse/pull/77392) ([Dmitry Novikov](https://github.com/dmitry-sles-novikov)). +* Fix logical error while reading from empty `FileLog` via `merge` table function, close [#75575](https://github.com/ClickHouse/ClickHouse/issues/75575). [#77441](https://github.com/ClickHouse/ClickHouse/pull/77441) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Use default format settings in `Dynamic` serialization from shared variant. [#77572](https://github.com/ClickHouse/ClickHouse/pull/77572) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix checking if the table data path exists on the local disk. [#77608](https://github.com/ClickHouse/ClickHouse/pull/77608) ([Tuan Pham Anh](https://github.com/tuanpach)). +* Fix sending constant values to remote for some types. [#77634](https://github.com/ClickHouse/ClickHouse/pull/77634) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix a crash because of expired context in S3/AzureQueue. [#77720](https://github.com/ClickHouse/ClickHouse/pull/77720) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Hide credentials in RabbitMQ, Nats, Redis, AzureQueue table engines. [#77755](https://github.com/ClickHouse/ClickHouse/pull/77755) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix undefined behaviour on `NaN` comparison in `argMin`/`argMax`. [#77756](https://github.com/ClickHouse/ClickHouse/pull/77756) ([Raúl Marín](https://github.com/Algunenano)). +* Regularly check if merges and mutations were cancelled even in case when the operation doesn't produce any blocks to write. [#77766](https://github.com/ClickHouse/ClickHouse/pull/77766) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). +* Fixed refreshable materialized view in Replicated database not working on newly added replicas. [#77774](https://github.com/ClickHouse/ClickHouse/pull/77774) ([Michael Kolupaev](https://github.com/al13n321)). +* Fix possible crash when `NOT_FOUND_COLUMN_IN_BLOCK` error occurs. [#77854](https://github.com/ClickHouse/ClickHouse/pull/77854) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Fix crash that happens in the S3/AzureQueue while filling data. [#77878](https://github.com/ClickHouse/ClickHouse/pull/77878) ([Bharat Nallan](https://github.com/bharatnc)). +* Disable fuzzy search for history in SSH server (since it requires the skim library). [#78002](https://github.com/ClickHouse/ClickHouse/pull/78002) ([Azat Khuzhin](https://github.com/azat)). +* Fixes a bug that a vector search query on a non-indexed column was returning incorrect results if there was another vector column in the table with a defined vector similarity index. (Issue [#77978](https://github.com/ClickHouse/ClickHouse/issues/77978)). [#78069](https://github.com/ClickHouse/ClickHouse/pull/78069) ([Shankar Iyer](https://github.com/shankar-iyer)). +* Fix a minuscule error "The requested output format {} is binary... Do you want to output it anyway? [y/N]" prompt. [#78095](https://github.com/ClickHouse/ClickHouse/pull/78095) ([Azat Khuzhin](https://github.com/azat)). +* Fix of a bug in case of `toStartOfInterval` with zero origin argument. [#78096](https://github.com/ClickHouse/ClickHouse/pull/78096) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Disallow specifying an empty `session_id` query parameter for HTTP interface. [#78098](https://github.com/ClickHouse/ClickHouse/pull/78098) ([Alexey Katsman](https://github.com/alexkats)). +* Fix metadata override in `Replicated` database which could have happened due to a `RENAME` query executed right after an `ALTER` query. [#78107](https://github.com/ClickHouse/ClickHouse/pull/78107) ([Nikolay Degterinsky](https://github.com/evillique)). +* Fix crash in `NATS` engine. [#78108](https://github.com/ClickHouse/ClickHouse/pull/78108) ([Dmitry Novikov](https://github.com/dmitry-sles-novikov)). +* Do not try to create history_file in embedded client for SSH (in previous versions the creation was always unsuccessful, but attempted). [#78112](https://github.com/ClickHouse/ClickHouse/pull/78112) ([Azat Khuzhin](https://github.com/azat)). +* Fix `system.detached_tables` displaying incorrect information after `RENAME DATABASE` or `DROP TABLE` queries. [#78126](https://github.com/ClickHouse/ClickHouse/pull/78126) ([Nikolay Degterinsky](https://github.com/evillique)). +* Fix for checks for too many tables with `Replicated` database after [#77274](https://github.com/ClickHouse/ClickHouse/pull/77274). Also, perform the check before creating the storage to avoid creating unaccounted nodes in Keeper in the case of `ReplicatedMergeTree` or `KeeperMap`. [#78127](https://github.com/ClickHouse/ClickHouse/pull/78127) ([Nikolay Degterinsky](https://github.com/evillique)). +* Fix possible crash due to concurrent `S3Queue` metadata initialization. [#78131](https://github.com/ClickHouse/ClickHouse/pull/78131) ([Azat Khuzhin](https://github.com/azat)). +* `groupArray*` functions now produce `BAD_ARGUMENTS` error for Int-typed 0 value of the `max_size` argument, like it's already done for UInt one, instead of trying to execute with it. [#78140](https://github.com/ClickHouse/ClickHouse/pull/78140) ([Eduard Karacharov](https://github.com/korowa)). +* Prevent crash on recovering a lost replica if the local table is removed before it's detached. [#78173](https://github.com/ClickHouse/ClickHouse/pull/78173) ([Raúl Marín](https://github.com/Algunenano)). +* Fix the fact that "alterable" column in `system.s3_queue_settings` returning always `false`. [#78187](https://github.com/ClickHouse/ClickHouse/pull/78187) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Mask Azure access signature to be not visible to user or in logs. [#78189](https://github.com/ClickHouse/ClickHouse/pull/78189) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix prefetching of substreams with prefixes in Wide parts. [#78205](https://github.com/ClickHouse/ClickHouse/pull/78205) ([Pavel Kruglov](https://github.com/Avogar)). +* Fixed crashes / incorrect result for `mapFromArrays` in case of `LowCardinality(Nullable)` type of keys array. [#78240](https://github.com/ClickHouse/ClickHouse/pull/78240) ([Eduard Karacharov](https://github.com/korowa)). +* Fix delta-kernel-rs auth options. [#78255](https://github.com/ClickHouse/ClickHouse/pull/78255) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Not schedule Refreshable Materialized Views task if a replica's `disable_insertion_and_mutation` is true. A task is some insertion, it will failed if `disable_insertion_and_mutation` is true. [#78277](https://github.com/ClickHouse/ClickHouse/pull/78277) ([Xu Jia](https://github.com/XuJia0210)). +* Validate access to underlying tables for the `Merge` engine. [#78339](https://github.com/ClickHouse/ClickHouse/pull/78339) ([Pervakov Grigorii](https://github.com/GrigoryPervakov)). +* `FINAL` modifier can be ignored when querying a `Distributed` table. [#78428](https://github.com/ClickHouse/ClickHouse/pull/78428) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* `bitmapMin` returns the uint32_max when the bitmap is empty (and uint64_max when the input type is larger), which matches the behavior of empty roaring_bitmap's minimum. [#78444](https://github.com/ClickHouse/ClickHouse/pull/78444) ([wxybear](https://github.com/wxybear)). +* Disable parallelization of query processing right after reading FROM when `distributed_aggregation_memory_efficient` enabled, it may lead to logical error. Closes [#76934](https://github.com/ClickHouse/ClickHouse/issues/76934). [#78500](https://github.com/ClickHouse/ClickHouse/pull/78500) ([flynn](https://github.com/ucasfl)). +* Set at least one stream for reading in case there are zero planned streams after applying `max_streams_to_max_threads_ratio` setting. [#78505](https://github.com/ClickHouse/ClickHouse/pull/78505) ([Eduard Karacharov](https://github.com/korowa)). +* In storage `S3Queue` fix logical error "Cannot unregister: table uuid is not registered". Closes [#78285](https://github.com/ClickHouse/ClickHouse/issues/78285). [#78541](https://github.com/ClickHouse/ClickHouse/pull/78541) ([Kseniia Sumarokova](https://github.com/kssenii)). +* ClickHouse is now able to figure out its cgroup v2 on systems with both cgroups v1 and v2 enabled. [#78566](https://github.com/ClickHouse/ClickHouse/pull/78566) ([Grigory Korolev](https://github.com/gkorolev)). +* `-Cluster` table functions were failing when used with table-level settings. [#78587](https://github.com/ClickHouse/ClickHouse/pull/78587) ([Daniil Ivanik](https://github.com/divanik)). +* Better checks when transactions are not supported by ReplicatedMergeTree on INSERT. [#78633](https://github.com/ClickHouse/ClickHouse/pull/78633) ([Azat Khuzhin](https://github.com/azat)). +* Cleanup query settings during attach. [#78637](https://github.com/ClickHouse/ClickHouse/pull/78637) ([Raúl Marín](https://github.com/Algunenano)). +* Fix a crash when an invalid path was specified in `iceberg_metadata_file_path`. [#78688](https://github.com/ClickHouse/ClickHouse/pull/78688) ([alesapin](https://github.com/alesapin)). +* In `DeltaLake` table engine with delta-kernel-s implementation, fix the case when the read schema is different from the table schema and there are partition columns at the same time leading to a "not found column" error. [#78690](https://github.com/ClickHouse/ClickHouse/pull/78690) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix a problem when after scheduling to close a named session (but before timeout expiration), creation of a new named session with the same name led to it being closed at a time point when the first session was scheduled to close. [#78698](https://github.com/ClickHouse/ClickHouse/pull/78698) ([Alexey Katsman](https://github.com/alexkats)). +* Fixed several types of `SELECT` queries that read from tables with `MongoDB` engine or `mongodb` table function: queries with implicit conversion of const value in `WHERE` clause (e.g. `WHERE datetime = '2025-03-10 00:00:00'`) ; queries with `LIMIT` and `GROUP BY`. Previously, they could return the wrong result. [#78777](https://github.com/ClickHouse/ClickHouse/pull/78777) ([Anton Popov](https://github.com/CurtizJ)). +* Don't block table shutdown while running `CHECK TABLE`. [#78782](https://github.com/ClickHouse/ClickHouse/pull/78782) ([Raúl Marín](https://github.com/Algunenano)). +* Keeper fix: fix ephemeral count in all cases. [#78799](https://github.com/ClickHouse/ClickHouse/pull/78799) ([Antonio Andelic](https://github.com/antonio2368)). +* Fix bad cast in `StorageDistributed` when using table functions other than `view`. Closes [#78464](https://github.com/ClickHouse/ClickHouse/issues/78464). [#78828](https://github.com/ClickHouse/ClickHouse/pull/78828) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Fix the consistency of formatting for `tupleElement(*, 1)`. Closes [#78639](https://github.com/ClickHouse/ClickHouse/issues/78639). [#78832](https://github.com/ClickHouse/ClickHouse/pull/78832) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Dictionaries of type `ssd_cache` now reject zero or negative `block_size` and `write_buffer_size` parameters (issue [#78314](https://github.com/ClickHouse/ClickHouse/issues/78314)). [#78854](https://github.com/ClickHouse/ClickHouse/pull/78854) ([Elmi Ahmadov](https://github.com/ahmadov)). +* Fix crash in Refreshable MATERIALIZED VIEW inthe case of ALTER after an incorrect shutdown. [#78858](https://github.com/ClickHouse/ClickHouse/pull/78858) ([Azat Khuzhin](https://github.com/azat)). +* Fix parsing of bad `DateTime` values in `CSV` format. [#78919](https://github.com/ClickHouse/ClickHouse/pull/78919) ([Pavel Kruglov](https://github.com/Avogar)). +* Keeper fix: Avoid triggering watches on failed multi requests. [#79247](https://github.com/ClickHouse/ClickHouse/pull/79247) ([Antonio Andelic](https://github.com/antonio2368)). +* Fix reading Iceberg table failed when min-max value is specified explicitly but is `NULL`. The Go Iceberg library was noted for generating such an atrocious files. Closes [#78740](https://github.com/ClickHouse/ClickHouse/issues/78740). [#78764](https://github.com/ClickHouse/ClickHouse/pull/78764) ([flynn](https://github.com/ucasfl)). + +#### Build/Testing/Packaging Improvement +* Respect CPU target features in Rust and enable LTO in all crates. [#78590](https://github.com/ClickHouse/ClickHouse/pull/78590) ([Raúl Marín](https://github.com/Algunenano)). + + +### ClickHouse release 25.3 LTS, 2025-03-20 {#253} + +#### Backward Incompatible Change +* Disallow truncating replicated databases. [#76651](https://github.com/ClickHouse/ClickHouse/pull/76651) ([Bharat Nallan](https://github.com/bharatnc)). +* Skipping index cache is reverted. [#77447](https://github.com/ClickHouse/ClickHouse/pull/77447) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). + +#### New Feature +* `JSON` data type is production-ready. See https://jsonbench.com/. `Dynamic` and `Variant` data types are production-ready. [#77785](https://github.com/ClickHouse/ClickHouse/pull/77785) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Introduce the SSH protocol for clickhouse-server. Now, it is possible to connect to ClickHouse using any SSH client. This closes: [#74340](https://github.com/ClickHouse/ClickHouse/issues/74340). [#74989](https://github.com/ClickHouse/ClickHouse/pull/74989) ([George Gamezardashvili](https://github.com/Infjoker)). +* Replace table functions with their -Cluster alternatives if parallel replicas are enabled. Fixes [#65024](https://github.com/ClickHouse/ClickHouse/issues/65024). [#70659](https://github.com/ClickHouse/ClickHouse/pull/70659) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* A new implementation of the Userspace Page Cache, which allows caching data in the in-process memory instead of relying on the OS page cache, which is useful when the data is stored on a remote virtual filesystem without backing with the local filesystem cache. [#70509](https://github.com/ClickHouse/ClickHouse/pull/70509) ([Michael Kolupaev](https://github.com/al13n321)). +* Added `concurrent_threads_scheduler` server setting that governs how CPU slots are distributed among concurrent queries. Could be set to `round_robin` (previous behavior) or `fair_round_robin` to address the issue of unfair CPU distribution between INSERTs and SELECTs. [#75949](https://github.com/ClickHouse/ClickHouse/pull/75949) ([Sergei Trifonov](https://github.com/serxa)). +* Add `estimateCompressionRatio` aggregate function [#70801](https://github.com/ClickHouse/ClickHouse/issues/70801). [#76661](https://github.com/ClickHouse/ClickHouse/pull/76661) ([Tariq Almawash](https://github.com/talmawash)). +* Added function `arraySymmetricDifference`. It returns all elements from multiple array arguments which do not occur in all arguments. Example: `SELECT arraySymmetricDifference([1, 2], [2, 3])` returns `[1, 3]`. (issue [#61673](https://github.com/ClickHouse/ClickHouse/issues/61673)). [#76231](https://github.com/ClickHouse/ClickHouse/pull/76231) ([Filipp Abapolov](https://github.com/pheepa)). +* Allow to explicitly specify metadata file to read for Iceberg with storage/table function setting `iceberg_metadata_file_path `. Fixes [#47412](https://github.com/ClickHouse/ClickHouse/issues/47412). [#77318](https://github.com/ClickHouse/ClickHouse/pull/77318) ([alesapin](https://github.com/alesapin)). +* Added the `keccak256` hash function, commonly used in blockchain implementations, especially in EVM-based systems. [#76669](https://github.com/ClickHouse/ClickHouse/pull/76669) ([Arnaud Briche](https://github.com/arnaudbriche)). +* Add three new functions. `icebergTruncate` according to specification. https://iceberg.apache.org/spec/#truncate-transform-details, `toYearNumSinceEpoch` and `toMonthNumSinceEpoch`. Support `truncate` transform in partition pruning for `Iceberg` engine. [#77403](https://github.com/ClickHouse/ClickHouse/pull/77403) ([alesapin](https://github.com/alesapin)). +* Support `LowCardinality(Decimal)` data types [#72256](https://github.com/ClickHouse/ClickHouse/issues/72256). [#72833](https://github.com/ClickHouse/ClickHouse/pull/72833) ([zhanglistar](https://github.com/zhanglistar)). +* `FilterTransformPassedRows` and `FilterTransformPassedBytes` profile events will show the number of rows and number of bytes filtered during the query execution. [#76662](https://github.com/ClickHouse/ClickHouse/pull/76662) ([Onkar Deshpande](https://github.com/onkar)). +* Support for the histogram metric type. The interface closely mirrors the Prometheus client, where you simply call `observe(value)` to increment the counter in the bucket corresponding to the value. The histogram metrics are exposed via `system.histogram_metrics`. [#75736](https://github.com/ClickHouse/ClickHouse/pull/75736) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). +* Non-constant CASE support for switching on explicit values. [#77399](https://github.com/ClickHouse/ClickHouse/pull/77399) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). + +#### Experimental Feature +* Add support [for Unity Catalog](https://www.databricks.com/product/unity-catalog) for DeltaLake tables on top of AWS S3 and local filesystem. [#76988](https://github.com/ClickHouse/ClickHouse/pull/76988) ([alesapin](https://github.com/alesapin)). +* Introduce experimental integration with AWS Glue service catalog for Iceberg tables. [#77257](https://github.com/ClickHouse/ClickHouse/pull/77257) ([alesapin](https://github.com/alesapin)). +* Added support for dynamic cluster autodiscovery. This extends the existing _node_ autodiscovery feature. ClickHouse can now automatically detect and register new _clusters_ under a common ZooKeeper path using ``. [#76001](https://github.com/ClickHouse/ClickHouse/pull/76001) ([Anton Ivashkin](https://github.com/ianton-ru)). +* Allow automatic cleanup merges of entire partitions after a configurable timeout with a new setting `enable_replacing_merge_with_cleanup_for_min_age_to_force_merge`. [#76440](https://github.com/ClickHouse/ClickHouse/pull/76440) ([Christoph Wurm](https://github.com/cwurm)). + +#### Performance Improvement +* Implement query condition cache to improve query performance using repeated conditions. The range of the portion of data that does not meet the condition is remembered as a temporary index in memory. Subsequent queries will use this index. Close [#67768](https://github.com/ClickHouse/ClickHouse/issues/67768) [#69236](https://github.com/ClickHouse/ClickHouse/pull/69236) ([zhongyuankai](https://github.com/zhongyuankai)). +* Actively evict data from the cache on parts removal. Do not let the cache grow to the maximum size if the amount of data is less. [#76641](https://github.com/ClickHouse/ClickHouse/pull/76641) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Replace Int256 and UInt256 with clang builtin i256 in arithmetic calculation, and it gives a performance improvement [#70502](https://github.com/ClickHouse/ClickHouse/issues/70502). [#73658](https://github.com/ClickHouse/ClickHouse/pull/73658) ([李扬](https://github.com/taiyang-li)). +* In some cases (e.g. empty array column) data parts can contain empty files. We can skip writing empty blobs to ObjectStorage and only store metadata for such files when the table resides on disk with separated metadata and object storages. [#75860](https://github.com/ClickHouse/ClickHouse/pull/75860) ([Alexander Gololobov](https://github.com/davenger)). +* Improve min/max performance for Decimal32/Decimal64/DateTime64. [#76570](https://github.com/ClickHouse/ClickHouse/pull/76570) ([李扬](https://github.com/taiyang-li)). +* Query compilation (setting `compile_expressions`) now considers the machine type. This speeds up such queries significantly. [#76753](https://github.com/ClickHouse/ClickHouse/pull/76753) ([ZhangLiStar](https://github.com/zhanglistar)). +* Optimize `arraySort`. [#76850](https://github.com/ClickHouse/ClickHouse/pull/76850) ([李扬](https://github.com/taiyang-li)). +* Disable `filesystem_cache_prefer_bigger_buffer_size` when the cache is used passively, such as for merges. [#77898](https://github.com/ClickHouse/ClickHouse/pull/77898) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Apply `preserve_most` attribute at some places in code, which allows slightly better code generation. [#67778](https://github.com/ClickHouse/ClickHouse/pull/67778) ([Nikita Taranov](https://github.com/nickitat)). +* Faster ClickHouse servers shutdown (get rid of 2.5sec delay). [#76550](https://github.com/ClickHouse/ClickHouse/pull/76550) ([Azat Khuzhin](https://github.com/azat)). +* Avoid excess allocation in ReadBufferFromS3 and other remote reading buffers, reduce their memory consumption in half. [#76692](https://github.com/ClickHouse/ClickHouse/pull/76692) ([Sema Checherinda](https://github.com/CheSema)). +* Update zstd from 1.5.5 to 1.5.7 which could lead to some [performance improvements](https://github.com/facebook/zstd/releases/tag/v1.5.7). [#77137](https://github.com/ClickHouse/ClickHouse/pull/77137) ([Pradeep Chhetri](https://github.com/chhetripradeep)). +* Reduce memory usage during prefetches of JSON column in Wide parts. This is relevant when ClickHouse is used on top of a shared storage, such as in ClickHouse Cloud. [#77640](https://github.com/ClickHouse/ClickHouse/pull/77640) ([Pavel Kruglov](https://github.com/Avogar)). + +#### Improvement +* Support atomic rename when `TRUNCATE` is used with `INTO OUTFILE`. Resolves [#70323](https://github.com/ClickHouse/ClickHouse/issues/70323). [#77181](https://github.com/ClickHouse/ClickHouse/pull/77181) ([Onkar Deshpande](https://github.com/onkar)). +* It's no longer possible to use `NaN` or `inf` for float values as settings. Not like it did make any sense before. [#77546](https://github.com/ClickHouse/ClickHouse/pull/77546) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Disable parallel replicas by default when analyzer is disabled regardless `compatibility` setting. It's still possible to change this behavior by explicitly setting `parallel_replicas_only_with_analyzer` to `false`. [#77115](https://github.com/ClickHouse/ClickHouse/pull/77115) ([Igor Nikonov](https://github.com/devcrafter)). +* Add the ability to define a list of headers that are forwarded from the headers of the client request to the external HTTP authenticator. [#77054](https://github.com/ClickHouse/ClickHouse/pull/77054) ([inv2004](https://github.com/inv2004)). +* Respect column insensitive column matching for fields in tuple columns. Close https://github.com/apache/incubator-gluten/issues/8324. [#73780](https://github.com/ClickHouse/ClickHouse/pull/73780) ([李扬](https://github.com/taiyang-li)). +* Parameters for the codec Gorilla will now always be saved in the table metadata in .sql file. This closes: [#70072](https://github.com/ClickHouse/ClickHouse/issues/70072). [#74814](https://github.com/ClickHouse/ClickHouse/pull/74814) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Implemented parsing enhancements for certain data lakes (Sequence ID parsing: Added functionality to parse sequence identifiers in manifest files AND Avro metadata parsing: Redesigned the Avro metadata parser to be easily extendable for future enhancements). [#75010](https://github.com/ClickHouse/ClickHouse/pull/75010) ([Daniil Ivanik](https://github.com/divanik)). +* Remove trace_id from default ORDER BY for `system.opentelemetry_span_log`. [#75907](https://github.com/ClickHouse/ClickHouse/pull/75907) ([Azat Khuzhin](https://github.com/azat)). +* Encryption (the attribute `encrypted_by`) can now be applied to any configuration file (config.xml, users.xml, nested configuration files). Previously, it worked only for the top-level config.xml file. [#75911](https://github.com/ClickHouse/ClickHouse/pull/75911) ([Mikhail Gorshkov](https://github.com/mgorshkov)). +* Improve the `system.warnings` table and add some dynamic warning messages that can be added, updated or removed. [#76029](https://github.com/ClickHouse/ClickHouse/pull/76029) ([Bharat Nallan](https://github.com/bharatnc)). +* This PR makes it impossible to run a query `ALTER USER user1 ADD PROFILES a, DROP ALL PROFILES` because all `DROP` operations should come first in the order. [#76242](https://github.com/ClickHouse/ClickHouse/pull/76242) ([pufit](https://github.com/pufit)). +* Various enhancements for SYNC REPLICA (better error messages, better tests, sanity checks). [#76307](https://github.com/ClickHouse/ClickHouse/pull/76307) ([Azat Khuzhin](https://github.com/azat)). +* Use correct fallback when multipart copy to S3 fails during backup with Access Denied. Multi part copy can generate Access Denied error when backup is done between buckets that have different credentials. [#76515](https://github.com/ClickHouse/ClickHouse/pull/76515) ([Antonio Andelic](https://github.com/antonio2368)). +* Upgraded librdkafka (which is a pile of crap) to version 2.8.0 (the pile does not get any better) and improved the shutdown sequence for Kafka tables, reducing delays during table drops and server restarts. The `engine=Kafka` no longer explicitly leaves the consumer group when a table is dropped. Instead, the consumer remains in the group until it is automatically removed after `session_timeout_ms` (default: 45 seconds) of inactivity. [#76621](https://github.com/ClickHouse/ClickHouse/pull/76621) ([filimonov](https://github.com/filimonov)). +* Fix validation of S3 request settings. [#76658](https://github.com/ClickHouse/ClickHouse/pull/76658) ([Vitaly Baranov](https://github.com/vitlibar)). +* System tables like `server_settings` or `settings` have a `default` value column which is convenient. Add them to `merge_tree_settings` and `replicated_merge_tree_settings`. [#76942](https://github.com/ClickHouse/ClickHouse/pull/76942) ([Diego Nieto](https://github.com/lesandie)). +* Added `ProfileEvents::QueryPreempted`, which has a similar logic to `CurrentMetrics::QueryPreempted`. [#77015](https://github.com/ClickHouse/ClickHouse/pull/77015) ([VicoWu](https://github.com/VicoWu)). +* Previously, a Replicated database could print credentials specified in a query to logs. This behaviour is fixed. This closes: [#77123](https://github.com/ClickHouse/ClickHouse/issues/77123). [#77133](https://github.com/ClickHouse/ClickHouse/pull/77133) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Allow ALTER TABLE DROP PARTITION for `plain_rewritable disk`. [#77138](https://github.com/ClickHouse/ClickHouse/pull/77138) ([Julia Kartseva](https://github.com/jkartseva)). +* Backup/restore setting `allow_s3_native_copy` now supports value three possible values: - `False` - s3 native copy will not be used; - `True` (old default) - ClickHouse will try s3 native copy first, if it fails then fallback to the reading+writing approach; - `'auto'` (new default) - ClickHouse will compare the source and destination credentials first. If they are same, ClickHouse will try s3 native copy and then may fallback to the reading+writing approach. If they are different, ClickHouse will go directly to the reading+writing approach. [#77401](https://github.com/ClickHouse/ClickHouse/pull/77401) ([Vitaly Baranov](https://github.com/vitlibar)). +* Support aws session token and environment credentials usage in delta kernel for DeltaLake table engine. [#77661](https://github.com/ClickHouse/ClickHouse/pull/77661) ([Kseniia Sumarokova](https://github.com/kssenii)). + +#### Bug Fix (user-visible misbehavior in an official stable release) +* Fix stuck while processing pending batch for async distributed INSERT (due to i.e. `No such file or directory`). [#72939](https://github.com/ClickHouse/ClickHouse/pull/72939) ([Azat Khuzhin](https://github.com/azat)). +* Improved datetime conversion during index analysis by enforcing saturating behavior for implicit Date to DateTime conversions. This resolves potential index analysis inaccuracies caused by datetime range limitations. This fixes [#73307](https://github.com/ClickHouse/ClickHouse/issues/73307). It also fixes explicit `toDateTime` conversion when `date_time_overflow_behavior = 'ignore'` which is the default value. [#73326](https://github.com/ClickHouse/ClickHouse/pull/73326) ([Amos Bird](https://github.com/amosbird)). +* Fix all sort of bugs due to race between UUID and table names (for instance it will fix the race between `RENAME` and `RESTART REPLICA`, in case of concurrent `RENAME` with `SYSTEM RESTART REPLICA` you may get end up restarting wrong replica, or/and leaving one of the tables in a `Table X is being restarted` state). [#76308](https://github.com/ClickHouse/ClickHouse/pull/76308) ([Azat Khuzhin](https://github.com/azat)). +* Fix data loss when enable async insert and insert into ... from file ... with unequal block size if the first block size < async_max_size but the second block > async_max_size, the second block will not be inserted. these data is left in `squashing`. [#76343](https://github.com/ClickHouse/ClickHouse/pull/76343) ([Han Fei](https://github.com/hanfei1991)). +* Renamed field 'marks' to 'marks_bytes' in `system.data_skipping_indices`. [#76374](https://github.com/ClickHouse/ClickHouse/pull/76374) ([Robert Schulze](https://github.com/rschu1ze)). +* Fix dynamic filesystem cache resize handling unexpected errors during eviction. [#76466](https://github.com/ClickHouse/ClickHouse/pull/76466) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fixed `used_flag` initialization in parallel hash. It might cause a server crash. [#76580](https://github.com/ClickHouse/ClickHouse/pull/76580) ([Nikita Taranov](https://github.com/nickitat)). +* Fix a logical error when calling `defaultProfiles` function inside a projection. [#76627](https://github.com/ClickHouse/ClickHouse/pull/76627) ([pufit](https://github.com/pufit)). +* Do not request interactive basic auth in the browser in Web UI. Closes [#76319](https://github.com/ClickHouse/ClickHouse/issues/76319). [#76637](https://github.com/ClickHouse/ClickHouse/pull/76637) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Fix THERE_IS_NO_COLUMN exception when selecting boolean literal from distributed tables. [#76656](https://github.com/ClickHouse/ClickHouse/pull/76656) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* The subpath inside the table directory is chosen in a more profound way. [#76681](https://github.com/ClickHouse/ClickHouse/pull/76681) ([Daniil Ivanik](https://github.com/divanik)). +* Fix an error `Not found column in block` after altering a table with a subcolumn in PK. After https://github.com/ClickHouse/ClickHouse/pull/72644, requires https://github.com/ClickHouse/ClickHouse/pull/74403. [#76686](https://github.com/ClickHouse/ClickHouse/pull/76686) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Add performance tests for null shortcircuit and fix bugs. [#76708](https://github.com/ClickHouse/ClickHouse/pull/76708) ([李扬](https://github.com/taiyang-li)). +* Flush output write buffers before finalizing them. Fix `LOGICAL_ERROR` generated during the finalization of some output format, e.g. `JSONEachRowWithProgressRowOutputFormat`. [#76726](https://github.com/ClickHouse/ClickHouse/pull/76726) ([Antonio Andelic](https://github.com/antonio2368)). +* Added support for MongoDB's binary UUID ([#74452](https://github.com/ClickHouse/ClickHouse/issues/74452)) - Fixed WHERE pushdown to MongoDB when using the table function ([#72210](https://github.com/ClickHouse/ClickHouse/issues/72210)) - Changed the MongoDB - ClickHouse type mapping such that MongoDB's binary UUID can only be parsed to ClickHouse's UUID. This should avoid ambiguities and surprises in future. - Fixed OID mapping, preserving backward compatibility. [#76762](https://github.com/ClickHouse/ClickHouse/pull/76762) ([Kirill Nikiforov](https://github.com/allmazz)). +* Fix exception handling in parallel prefixes deserialization of JSON subcolumns. [#76809](https://github.com/ClickHouse/ClickHouse/pull/76809) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix lgamma function behavior for negative integers. [#76840](https://github.com/ClickHouse/ClickHouse/pull/76840) ([Ilya Kataev](https://github.com/IlyaKataev)). +* Fix reverse key analysis for explicitly defined primary keys. Similar to [#76654](https://github.com/ClickHouse/ClickHouse/issues/76654). [#76846](https://github.com/ClickHouse/ClickHouse/pull/76846) ([Amos Bird](https://github.com/amosbird)). +* Fix pretty print of Bool values in JSON format. [#76905](https://github.com/ClickHouse/ClickHouse/pull/76905) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix possible crash because of bad JSON column rollback on error during async inserts. [#76908](https://github.com/ClickHouse/ClickHouse/pull/76908) ([Pavel Kruglov](https://github.com/Avogar)). +* Previously, `multiIf` may return different types of columns during planning and main execution. This resulted in code producing undefined behavior from the C++ perspective. [#76914](https://github.com/ClickHouse/ClickHouse/pull/76914) ([Nikita Taranov](https://github.com/nickitat)). +* Fixed incorrect serialization of constant nullable keys in MergeTree. This fixes [#76939](https://github.com/ClickHouse/ClickHouse/issues/76939). [#76985](https://github.com/ClickHouse/ClickHouse/pull/76985) ([Amos Bird](https://github.com/amosbird)). +* Fix sorting of `BFloat16` values. This closes [#75487](https://github.com/ClickHouse/ClickHouse/issues/75487). This closes [#75669](https://github.com/ClickHouse/ClickHouse/issues/75669). [#77000](https://github.com/ClickHouse/ClickHouse/pull/77000) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Bug fix JSON with Variant subcolumn by adding check to skip ephemeral subcolumns in part consistency check. [#72187](https://github.com/ClickHouse/ClickHouse/issues/72187). [#77034](https://github.com/ClickHouse/ClickHouse/pull/77034) ([Smita Kulkarni](https://github.com/SmitaRKulkarni)). +* Fix crash in template parsing in Values format in case of types mismatch. [#77071](https://github.com/ClickHouse/ClickHouse/pull/77071) ([Pavel Kruglov](https://github.com/Avogar)). +* Don't allow creating EmbeddedRocksDB table with subcolumn in primary key. Previosly such table could be created but select queries were failing. [#77074](https://github.com/ClickHouse/ClickHouse/pull/77074) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix illegal comparison in distributed queries because pushing down predicates to remote doesn't respect literal types. [#77093](https://github.com/ClickHouse/ClickHouse/pull/77093) ([Duc Canh Le](https://github.com/canhld94)). +* Fix crash during Kafka table creation with exception. [#77121](https://github.com/ClickHouse/ClickHouse/pull/77121) ([Pavel Kruglov](https://github.com/Avogar)). +* Support JSON and subcolumns in Kafka and RabbitMQ engines. [#77122](https://github.com/ClickHouse/ClickHouse/pull/77122) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix exceptions stack unwinding on MacOS. [#77126](https://github.com/ClickHouse/ClickHouse/pull/77126) ([Eduard Karacharov](https://github.com/korowa)). +* Fix reading 'null' subcolumn in getSubcolumn function. [#77163](https://github.com/ClickHouse/ClickHouse/pull/77163) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix bloom filter index with Array and not supported functions. [#77271](https://github.com/ClickHouse/ClickHouse/pull/77271) ([Pavel Kruglov](https://github.com/Avogar)). +* We should only check the restriction on the amount of tables during the initial CREATE query. [#77274](https://github.com/ClickHouse/ClickHouse/pull/77274) ([Nikolay Degterinsky](https://github.com/evillique)). +* Not a bug: `SELECT toBFloat16(-0.0) == toBFloat16(0.0)` now correctly returns `true` (from previously `false`). This makes the behavior consistent with `Float32` and `Float64`. [#77290](https://github.com/ClickHouse/ClickHouse/pull/77290) ([Shankar Iyer](https://github.com/shankar-iyer)). +* Fix posbile incorrect reference to unintialized key_index variable, which may lead to crash in debug builds (this uninitialized reference won't cause issues in release builds because subsequent code are likely to throw errors.) ### documentation entry for user-facing changes. [#77305](https://github.com/ClickHouse/ClickHouse/pull/77305) ([wxybear](https://github.com/wxybear)). +* Fix name for partition with a Bool value. It was broken in https://github.com/ClickHouse/ClickHouse/pull/74533. [#77319](https://github.com/ClickHouse/ClickHouse/pull/77319) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix comparison between tuples with nullable elements inside and strings. As an example, before the change comparison between a Tuple `(1, null)` and a String `'(1,null)'` would result in an error. Another example would be a comparison between a Tuple `(1, a)`, where `a` is a Nullable column, and a String `'(1, 2)'`. This change addresses these issues. [#77323](https://github.com/ClickHouse/ClickHouse/pull/77323) ([Alexey Katsman](https://github.com/alexkats)). +* Fix crash in ObjectStorageQueueSource. Was intoduced in https://github.com/ClickHouse/ClickHouse/pull/76358. [#77325](https://github.com/ClickHouse/ClickHouse/pull/77325) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix `async_insert` with `input`. [#77340](https://github.com/ClickHouse/ClickHouse/pull/77340) ([Azat Khuzhin](https://github.com/azat)). +* Fix: `WITH FILL` may fail with NOT_FOUND_COLUMN_IN_BLOCK when sorting column is removed by planer. Similar issue related to inconsistent DAG calculated for INTERPOLATE expression. [#77343](https://github.com/ClickHouse/ClickHouse/pull/77343) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* Fix several LOGICAL_ERRORs around setting alias of invalid AST nodes. [#77445](https://github.com/ClickHouse/ClickHouse/pull/77445) ([Raúl Marín](https://github.com/Algunenano)). +* In filesystem cache impementation fix error processing during file segment write. [#77471](https://github.com/ClickHouse/ClickHouse/pull/77471) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Make DatabaseIceberg use correct metadata file provided by catalog. Closes [#75187](https://github.com/ClickHouse/ClickHouse/issues/75187). [#77486](https://github.com/ClickHouse/ClickHouse/pull/77486) ([Kseniia Sumarokova](https://github.com/kssenii)). +* The query cache now assumes that UDFs are non-deterministic. Accordingly, results of queries with UDFs are no longer cached. Previously, users were able to define non-deterministic UDFs whose result would erronously be cached (issue [#77553](https://github.com/ClickHouse/ClickHouse/issues/77553)). [#77633](https://github.com/ClickHouse/ClickHouse/pull/77633) ([Jimmy Aguilar Mena](https://github.com/Ergus)). +* Fix system.filesystem_cache_log working only under setting `enable_filesystem_cache_log`. [#77650](https://github.com/ClickHouse/ClickHouse/pull/77650) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix a logical error when calling `defaultRoles` function inside a projection. Follow-up for [#76627](https://github.com/ClickHouse/ClickHouse/issues/76627). [#77667](https://github.com/ClickHouse/ClickHouse/pull/77667) ([pufit](https://github.com/pufit)). +* Second arguments of type `Nullable` for function `arrayResize` are now disallowed. Previously, anything from errors to wrong results could happen with `Nullable` as second argument. (issue [#48398](https://github.com/ClickHouse/ClickHouse/issues/48398)). [#77724](https://github.com/ClickHouse/ClickHouse/pull/77724) ([Manish Gill](https://github.com/mgill25)). +* Regularly check if merges and mutations were cancelled even in case when the operation doesn't produce any blocks to write. [#77766](https://github.com/ClickHouse/ClickHouse/pull/77766) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). + +#### Build/Testing/Packaging Improvement +* `clickhouse-odbc-bridge` and `clickhouse-library-bridge` are moved to a separate repository, https://github.com/ClickHouse/odbc-bridge/. [#76225](https://github.com/ClickHouse/ClickHouse/pull/76225) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Fix Rust cross-compilation and allow disabling Rust completely. [#76921](https://github.com/ClickHouse/ClickHouse/pull/76921) ([Raúl Marín](https://github.com/Algunenano)). + + +### ClickHouse release 25.2, 2025-02-27 {#252} + +#### Backward Incompatible Change +* Completely enable `async_load_databases` by default (even for those installations that do not upgrade `config.xml`). [#74772](https://github.com/ClickHouse/ClickHouse/pull/74772) ([Azat Khuzhin](https://github.com/azat)). +* Add `JSONCompactEachRowWithProgress` and `JSONCompactStringsEachRowWithProgress` formats. Continuation of [#69989](https://github.com/ClickHouse/ClickHouse/issues/69989). The `JSONCompactWithNames` and `JSONCompactWithNamesAndTypes` no longer output "totals" - apparently, it was a mistake in the implementation. [#75037](https://github.com/ClickHouse/ClickHouse/pull/75037) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Change the `format_alter_operations_with_parentheses` default to true to disambiguate the ALTER commands list (see https://github.com/ClickHouse/ClickHouse/pull/59532). This breaks replication with clusters prior to 24.3. If you are upgrading a cluster using older releases turn off the setting in the server config or upgrade to 24.3 first. [#75302](https://github.com/ClickHouse/ClickHouse/pull/75302) ([Raúl Marín](https://github.com/Algunenano)). +* Remove the possibility of filtering log messages using regular expressions. The implementation introduced a data race, so it has to be removed. [#75577](https://github.com/ClickHouse/ClickHouse/pull/75577) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). +* The setting `min_chunk_bytes_for_parallel_parsing` cannot be zero anymore. This fixes: [#71110](https://github.com/ClickHouse/ClickHouse/issues/71110). [#75239](https://github.com/ClickHouse/ClickHouse/pull/75239) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Validate settings in the cache configuration. Non existing settings were ignored before, now they will throw an error and they should be removed. [#75452](https://github.com/ClickHouse/ClickHouse/pull/75452) ([Kseniia Sumarokova](https://github.com/kssenii)). + +#### New Feature +* Support type `Nullable(JSON)`. [#73556](https://github.com/ClickHouse/ClickHouse/pull/73556) ([Pavel Kruglov](https://github.com/Avogar)). +* Support subcolumns in the DEFAULT and MATERIALIZED expressions. [#74403](https://github.com/ClickHouse/ClickHouse/pull/74403) ([Pavel Kruglov](https://github.com/Avogar)). +* Support writing Parquet bloom filters using the `output_format_parquet_write_bloom_filter` setting (enabled by default). [#71681](https://github.com/ClickHouse/ClickHouse/pull/71681) ([Michael Kolupaev](https://github.com/al13n321)). +* Web UI now has interactive database navigation. [#75777](https://github.com/ClickHouse/ClickHouse/pull/75777) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Allow the combination of read-only and read-write disks in the storage policy (as multiple volumes or multiple disks). This allows data to be read from the entire volume, while inserts will prefer the writable disk (i.e., Copy-on-Write storage policy). [#75862](https://github.com/ClickHouse/ClickHouse/pull/75862) ([Azat Khuzhin](https://github.com/azat)). +* Add a new Database engine, `DatabaseBackup,` that allows to instantly attach table/database from backup. [#75725](https://github.com/ClickHouse/ClickHouse/pull/75725) ([Maksim Kita](https://github.com/kitaisreal)). +* Support prepared statements in the Postgres wire protocol. [#75035](https://github.com/ClickHouse/ClickHouse/pull/75035) ([scanhex12](https://github.com/scanhex12)). +* Add the ability to ATTACH tables without a database layer, which is useful for MergeTree tables located on Web, S3, and similar external virtual filesystems. [#75788](https://github.com/ClickHouse/ClickHouse/pull/75788) ([Azat Khuzhin](https://github.com/azat)). +* Add a new string comparison function, `compareSubstrings` to compare parts of two strings. Example: `SELECT compareSubstrings('Saxony', 'Anglo-Saxon', 0, 6, 5) AS result` means "compare 6 bytes of strings 'Saxon' and 'Anglo-Saxon' lexicographically, starting at offset 0 in the first string, offset 5 in the second string". [#74070](https://github.com/ClickHouse/ClickHouse/pull/74070) ([lgbo](https://github.com/lgbo-ustc)). +* A new function, `initialQueryStartTime` is added. It returns the start time of the current query. The value is the same across all shards during a distributed query. [#75087](https://github.com/ClickHouse/ClickHouse/pull/75087) ([Roman Lomonosov](https://github.com/lomik)). +* Add support for SSL authentication with named collections for MySQL. Closes [#59111](https://github.com/ClickHouse/ClickHouse/issues/59111). [#59452](https://github.com/ClickHouse/ClickHouse/pull/59452) ([Nikolay Degterinsky](https://github.com/evillique)). + +#### Experimental Features +* Added a new setting, `enable_adaptive_memory_spill_scheduler` that allows multiple Grace JOINs in the same query to monitor their combined memory footprint and trigger spilling into external storage adaptively to prevent MEMORY_LIMIT_EXCEEDED. [#72728](https://github.com/ClickHouse/ClickHouse/pull/72728) ([lgbo](https://github.com/lgbo-ustc)). +* Make the new, experimental `Kafka` table engine fully respect Keeper feature flags. [#76004](https://github.com/ClickHouse/ClickHouse/pull/76004) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). +* Restore the (Intel) QPL codec, which was removed in v24.10 due to licensing issues. [#76021](https://github.com/ClickHouse/ClickHouse/pull/76021) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* For the integration with HDFS added support for the `dfs.client.use.datanode.hostname` configuration option. [#74635](https://github.com/ClickHouse/ClickHouse/pull/74635) ([Mikhail Tiukavkin](https://github.com/freshertm)). + +#### Performance Improvement +* Improve performance of the whole JSON column reading in Wide parts from S3. It's done by adding prefetches for subcolumn prefixes deserialization, cache of deserialized prefixes, and parallel deserialization of subcolumn prefixes. It improves the reading of the JSON column from S3 4 times in queries like `SELECT data FROM table` and about 10 times in queries like `SELECT data FROM table LIMIT 10`. [#74827](https://github.com/ClickHouse/ClickHouse/pull/74827) ([Pavel Kruglov](https://github.com/Avogar)). +* Fixed unnecessary contention in `parallel_hash` when `max_rows_in_join = max_bytes_in_join = 0`. [#75155](https://github.com/ClickHouse/ClickHouse/pull/75155) ([Nikita Taranov](https://github.com/nickitat)). +* Fixed double preallocation in `ConcurrentHashJoin` in case join sides are swapped by the optimizer. [#75149](https://github.com/ClickHouse/ClickHouse/pull/75149) ([Nikita Taranov](https://github.com/nickitat)). +* Slight improvement in some join scenarios: precalculate the number of output rows and reserve memory for them. [#75376](https://github.com/ClickHouse/ClickHouse/pull/75376) ([Alexander Gololobov](https://github.com/davenger)). +* For queries like `WHERE a < b AND b < c AND c < 5`, we can infer new comparing conditions (`a < 5 AND b < 5`) to have better filter ability. [#73164](https://github.com/ClickHouse/ClickHouse/pull/73164) ([Shichao Jin](https://github.com/jsc0218)). +* Keeper improvement: disable digest calculation when committing to in-memory storage for better performance. It can be enabled with `keeper_server.digest_enabled_on_commit` config. Digest is still calculated when preprocessing requests. [#75490](https://github.com/ClickHouse/ClickHouse/pull/75490) ([Antonio Andelic](https://github.com/antonio2368)). +* Push down filter expression from JOIN ON when possible. [#75536](https://github.com/ClickHouse/ClickHouse/pull/75536) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Calculate columns and indices sizes lazily in MergeTree. [#75938](https://github.com/ClickHouse/ClickHouse/pull/75938) ([Pavel Kruglov](https://github.com/Avogar)). +* Reintroduce respect `ttl_only_drop_parts` on `MATERIALIZE TTL`; only read necessary columns to recalculate TTL and drop parts by replacing them with an empty ones. [#72751](https://github.com/ClickHouse/ClickHouse/pull/72751) ([Andrey Zvonov](https://github.com/zvonand)). +* Reduce write buffer size for plain_rewritable metadata files [#75758](https://github.com/ClickHouse/ClickHouse/pull/75758) ([Julia Kartseva](https://github.com/jkartseva)). +* Reduce memory usage for some window functions. [#65647](https://github.com/ClickHouse/ClickHouse/pull/65647) ([lgbo](https://github.com/lgbo-ustc)). +* Evaluate parquet bloom filters and min/max indexes together. Necessary to properly support: `x = 3 or x > 5` where data = [1, 2, 4, 5]. [#71383](https://github.com/ClickHouse/ClickHouse/pull/71383) ([Arthur Passos](https://github.com/arthurpassos)). +* Queries passed to the `Executable` storage are no longer limited to a single-threaded execution. [#70084](https://github.com/ClickHouse/ClickHouse/pull/70084) ([yawnt](https://github.com/yawnt)). +* Fetch parts in parallel in ALTER TABLE FETCH PARTITION (thread pool size is controlled with `max_fetch_partition_thread_pool_size`). [#74978](https://github.com/ClickHouse/ClickHouse/pull/74978) ([Azat Khuzhin](https://github.com/azat)). +* Allow to move predicates with the `indexHint` function to `PREWHERE`. [#74987](https://github.com/ClickHouse/ClickHouse/pull/74987) ([Anton Popov](https://github.com/CurtizJ)). + +#### Improvement +* Fixed calculation of size in memory for `LowCardinality` columns. [#74688](https://github.com/ClickHouse/ClickHouse/pull/74688) ([Nikita Taranov](https://github.com/nickitat)). +* `processors_profile_log` table now has a default configuration with a TTL of 30 days. [#66139](https://github.com/ClickHouse/ClickHouse/pull/66139) ([Ilya Yatsishin](https://github.com/qoega)). +* Allow naming shards in the cluster configuration. [#72276](https://github.com/ClickHouse/ClickHouse/pull/72276) ([MikhailBurdukov](https://github.com/MikhailBurdukov)). +* Change Prometheus remote write response success status from 200/OK to 204/NoContent. [#74170](https://github.com/ClickHouse/ClickHouse/pull/74170) ([Michael Dempsey](https://github.com/bluestealth)). +* Add ability to reload `max_remote_read_network_bandwidth_for_serve` and `max_remote_write_network_bandwidth_for_server` on the fly without restart server. [#74206](https://github.com/ClickHouse/ClickHouse/pull/74206) ([Kai Zhu](https://github.com/nauu)). +* Allow using blob paths to calculate checksums while making a backup. [#74729](https://github.com/ClickHouse/ClickHouse/pull/74729) ([Vitaly Baranov](https://github.com/vitlibar)). +* Added a query ID column to `system.query_cache` (closes [#68205](https://github.com/ClickHouse/ClickHouse/issues/68205)). [#74982](https://github.com/ClickHouse/ClickHouse/pull/74982) ([NamHoaiNguyen](https://github.com/NamHoaiNguyen)). +* It is allowed to cancel `ALTER TABLE ... FREEZE ...` queries with `KILL QUERY` and automatically by a timeout (`max_execution_time`). [#75016](https://github.com/ClickHouse/ClickHouse/pull/75016) ([Kirill](https://github.com/kirillgarbar)). +* Add support for `groupUniqArrayArrayMap` as `SimpleAggregateFunction`. [#75034](https://github.com/ClickHouse/ClickHouse/pull/75034) ([Miel Donkers](https://github.com/mdonkers)). +* Hide catalog credential settings in database engine `Iceberg`. Closes [#74559](https://github.com/ClickHouse/ClickHouse/issues/74559). [#75080](https://github.com/ClickHouse/ClickHouse/pull/75080) ([Kseniia Sumarokova](https://github.com/kssenii)). +* `intExp2` / `intExp10`: Define undefined behaviour: return 0 for too small argument, `18446744073709551615` for too big argument, throw exception if `nan`. [#75312](https://github.com/ClickHouse/ClickHouse/pull/75312) ([Vitaly Baranov](https://github.com/vitlibar)). +* Support `s3.endpoint` natively from catalog config in `DatabaseIceberg`. Closes [#74558](https://github.com/ClickHouse/ClickHouse/issues/74558). [#75375](https://github.com/ClickHouse/ClickHouse/pull/75375) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Don't fail silently if a user executing `SYSTEM DROP REPLICA` doesn't have enough permissions. [#75377](https://github.com/ClickHouse/ClickHouse/pull/75377) ([Bharat Nallan](https://github.com/bharatnc)). +* Add a ProfileEvent about the number of times any of the system logs have failed to flush. [#75466](https://github.com/ClickHouse/ClickHouse/pull/75466) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Add a check and extra logging for decrypting and decompressing. [#75471](https://github.com/ClickHouse/ClickHouse/pull/75471) ([Vitaly Baranov](https://github.com/vitlibar)). +* Added support for the micro sign (U+00B5) in the `parseTimeDelta` function. Now both the micro sign (U+00B5) and the Greek letter mu (U+03BC) are recognized as valid representations for microseconds, aligning ClickHouse's behavior with Go’s implementation ([see time.go](https://github.com/golang/go/blob/ad7b46ee4ac1cee5095d64b01e8cf7fcda8bee5e/src/time/time.go#L983C19-L983C20) and [time/format.go](https://github.com/golang/go/blob/ad7b46ee4ac1cee5095d64b01e8cf7fcda8bee5e/src/time/format.go#L1608-L1609)). [#75472](https://github.com/ClickHouse/ClickHouse/pull/75472) ([Vitaly Orlov](https://github.com/orloffv)). +* Replace server setting (`send_settings_to_client`) with client setting (`apply_settings_from_server`) that controls whether client-side code (e.g., parsing INSERT data and formatting query output) should use settings from server's `users.xml` and user profile. Otherwise, only settings from the client command line, session, and query are used. Note that this only applies to native client (not e.g. HTTP), and doesn't apply to most of query processing (which happens on the server). [#75478](https://github.com/ClickHouse/ClickHouse/pull/75478) ([Michael Kolupaev](https://github.com/al13n321)). +* Better error messages for syntax errors. Previously, if the query was too large, and the token whose length exceeds the limit is a very large string literal, the message about the reason was lost in the middle of two examples of this very long token. Fix the issue when a query with UTF-8 was cut incorrectly in the error message. Fix excessive quoting of query fragments. This closes [#75473](https://github.com/ClickHouse/ClickHouse/issues/75473). [#75561](https://github.com/ClickHouse/ClickHouse/pull/75561) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Add profile events in storage `S3(Azure)Queue`. [#75618](https://github.com/ClickHouse/ClickHouse/pull/75618) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Disable sending settings from server to client (`send_settings_to_client=false`) for compatibility (This feature will be re-implemented as a client setting later for better usability). [#75648](https://github.com/ClickHouse/ClickHouse/pull/75648) ([Michael Kolupaev](https://github.com/al13n321)). +* Add a config `memory_worker_correct_memory_tracker` to enable correction of the internal memory tracker with information from different sources read in the background thread periodically. [#75714](https://github.com/ClickHouse/ClickHouse/pull/75714) ([Antonio Andelic](https://github.com/antonio2368)). +* Add column `normalized_query_hash` into `system.processes`. Note: while it can be easily calculated on the fly with the `normalizedQueryHash` function, this is needed to prepare for subsequent changes. [#75756](https://github.com/ClickHouse/ClickHouse/pull/75756) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Querying `system.tables` will not throw even if there is a `Merge` table created over a database that no longer exists. Remove the `getTotalRows` method from `Hive` tables because we don't allow it to do complex work. [#75772](https://github.com/ClickHouse/ClickHouse/pull/75772) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Store start_time/end_time for Backups with microseconds. [#75929](https://github.com/ClickHouse/ClickHouse/pull/75929) ([Aleksandr Musorin](https://github.com/AVMusorin)). +* Add `MemoryTrackingUncorrected` metric showing the value of the internal global memory tracker which is not corrected by RSS. [#75935](https://github.com/ClickHouse/ClickHouse/pull/75935) ([Antonio Andelic](https://github.com/antonio2368)). +* Allow parsing endpoints like `localhost:1234/handle` in `PostgreSQL` or `MySQL` table functions. This fixes a regression introduced in https://github.com/ClickHouse/ClickHouse/pull/52503. [#75944](https://github.com/ClickHouse/ClickHouse/pull/75944) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Added a server setting `throw_on_unknown_workload` that allows to choose behavior on query with `workload` setting set to unknown value: either allow unlimited access (default) or throw a `RESOURCE_ACCESS_DENIED` error. It is useful to force all queries to use workload scheduling. [#75999](https://github.com/ClickHouse/ClickHouse/pull/75999) ([Sergei Trifonov](https://github.com/serxa)). +* Don't rewrite subcolumns to `getSubcolumn` in `ARRAY JOIN` if not necessary. [#76018](https://github.com/ClickHouse/ClickHouse/pull/76018) ([Pavel Kruglov](https://github.com/Avogar)). +* Retry coordination errors when loading tables. [#76020](https://github.com/ClickHouse/ClickHouse/pull/76020) ([Alexander Tokmakov](https://github.com/tavplubix)). +* Support flushing individual logs in `SYSTEM FLUSH LOGS`. [#76132](https://github.com/ClickHouse/ClickHouse/pull/76132) ([Raúl Marín](https://github.com/Algunenano)). +* Improved the `/binary` server's page. Using the Hilbert curve instead of the Morton curve. Display 512 MB worth of addresses in the square, which fills the square better (in previous versions, addresses fill only half of the square). Color addresses closer to the library name rather than the function name. Allow scrolling a bit more outside of the area. [#76192](https://github.com/ClickHouse/ClickHouse/pull/76192) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Retry ON CLUSTER queries in case of TOO_MANY_SIMULTANEOUS_QUERIES. [#76352](https://github.com/ClickHouse/ClickHouse/pull/76352) ([Patrick Galbraith](https://github.com/CaptTofu)). +* Add the `CPUOverload` asynchronous metric, which calculates the server's relative CPU deficit. [#76404](https://github.com/ClickHouse/ClickHouse/pull/76404) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Changed the default value of `output_format_pretty_max_rows` from 10000 to 1000. I think it is better for usability. [#76407](https://github.com/ClickHouse/ClickHouse/pull/76407) ([Alexey Milovidov](https://github.com/alexey-milovidov)). + +#### Bug Fix (user-visible misbehavior in an official stable release) +* Fix formatting of exceptions using a custom format if they appear during query interpretation. In previous versions, exceptions were formatted using the default format rather than the format specified in the query. This closes [#55422](https://github.com/ClickHouse/ClickHouse/issues/55422). [#74994](https://github.com/ClickHouse/ClickHouse/pull/74994) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Fix type mapping for SQLite (integer types into `int64`, floating points into `float64`). [#73853](https://github.com/ClickHouse/ClickHouse/pull/73853) ([Joanna Hulboj](https://github.com/jh0x)). +* Fix identifier resolution from parent scopes. Allow the use of aliases to expressions in the WITH clause. Fixes [#58994](https://github.com/ClickHouse/ClickHouse/issues/58994). Fixes [#62946](https://github.com/ClickHouse/ClickHouse/issues/62946). Fixes [#63239](https://github.com/ClickHouse/ClickHouse/issues/63239). Fixes [#65233](https://github.com/ClickHouse/ClickHouse/issues/65233). Fixes [#71659](https://github.com/ClickHouse/ClickHouse/issues/71659). Fixes [#71828](https://github.com/ClickHouse/ClickHouse/issues/71828). Fixes [#68749](https://github.com/ClickHouse/ClickHouse/issues/68749). [#66143](https://github.com/ClickHouse/ClickHouse/pull/66143) ([Dmitry Novik](https://github.com/novikd)). +* Fix negate function monotonicity. In previous versions, the query `select * from a where -x = -42;` where `x` is the primary key, can return a wrong result. [#71440](https://github.com/ClickHouse/ClickHouse/pull/71440) ([Michael Kolupaev](https://github.com/al13n321)). +* Fix empty tuple handling in arrayIntersect. This fixes [#72578](https://github.com/ClickHouse/ClickHouse/issues/72578). [#72581](https://github.com/ClickHouse/ClickHouse/pull/72581) ([Amos Bird](https://github.com/amosbird)). +* Fix reading JSON sub-object subcolumns with incorrect prefix. [#73182](https://github.com/ClickHouse/ClickHouse/pull/73182) ([Pavel Kruglov](https://github.com/Avogar)). +* Propagate Native format settings properly for client-server communication. [#73924](https://github.com/ClickHouse/ClickHouse/pull/73924) ([Pavel Kruglov](https://github.com/Avogar)). +* Check for not supported types for some storages. [#74218](https://github.com/ClickHouse/ClickHouse/pull/74218) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix crash with query `INSERT INTO SELECT` over PostgreSQL interface on macOS (issue [#72938](https://github.com/ClickHouse/ClickHouse/issues/72938)). [#74231](https://github.com/ClickHouse/ClickHouse/pull/74231) ([Artem Yurov](https://github.com/ArtemYurov)). +* Fixed uninitialized max_log_ptr in the replicated database. [#74336](https://github.com/ClickHouse/ClickHouse/pull/74336) ([Konstantin Morozov](https://github.com/k-morozov)). +* Fix crash when inserting interval (issue [#74299](https://github.com/ClickHouse/ClickHouse/issues/74299)). [#74478](https://github.com/ClickHouse/ClickHouse/pull/74478) ([NamHoaiNguyen](https://github.com/NamHoaiNguyen)). +* Fix formatting constant JSON literals. Previously it could lead to syntax errors during sending the query to another server. [#74533](https://github.com/ClickHouse/ClickHouse/pull/74533) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix broken create query when using constant partition expressions with implicit projections enabled. This fixes [#74596](https://github.com/ClickHouse/ClickHouse/issues/74596) . [#74634](https://github.com/ClickHouse/ClickHouse/pull/74634) ([Amos Bird](https://github.com/amosbird)). +* Avoid leaving connection in broken state after INSERT finishes with exception. [#74740](https://github.com/ClickHouse/ClickHouse/pull/74740) ([Azat Khuzhin](https://github.com/azat)). +* Avoid reusing connections that had been left in the intermediate state. [#74749](https://github.com/ClickHouse/ClickHouse/pull/74749) ([Azat Khuzhin](https://github.com/azat)). +* Fix crash during JSON type declaration parsing when type name is not uppercase. [#74784](https://github.com/ClickHouse/ClickHouse/pull/74784) ([Pavel Kruglov](https://github.com/Avogar)). +* Keeper: fix logical_error when the connection had been terminated before establishing. [#74844](https://github.com/ClickHouse/ClickHouse/pull/74844) ([Michael Kolupaev](https://github.com/al13n321)). +* Fix a behavior when the server couldn't startup when there's a table using `AzureBlobStorage`. Tables are loaded without any requests to Azure. [#74880](https://github.com/ClickHouse/ClickHouse/pull/74880) ([Alexey Katsman](https://github.com/alexkats)). +* Fix missing `used_privileges` and `missing_privileges` fields in `query_log` for BACKUP and RESTORE operations. [#74887](https://github.com/ClickHouse/ClickHouse/pull/74887) ([Alexey Katsman](https://github.com/alexkats)). +* HDFS refresh krb ticket if sasl error during hdfs select request. [#74930](https://github.com/ClickHouse/ClickHouse/pull/74930) ([inv2004](https://github.com/inv2004)). +* Fix queries to Replicated database in startup_scripts. [#74942](https://github.com/ClickHouse/ClickHouse/pull/74942) ([Azat Khuzhin](https://github.com/azat)). +* Fix issues with expressions type aliased in the JOIN ON clause when a null-safe comparison is used. [#74970](https://github.com/ClickHouse/ClickHouse/pull/74970) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Revert part's state from deleting back to outdated when remove operation has failed. [#74985](https://github.com/ClickHouse/ClickHouse/pull/74985) ([Sema Checherinda](https://github.com/CheSema)). +* In previous versions, when there was a scalar subquery, we started writing the progress (accumulated from processing the subquery) during the initialization of the data format, which was before HTTP headers were written. This led to the loss of HTTP headers, such as X-ClickHouse-QueryId and X-ClickHouse-Format, as well as Content-Type. [#74991](https://github.com/ClickHouse/ClickHouse/pull/74991) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Fix `CREATE TABLE AS...` queries for `database_replicated_allow_replicated_engine_arguments=0`. [#75000](https://github.com/ClickHouse/ClickHouse/pull/75000) ([Bharat Nallan](https://github.com/bharatnc)). +* Fix leaving connection in a bad state in client after INSERT exceptions. [#75030](https://github.com/ClickHouse/ClickHouse/pull/75030) ([Azat Khuzhin](https://github.com/azat)). +* Fix crash due to uncaught exception in PSQL replication. [#75062](https://github.com/ClickHouse/ClickHouse/pull/75062) ([Azat Khuzhin](https://github.com/azat)). +* Sasl can fail any rpc call, the fix helps to repeat the call in case if krb5 ticker is expired. [#75063](https://github.com/ClickHouse/ClickHouse/pull/75063) ([inv2004](https://github.com/inv2004)). +* Fixed usage of indexes (primary and secondary) for `Array`, `Map` and `Nullable(..)` columns with enabled setting `optimize_function_to_subcolumns`. Previously, indexes for these columns could have been ignored. [#75081](https://github.com/ClickHouse/ClickHouse/pull/75081) ([Anton Popov](https://github.com/CurtizJ)). +* Disable `flatten_nested` when creating materialized views with inner tables since it will not be possible to use such flattened columns. [#75085](https://github.com/ClickHouse/ClickHouse/pull/75085) ([Christoph Wurm](https://github.com/cwurm)). +* Fix for some of IPv6 addresses (such as ::ffff:1.1.1.1) in forwarded_for field is wrongly interpreted resulting in client disconnect with exception. [#75133](https://github.com/ClickHouse/ClickHouse/pull/75133) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* Fix nullsafe JOIN handling for LowCardinality nullable data type. Previously JOIN ON with nullsafe comparison, such as `IS NOT DISTINCT FROM`, `<=>` , `a IS NULL AND b IS NULL OR a == b` didn't work correctly with LowCardinality columns. [#75143](https://github.com/ClickHouse/ClickHouse/pull/75143) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Checks that we don't specify key_condition when counting total_number_of_rows for NumRowsCache. [#75164](https://github.com/ClickHouse/ClickHouse/pull/75164) ([Daniil Ivanik](https://github.com/divanik)). +* Fix queries with unused interpolation with the new analyzer. [#75173](https://github.com/ClickHouse/ClickHouse/pull/75173) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). +* Fix the crash bug of CTE with Insert. [#75188](https://github.com/ClickHouse/ClickHouse/pull/75188) ([Shichao Jin](https://github.com/jsc0218)). +* Keeper fix: avoid writing to broken changelogs when rolling back logs. [#75197](https://github.com/ClickHouse/ClickHouse/pull/75197) ([Antonio Andelic](https://github.com/antonio2368)). +* Use `BFloat16` as a supertype where appropriate. This closes: [#74404](https://github.com/ClickHouse/ClickHouse/issues/74404). [#75236](https://github.com/ClickHouse/ClickHouse/pull/75236) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Fix unexpected defaults in join result with any_join_distinct_right_table_keys and OR in JOIN ON. [#75262](https://github.com/ClickHouse/ClickHouse/pull/75262) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Mask azureblobstorage table engine credentials. [#75319](https://github.com/ClickHouse/ClickHouse/pull/75319) ([Garrett Thomas](https://github.com/garrettthomaskth)). +* Fixed behavior when ClickHouse may erroneously do a filter pushdown to an external database like PostgreSQL, MySQL, or SQLite. This closes: [#71423](https://github.com/ClickHouse/ClickHouse/issues/71423). [#75320](https://github.com/ClickHouse/ClickHouse/pull/75320) ([Nikita Mikhaylov](https://github.com/nikitamikhaylov)). +* Fix crash in protobuf schema cache that can happen during output in Protobuf format and parallel query `SYSTEM DROP FORMAT SCHEMA CACHE`. [#75357](https://github.com/ClickHouse/ClickHouse/pull/75357) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix a possible logical error or uninitialized memory issue when a filter from `HAVING` is pushed down with parallel replicas. [#75363](https://github.com/ClickHouse/ClickHouse/pull/75363) ([Vladimir Cherkasov](https://github.com/vdimir)). +* Hide sensitive info for `icebergS3`, `icebergAzure` table functions and table engines. [#75378](https://github.com/ClickHouse/ClickHouse/pull/75378) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Function `TRIM` with computed empty trim characters are now correctly handled. Example: `SELECT TRIM(LEADING concat('') FROM 'foo')` (Issue [#69922](https://github.com/ClickHouse/ClickHouse/issues/69922)). [#75399](https://github.com/ClickHouse/ClickHouse/pull/75399) ([Manish Gill](https://github.com/mgill25)). +* Fix data race in IOutputFormat. [#75448](https://github.com/ClickHouse/ClickHouse/pull/75448) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix possible error `Elements ... and ... of Nested data structure ... (Array columns) have different array sizes` when JSON subcolumns with Array type are used in JOIN over distributed tables. [#75512](https://github.com/ClickHouse/ClickHouse/pull/75512) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix data corruption with `CODEC(ZSTD, DoubleDelta)`. Closes [#70031](https://github.com/ClickHouse/ClickHouse/issues/70031). [#75548](https://github.com/ClickHouse/ClickHouse/pull/75548) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Fix interaction between allow_feature_tier and compatibility mergetree setting. [#75635](https://github.com/ClickHouse/ClickHouse/pull/75635) ([Raúl Marín](https://github.com/Algunenano)). +* Fix incorrect processed_rows value in system.s3queue_log in case file was retried. [#75666](https://github.com/ClickHouse/ClickHouse/pull/75666) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Respect `materialized_views_ignore_errors` when a materialized view writes to a URL engine and there is a connectivity issue. [#75679](https://github.com/ClickHouse/ClickHouse/pull/75679) ([Christoph Wurm](https://github.com/cwurm)). +* Fixed rare crashes while reading from `MergeTree` table after multiple asynchronous `RENAME` queries (with `alter_sync = 0`) between columns with different types. [#75693](https://github.com/ClickHouse/ClickHouse/pull/75693) ([Anton Popov](https://github.com/CurtizJ)). +* Fix `Block structure mismatch in QueryPipeline stream` error for some queries with `UNION ALL`. [#75715](https://github.com/ClickHouse/ClickHouse/pull/75715) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Rebuild projection on alter modify of its PK column. Previously it could lead to `CANNOT_READ_ALL_DATA` errors during selects after alter modify of the column used in projection PK. [#75720](https://github.com/ClickHouse/ClickHouse/pull/75720) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix incorrect result of `ARRAY JOIN` for scalar subqueries (with analyzer). [#75732](https://github.com/ClickHouse/ClickHouse/pull/75732) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fixed null pointer dereference in `DistinctSortedStreamTransform`. [#75734](https://github.com/ClickHouse/ClickHouse/pull/75734) ([Nikita Taranov](https://github.com/nickitat)). +* Fix `allow_suspicious_ttl_expressions` behaviour. [#75771](https://github.com/ClickHouse/ClickHouse/pull/75771) ([Aleksei Filatov](https://github.com/aalexfvk)). +* Fix uninitialized memory read in function `translate`. This closes [#75592](https://github.com/ClickHouse/ClickHouse/issues/75592). [#75794](https://github.com/ClickHouse/ClickHouse/pull/75794) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Propagate format settings to JSON as string formatting in Native format. [#75832](https://github.com/ClickHouse/ClickHouse/pull/75832) ([Pavel Kruglov](https://github.com/Avogar)). +* Recorded the default enablement of parallel hash as join algorithm in v24.12 in the settings change history. This means that ClickHouse will continue to join using non-parallel hash if an older compatibility level than v24.12 is configured. [#75870](https://github.com/ClickHouse/ClickHouse/pull/75870) ([Robert Schulze](https://github.com/rschu1ze)). +* Fixed a bug that tables with implicitly added min-max indices could not be copied into a new table (issue [#75677](https://github.com/ClickHouse/ClickHouse/issues/75677)). [#75877](https://github.com/ClickHouse/ClickHouse/pull/75877) ([Smita Kulkarni](https://github.com/SmitaRKulkarni)). +* `clickhouse-library-bridge` allows opening arbitrary libraries from the filesystem, which makes it safe to run only inside an isolated environment. To prevent a vulnerability when it is run near the clickhouse-server, we will limit the paths of libraries to a location, provided in the configuration. This vulnerability was found with the [ClickHouse Bug Bounty Program](https://github.com/ClickHouse/ClickHouse/issues/38986) by **Arseniy Dugin**. [#75954](https://github.com/ClickHouse/ClickHouse/pull/75954) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* We happened to use JSON serialization for some metadata, which was a mistake, because JSON does not support binary data inside string literals, including zero bytes. SQL queries can contain binary data and invalid UTF-8, so we have to support this in our metadata files as well. At the same time, ClickHouse's `JSONEachRow` and similar formats work around that by deviating from the JSON standard in favor of a perfect roundtrip for the binary data. See the motivation here: https://github.com/ClickHouse/ClickHouse/pull/73668#issuecomment-2560501790. The solution is to make `Poco::JSON` library consistent with the JSON format serialization in ClickHouse. This closes [#73668](https://github.com/ClickHouse/ClickHouse/issues/73668). [#75963](https://github.com/ClickHouse/ClickHouse/pull/75963) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Fix check for commit limits in storage `S3Queue`. [#76104](https://github.com/ClickHouse/ClickHouse/pull/76104) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix attaching MergeTree tables with auto indexes (`add_minmax_index_for_numeric_columns`/`add_minmax_index_for_string_columns`). [#76139](https://github.com/ClickHouse/ClickHouse/pull/76139) ([Azat Khuzhin](https://github.com/azat)). +* Fixed issue of stack traces from parent threads of a job (`enable_job_stack_trace` setting) are not printed out. Fixed issue `enable_job_stack_trace` setting is not properly propagated to the threads resulting stack trace content not always respects this setting. [#76191](https://github.com/ClickHouse/ClickHouse/pull/76191) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* Fix incorrect permission check where `ALTER RENAME` required `CREATE USER` grant. Closes [#74372](https://github.com/ClickHouse/ClickHouse/issues/74372). [#76241](https://github.com/ClickHouse/ClickHouse/pull/76241) ([pufit](https://github.com/pufit)). +* Fix reinterpretAs with FixedString on big-endian architecture. [#76253](https://github.com/ClickHouse/ClickHouse/pull/76253) ([Azat Khuzhin](https://github.com/azat)). +* Fix logical error in S3Queue "Expected current processor {} to be equal to {} for bucket {}". [#76358](https://github.com/ClickHouse/ClickHouse/pull/76358) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix deadlock for ALTER with Memory database. [#76359](https://github.com/ClickHouse/ClickHouse/pull/76359) ([Azat Khuzhin](https://github.com/azat)). +* Fix logical error in index analysis if condition in `WHERE` has `pointInPolygon` function. [#76360](https://github.com/ClickHouse/ClickHouse/pull/76360) ([Anton Popov](https://github.com/CurtizJ)). +* Fix potentially unsafe call in signal handler. [#76549](https://github.com/ClickHouse/ClickHouse/pull/76549) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* Fix reverse key support in PartsSplitter. This fixes [#73400](https://github.com/ClickHouse/ClickHouse/issues/73400). [#73418](https://github.com/ClickHouse/ClickHouse/pull/73418) ([Amos Bird](https://github.com/amosbird)). + +#### Build/Testing/Packaging Improvement +* Support build HDFS on both ARM and Intel Mac. [#74244](https://github.com/ClickHouse/ClickHouse/pull/74244) ([Yan Xin](https://github.com/yxheartipp)). +* Enable ICU and GRPC when cross-compiling for Darwin. [#75922](https://github.com/ClickHouse/ClickHouse/pull/75922) ([Raúl Marín](https://github.com/Algunenano)). +* Update to embedded LLVM 19. [#75148](https://github.com/ClickHouse/ClickHouse/pull/75148) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Disable network access for user default in the docker image. [#75259](https://github.com/ClickHouse/ClickHouse/pull/75259) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). Make all clickhouse-server related actions a function, and execute them only when launching the default binary in `entrypoint.sh`. A long-postponed improvement was suggested in [#50724](https://github.com/ClickHouse/ClickHouse/issues/50724). Added switch `--users` to `clickhouse-extract-from-config` to get values from the `users.xml`. [#75643](https://github.com/ClickHouse/ClickHouse/pull/75643) ([Mikhail f. Shiryaev](https://github.com/Felixoid)). +* Remove about 20MB of dead code from the binary. [#76226](https://github.com/ClickHouse/ClickHouse/pull/76226) ([Alexey Milovidov](https://github.com/alexey-milovidov)). + +### ClickHouse release 25.1, 2025-01-28 {#251} + +#### Backward Incompatible Change +* `JSONEachRowWithProgress` will write the progress whenever the progress happens. In previous versions, the progress was shown only after each block of the result, which made it useless. Change the way how the progress is displayed: it will not show zero values. This closes [#70800](https://github.com/ClickHouse/ClickHouse/issues/70800). [#73834](https://github.com/ClickHouse/ClickHouse/pull/73834) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* `Merge` tables will unify the structure of underlying tables by using a union of their columns and deriving common types. This closes [#64864](https://github.com/ClickHouse/ClickHouse/issues/64864). In certain cases, this change could be backward incompatible. One example is when there is no common type between tables, but conversion to the type of the first table is still possible, such as in the case of UInt64 and Int64 or any numeric type and String. If you want to return to the old behavior, set `merge_table_max_tables_to_look_for_schema_inference` to `1` or set `compatibility` to `24.12` or earlier. [#73956](https://github.com/ClickHouse/ClickHouse/pull/73956) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Parquet output format converts Date and DateTime columns to date/time types supported by Parquet, instead of writing them as raw numbers. `DateTime` becomes `DateTime64(3)` (was: `UInt32`); setting `output_format_parquet_datetime_as_uint32` brings back the old behavior. `Date` becomes `Date32` (was: `UInt16`). [#70950](https://github.com/ClickHouse/ClickHouse/pull/70950) ([Michael Kolupaev](https://github.com/al13n321)). +* Don't allow not comparable types (like `JSON`/`Object`/`AggregateFunction`) in `ORDER BY` and comparison functions `less/greater/equal/etc` by default. [#73276](https://github.com/ClickHouse/ClickHouse/pull/73276) ([Pavel Kruglov](https://github.com/Avogar)). +* The obsolete `MaterializedMySQL` database engine has been removed and is no longer available. [#73879](https://github.com/ClickHouse/ClickHouse/pull/73879) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* The `mysql` dictionary source no longer does `SHOW TABLE STATUS` query, because it does not provide any value for InnoDB tables, as long as for any recent MySQL versions. This closes [#72636](https://github.com/ClickHouse/ClickHouse/issues/72636). This change is backward compatible, but put in this category so you have a chance to notice it. [#73914](https://github.com/ClickHouse/ClickHouse/pull/73914) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* `CHECK TABLE` queries now require a separate, `CHECK` grant. In previous versions, it was enough to have `SHOW TABLES` grant to run these queries. But a `CHECK TABLE` query can be heavy, and usual query complexity limits for `SELECT` queries don't apply to it. It led to the potential of DoS. [#74471](https://github.com/ClickHouse/ClickHouse/pull/74471) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Function `h3ToGeo()` now returns the results in the order `(lat, lon)` (which is the standard order for geometric functions). Users who wish to retain the legacy result order `(lon, lat)` can set setting `h3togeo_lon_lat_result_order = true`. [#74719](https://github.com/ClickHouse/ClickHouse/pull/74719) ([Manish Gill](https://github.com/mgill25)). +* A new MongoDB driver is now the default. Users who like to continue using the legacy driver can set server setting `use_legacy_mongodb_integration` to true. [#73359](https://github.com/ClickHouse/ClickHouse/pull/73359) ([Robert Schulze](https://github.com/rschu1ze)). + +#### New Feature +* Added an ability to apply non-finished (not materialized by background process) mutations during the execution of `SELECT` queries immediately after submitting. It can be enabled by setting `apply_mutations_on_fly`. [#74877](https://github.com/ClickHouse/ClickHouse/pull/74877) ([Anton Popov](https://github.com/CurtizJ)). +* Implement `Iceberg` tables partition pruning for time-related transform partition operations in Iceberg. [#72044](https://github.com/ClickHouse/ClickHouse/pull/72044) ([Daniil Ivanik](https://github.com/divanik)). +* Support subcolumns in MergeTree sorting key and skip indexes. [#72644](https://github.com/ClickHouse/ClickHouse/pull/72644) ([Pavel Kruglov](https://github.com/Avogar)). +* Support reading `HALF_FLOAT` values from `Apache Arrow`/`Parquet`/`ORC` (they are read into `Float32`). This closes [#72960](https://github.com/ClickHouse/ClickHouse/issues/72960). Keep in mind that IEEE-754 half float is not the same as `BFloat16`. Closes [#73835](https://github.com/ClickHouse/ClickHouse/issues/73835). [#73836](https://github.com/ClickHouse/ClickHouse/pull/73836) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* The `system.trace_log` table will contain two new columns, `symbols` and `lines` containing symbolized stack trace. It allows for easy collection and export of profile information. This is controlled by the server configuration value `symbolize` inside `trace_log` and is enabled by default. [#73896](https://github.com/ClickHouse/ClickHouse/pull/73896) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Add a new function, `generateSerialID`, which can be used to generate auto-incremental numbers in tables. Continuation of [#64310](https://github.com/ClickHouse/ClickHouse/issues/64310) by [kazalika](https://github.com/kazalika). This closes [#62485](https://github.com/ClickHouse/ClickHouse/issues/62485). [#73950](https://github.com/ClickHouse/ClickHouse/pull/73950) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Add syntax `query1 PARALLEL WITH query2 PARALLEL WITH query3 ... PARALLEL WITH queryN` for DDL queries. That means subqueries `{query1, query2, ... queryN}` are allowed to run in parallel with each other (and it's preferable). [#73983](https://github.com/ClickHouse/ClickHouse/pull/73983) ([Vitaly Baranov](https://github.com/vitlibar)). +* Added an in-memory cache for deserialized skipping index granules. This should make repeated queries that use skipping indexes faster. The size of the new cache is controlled by server settings `skipping_index_cache_size` and `skipping_index_cache_max_entries`. The original motivation for the cache were vector similarity indexes which became a lot faster now. [#70102](https://github.com/ClickHouse/ClickHouse/pull/70102) ([Robert Schulze](https://github.com/rschu1ze)). +* Now, the embedded Web UI has a progress bar during query runtime. It allows cancelling queries. It displays the total number of records and the extended information about the speed. The table can be rendered incrementally as soon as data arrives. Enable HTTP compression. Rendering of the table became faster. The table header became sticky. It allows selecting cells and navigating them by arrow keys. Fix the issue when the outline of the selected cell makes it smaller. Cells no longer expand on mouse hover but only on selection. The moment to stop rendering the incoming data is decided on the client rather than on the server side. Highlight digit groups for numbers. The overall design was refreshed and became bolder. It checks if the server is reachable and the correctness of credentials and displays the server version and uptime. The cloud icon is contoured in every font, even in Safari. Big integers inside nested data types will be rendered better. It will display inf/nan correctly. It will display data types when the mouse is over a column header. [#74204](https://github.com/ClickHouse/ClickHouse/pull/74204) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Add the ability to create min-max (skipping) indices by default for columns managed by MergeTree using settings `add_minmax_index_for_numeric_columns` (for numeric columns) and `add_minmax_index_for_string_columns` (for string columns). For now, both settings are disabled, so there is no behavior change yet. [#74266](https://github.com/ClickHouse/ClickHouse/pull/74266) ([Smita Kulkarni](https://github.com/SmitaRKulkarni)). +* Add `script_query_number` and `script_line_number` fields to `system.query_log`, to the ClientInfo in the native protocol, and to server logs. This closes [#67542](https://github.com/ClickHouse/ClickHouse/issues/67542). Credits to [pinsvin00](https://github.com/pinsvin00) for kicking off this feature earlier in [#68133](https://github.com/ClickHouse/ClickHouse/issues/68133). [#74477](https://github.com/ClickHouse/ClickHouse/pull/74477) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Added aggregation function `sequenceMatchEvents` which return timestamps of matched events for longest chain of events in pattern. [#72349](https://github.com/ClickHouse/ClickHouse/pull/72349) ([UnamedRus](https://github.com/UnamedRus)). +* Added function `arrayNormalizedGini`. [#72823](https://github.com/ClickHouse/ClickHouse/pull/72823) ([flynn](https://github.com/ucasfl)). +* Add minus operator support for `DateTime64`, to allow subtraction between `DateTime64` values, as well as `DateTime`. [#74482](https://github.com/ClickHouse/ClickHouse/pull/74482) ([Li Yin](https://github.com/liyinsg)). + +#### Experimental Features +* The `BFloat16` data type is production-ready. [#73840](https://github.com/ClickHouse/ClickHouse/pull/73840) ([Alexey Milovidov](https://github.com/alexey-milovidov)). + +#### Performance Improvement +* Optimized function `indexHint`. Now, columns that are only used as arguments of function `indexHint` are not read from the table. [#74314](https://github.com/ClickHouse/ClickHouse/pull/74314) ([Anton Popov](https://github.com/CurtizJ)). If the `indexHint` function is a central piece of your enterprise data architecture, this optimization will save your life. +* More accurate accounting for `max_joined_block_size_rows` setting for `parallel_hash` JOIN algorithm. Helps to avoid increased memory consumption compared to `hash` algorithm. [#74630](https://github.com/ClickHouse/ClickHouse/pull/74630) ([Nikita Taranov](https://github.com/nickitat)). +* Support predicate push down optimization on the query plan level for the `MergingAggregated` step. It improves performance for some queries with the analyzer. [#74073](https://github.com/ClickHouse/ClickHouse/pull/74073) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Splitting of left table blocks by hash was removed from the probe phase of the `parallel_hash` JOIN algorithm. [#73089](https://github.com/ClickHouse/ClickHouse/pull/73089) ([Nikita Taranov](https://github.com/nickitat)). +* Optimize RowBinary input format. Closes [#63805](https://github.com/ClickHouse/ClickHouse/issues/63805). [#65059](https://github.com/ClickHouse/ClickHouse/pull/65059) ([Pavel Kruglov](https://github.com/Avogar)). +* Write parts with level 1 if `optimize_on_insert` is enabled. It allows to use several optimizations of queries with `FINAL` for freshly written parts. [#73132](https://github.com/ClickHouse/ClickHouse/pull/73132) ([Anton Popov](https://github.com/CurtizJ)). +* Speedup string deserialization by some low-level optimisation. [#65948](https://github.com/ClickHouse/ClickHouse/pull/65948) ([Nikita Taranov](https://github.com/nickitat)). +* When running an equality comparison between records, such as during merges, start to compare rows from most likely unequal columns first. [#63780](https://github.com/ClickHouse/ClickHouse/pull/63780) ([UnamedRus](https://github.com/UnamedRus)). +* Improve grace hash join performance by re-ranking the right join table by keys. [#72237](https://github.com/ClickHouse/ClickHouse/pull/72237) ([kevinyhzou](https://github.com/KevinyhZou)). +* Allow `arrayROCAUC` and `arrayAUCPR` to compute partial area of the whole curve, so that its calculation can be parallelized over huge datasets. [#72904](https://github.com/ClickHouse/ClickHouse/pull/72904) ([Emmanuel](https://github.com/emmanuelsdias)). +* Avoid spawn too many idle threads. [#72920](https://github.com/ClickHouse/ClickHouse/pull/72920) ([Guo Wangyang](https://github.com/guowangy)). +* Don't list blob storage keys if we only have curly brackets expansion in table function. Closes [#73333](https://github.com/ClickHouse/ClickHouse/issues/73333). [#73518](https://github.com/ClickHouse/ClickHouse/pull/73518) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Short circuit optimization for functions executed over Nullable arguments. [#73820](https://github.com/ClickHouse/ClickHouse/pull/73820) ([李扬](https://github.com/taiyang-li)). +* Do not apply `maskedExecute` on non-function columns, improve the performance of short circuit execution. [#73965](https://github.com/ClickHouse/ClickHouse/pull/73965) ([lgbo](https://github.com/lgbo-ustc)). +* Disable the autodetection of headers in input formats for `Kafka`/`NATS`/`RabbitMQ`/`FileLog` to improve performance. [#74006](https://github.com/ClickHouse/ClickHouse/pull/74006) ([Azat Khuzhin](https://github.com/azat)). +* Execute pipeline with a higher degree of parallelism after aggregation with grouping sets. [#74082](https://github.com/ClickHouse/ClickHouse/pull/74082) ([Nikita Taranov](https://github.com/nickitat)). +* Reduce critical section in `MergeTreeReadPool`. [#74202](https://github.com/ClickHouse/ClickHouse/pull/74202) ([Guo Wangyang](https://github.com/guowangy)). +* Parallel replicas performance improvement. Packets deserialization on query initiator, for packets not related to parallel replicas protocol, now always happens in pipeline thread. Before, it could happen in a thread responsible for pipeline scheduling, which could make initiator less responsive and delay pipeline execution. [#74398](https://github.com/ClickHouse/ClickHouse/pull/74398) ([Igor Nikonov](https://github.com/devcrafter)). +* Improve performance of larger multi requests in Keeper. [#74849](https://github.com/ClickHouse/ClickHouse/pull/74849) ([Antonio Andelic](https://github.com/antonio2368)). +* Use log wrappers by value and don't allocate them in a heap. [#74034](https://github.com/ClickHouse/ClickHouse/pull/74034) ([Mikhail Artemenko](https://github.com/Michicosun)). +* Reestablish connection to MySQL and Postgres dictionary replicas in the background, so it wouldn't delay requests to corresponding dictionaries. [#71101](https://github.com/ClickHouse/ClickHouse/pull/71101) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* Parallel replicas used historical information about replica availability to improve replica selection but did not update the replica's error count when the connection was unavailable. This PR updates the replica's error count when unavailable. [#72666](https://github.com/ClickHouse/ClickHouse/pull/72666) ([zoomxi](https://github.com/zoomxi)). +* Added a merge tree setting `materialize_skip_indexes_on_merge` which suppresses the creation of skip indexes during merge. This allows users to control explicitly (via `ALTER TABLE [..] MATERIALIZE INDEX [...]`) when skip indexes are created. This can be useful if skip indexes are expensive to build (e.g. vector similarity indexes). [#74401](https://github.com/ClickHouse/ClickHouse/pull/74401) ([Robert Schulze](https://github.com/rschu1ze)). +* Optimize keeper requests in Storage(S3/Azure)Queue. [#74410](https://github.com/ClickHouse/ClickHouse/pull/74410) ([Kseniia Sumarokova](https://github.com/kssenii)). [#74538](https://github.com/ClickHouse/ClickHouse/pull/74538) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Use up to `1000` parallel replicas by default. [#74504](https://github.com/ClickHouse/ClickHouse/pull/74504) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Improve HTTP session reuse when reading from s3 disk ([#72401](https://github.com/ClickHouse/ClickHouse/issues/72401)). [#74548](https://github.com/ClickHouse/ClickHouse/pull/74548) ([Julian Maicher](https://github.com/jmaicher)). + +#### Improvement +* Support SETTINGS in a CREATE TABLE query with an implicit ENGINE and support mixing engine and query settings. [#73120](https://github.com/ClickHouse/ClickHouse/pull/73120) ([Raúl Marín](https://github.com/Algunenano)). +* Enable `use_hive_partitioning` by default. [#71636](https://github.com/ClickHouse/ClickHouse/pull/71636) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Support CAST and ALTER between JSON types with different parameters. [#72303](https://github.com/ClickHouse/ClickHouse/pull/72303) ([Pavel Kruglov](https://github.com/Avogar)). +* Support equal comparison for values of JSON column. [#72991](https://github.com/ClickHouse/ClickHouse/pull/72991) ([Pavel Kruglov](https://github.com/Avogar)). +* Improve formatting of identifiers with JSON subcolumns to avoid unnecessary back quotes. [#73085](https://github.com/ClickHouse/ClickHouse/pull/73085) ([Pavel Kruglov](https://github.com/Avogar)). +* Interactive metrics improvements. Fix metrics from parallel replicas not being fully displayed. Display the metrics in order of the most recent update, then lexicographically by name. Do not display stale metrics. [#71631](https://github.com/ClickHouse/ClickHouse/pull/71631) ([Julia Kartseva](https://github.com/jkartseva)). +* Make JSON output format pretty by default. Add new setting `output_format_json_pretty_print` to control it and enable it by default. [#72148](https://github.com/ClickHouse/ClickHouse/pull/72148) ([Pavel Kruglov](https://github.com/Avogar)). +* Allow `LowCardinality(UUID)` by default. This has proven practical among ClickHouse Cloud customers. [#73826](https://github.com/ClickHouse/ClickHouse/pull/73826) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Better message during installation. [#73827](https://github.com/ClickHouse/ClickHouse/pull/73827) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Better message about password reset for ClickHouse Cloud. [#73831](https://github.com/ClickHouse/ClickHouse/pull/73831) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Improve the error message with a File table that cannot perform appends into a file. [#73832](https://github.com/ClickHouse/ClickHouse/pull/73832) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Ask when a user accidentally requests to output binary format (such as Native, Parquet, Avro) in the terminal. This closes [#59524](https://github.com/ClickHouse/ClickHouse/issues/59524). [#73833](https://github.com/ClickHouse/ClickHouse/pull/73833) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Highlight trailing spaces in Pretty and Vertical formats in the terminal for better clarity. This is controlled with the `output_format_pretty_highlight_trailing_spaces` setting. Initial implementation by [Braden Burns](https://github.com/bradenburns) from [#72996](https://github.com/ClickHouse/ClickHouse/issues/72996). Closes [#71590](https://github.com/ClickHouse/ClickHouse/issues/71590). [#73847](https://github.com/ClickHouse/ClickHouse/pull/73847) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* `clickhouse-client` and `clickhouse-local` will autodetect compression of stdin when it is redirected from a file. This closes [#70865](https://github.com/ClickHouse/ClickHouse/issues/70865). [#73848](https://github.com/ClickHouse/ClickHouse/pull/73848) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Cut too long column names in pretty formats by default. This is controlled by the `output_format_pretty_max_column_name_width_cut_to` and `output_format_pretty_max_column_name_width_min_chars_to_cut` settings. This is the continuation of the work of [tanmaydatta](https://github.com/tanmaydatta) in [#66502](https://github.com/ClickHouse/ClickHouse/issues/66502). This closes [#65968](https://github.com/ClickHouse/ClickHouse/issues/65968). [#73851](https://github.com/ClickHouse/ClickHouse/pull/73851) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Make `Pretty` formats prettier: squash blocks if not much time passed since the output of the previous block. This is controlled by new settings `output_format_pretty_squash_consecutive_ms` (50 ms by default) and `output_format_pretty_squash_max_wait_ms` (1000 ms by default). Continuation of [#49537](https://github.com/ClickHouse/ClickHouse/issues/49537). This closes [#49153](https://github.com/ClickHouse/ClickHouse/issues/49153). [#73852](https://github.com/ClickHouse/ClickHouse/pull/73852) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Add a metric on the number of currently merging source parts. This closes [#70809](https://github.com/ClickHouse/ClickHouse/issues/70809). [#73868](https://github.com/ClickHouse/ClickHouse/pull/73868) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Highlight columns in the `Vertical` format if the output is to a terminal. This can be disabled with the `output_format_pretty_color` setting. [#73898](https://github.com/ClickHouse/ClickHouse/pull/73898) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Enhanced the MySQL compatibility to a level that now, `mysqlsh` (a rich MySQL CLI from Oracle) can connect to ClickHouse. This is needed to facilitate testing. [#73912](https://github.com/ClickHouse/ClickHouse/pull/73912) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Pretty formats can render multi-line fields inside a table cell, which improves readability. This is enabled by default and can be controlled by the setting `output_format_pretty_multiline_fields`. Continuation of the work by [Volodyachan](https://github.com/Volodyachan) in [#64094](https://github.com/ClickHouse/ClickHouse/issues/64094). This closes [#56912](https://github.com/ClickHouse/ClickHouse/issues/56912). [#74032](https://github.com/ClickHouse/ClickHouse/pull/74032) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Expose X-ClickHouse HTTP headers to JavaScript in the browser. It makes writing applications more convenient. [#74180](https://github.com/ClickHouse/ClickHouse/pull/74180) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* The `JSONEachRowWithProgress` format will include events with metadata, as well as totals and extremes. It also includes `rows_before_limit_at_least` and `rows_before_aggregation`. The format prints the exception properly if it arrives after partial results. The progress now includes elapsed nanoseconds. One final progress event is emitted at the end. The progress during query runtime will be printed no more frequently than the value of the `interactive_delay` setting. [#74181](https://github.com/ClickHouse/ClickHouse/pull/74181) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Hourglass will rotate smoothly in Play UI. [#74182](https://github.com/ClickHouse/ClickHouse/pull/74182) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Even if the HTTP response is compressed, send packets as soon as they arrive. This allows the browser to receive progress packets and compressed data. [#74201](https://github.com/ClickHouse/ClickHouse/pull/74201) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* If the number of output records is larger than N = `output_format_pretty_max_rows`, instead of displaying only the first N rows, we will cut the output table in the middle, displaying N/2 first rows and N/2 last rows. Continuation of [#64200](https://github.com/ClickHouse/ClickHouse/issues/64200). This closes [#59502](https://github.com/ClickHouse/ClickHouse/issues/59502). [#73929](https://github.com/ClickHouse/ClickHouse/pull/73929) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Allow more general join planning algorithm when hash join algorithm is enabled. [#71926](https://github.com/ClickHouse/ClickHouse/pull/71926) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). +* Allow to create bloom_filter index on columns with data type `DateTime64`. [#66416](https://github.com/ClickHouse/ClickHouse/pull/66416) ([Yutong Xiao](https://github.com/YutSean)). +* When `min_age_to_force_merge_seconds` and `min_age_to_force_merge_on_partition_only` are both enabled, the part merging will ignore the max bytes limit. [#73656](https://github.com/ClickHouse/ClickHouse/pull/73656) ([Kai Zhu](https://github.com/nauu)). +* Added HTTP headers to OpenTelemetry span logs table for enhanced traceability. [#70516](https://github.com/ClickHouse/ClickHouse/pull/70516) ([jonymohajanGmail](https://github.com/jonymohajanGmail)). +* Support writing `orc` file by custom time zone, not always by the `GMT` time zone. [#70615](https://github.com/ClickHouse/ClickHouse/pull/70615) ([kevinyhzou](https://github.com/KevinyhZou)). +* Respect IO scheduling settings when writing backups across clouds. [#71093](https://github.com/ClickHouse/ClickHouse/pull/71093) ([János Benjamin Antal](https://github.com/antaljanosbenjamin)). +* Add `metric` column alias `name` to `system.asynchronous_metrics`. [#71164](https://github.com/ClickHouse/ClickHouse/pull/71164) ([megao](https://github.com/jetgm)). +* Historically for some reason, the query `ALTER TABLE MOVE PARTITION TO TABLE` checked `SELECT` and `ALTER DELETE` rights instead of dedicated `ALTER_MOVE_PARTITION`. This PR makes use of this access type. For compatibility, this permission is also will be granted implicitly if `SELECT` and `ALTER DELETE` are granted, but this behavior will be removed in future releases. Closes [#16403](https://github.com/ClickHouse/ClickHouse/issues/16403). [#71632](https://github.com/ClickHouse/ClickHouse/pull/71632) ([pufit](https://github.com/pufit)). +* Throw an exception when trying to materialize a column in the sort key instead of allowing it to break the sort order. [#71891](https://github.com/ClickHouse/ClickHouse/pull/71891) ([Peter Nguyen](https://github.com/petern48)). +* Hide secrets in `EXPLAIN QUERY TREE`. [#72025](https://github.com/ClickHouse/ClickHouse/pull/72025) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* Support parquet integer logical types in the "native" reader. [#72105](https://github.com/ClickHouse/ClickHouse/pull/72105) ([Arthur Passos](https://github.com/arthurpassos)). +* Interactively request credentials in the browser if the default user requires a password. In previous versions, the server returned HTTP 403; now, it returns HTTP 401. [#72198](https://github.com/ClickHouse/ClickHouse/pull/72198) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Convert access types `CREATE_USER`, `ALTER_USER`, `DROP_USER`, `CREATE_ROLE`, `ALTER_ROLE`, `DROP_ROLE` from global to parameterized. That means users can now grant access management grants more precise:. [#72246](https://github.com/ClickHouse/ClickHouse/pull/72246) ([pufit](https://github.com/pufit)). +* Add the `latest_fail_error_code_name` column to `system.mutations`. We need this column to introduce a new metric on stuck mutations and use it to build graphs of the errors encountered in the cloud as well as, optionally, adding a new less-noisy alert. [#72398](https://github.com/ClickHouse/ClickHouse/pull/72398) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). +* Reduce amount of allocation in the `ATTACH PARTITION` query. [#72583](https://github.com/ClickHouse/ClickHouse/pull/72583) ([Konstantin Morozov](https://github.com/k-morozov)). +* Make `max_bytes_before_external_sort` limit depends on total query memory consumption (previously it was number of bytes in the sorting block for one sorting thread, now it has the same meaning as `max_bytes_before_external_group_by` - it is total limit for the whole query memory for all threads). Also one more setting added to control on disk block size - `min_external_sort_block_bytes`. [#72598](https://github.com/ClickHouse/ClickHouse/pull/72598) ([Azat Khuzhin](https://github.com/azat)). +* Ignore memory restrictions by trace collector. [#72606](https://github.com/ClickHouse/ClickHouse/pull/72606) ([Azat Khuzhin](https://github.com/azat)). +* Add server settings `dictionaries_lazy_load` and `wait_dictionaries_load_at_startup` to `system.server_settings`. [#72664](https://github.com/ClickHouse/ClickHouse/pull/72664) ([Christoph Wurm](https://github.com/cwurm)). +* Adds setting `max_backup_bandwidth` to the list of settings that can be specified as part of `BACKUP`/`RESTORE` queries. [#72665](https://github.com/ClickHouse/ClickHouse/pull/72665) ([Christoph Wurm](https://github.com/cwurm)). +* Reducing the log level for appearing replicated parts in the ReplicatedMergeTree engine to help minimize the volume of logs generated in a replicated cluster. [#72876](https://github.com/ClickHouse/ClickHouse/pull/72876) ([mor-akamai](https://github.com/morkalfon)). +* Improve extraction of common expression in disjunctions. Allow simplifying the resulting filter expression even if there's no common subexpression for all the disjuncts. Continuation of [#71537](https://github.com/ClickHouse/ClickHouse/issues/71537). [#73271](https://github.com/ClickHouse/ClickHouse/pull/73271) ([Dmitry Novik](https://github.com/novikd)). +* In Storage `S3Queue`/`AzureQueue` allow to add settings where table was created without settings. [#73283](https://github.com/ClickHouse/ClickHouse/pull/73283) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Introduce a setting `least_greatest_legacy_null_behavior` (default: `false`) which controls if functions `least` and `greatest` handle `NULL` arguments by unconditionally returning `NULL` (if `true`) or by ignoring them (if `false`). [#73344](https://github.com/ClickHouse/ClickHouse/pull/73344) ([Robert Schulze](https://github.com/rschu1ze)). +* Use Keeper multi requests in the cleanup thread of ObjectStorageQueueMetadata. [#73357](https://github.com/ClickHouse/ClickHouse/pull/73357) ([Antonio Andelic](https://github.com/antonio2368)). +* When ClickHouse runs under a cgroup we will still collect system-wide asynchronous metrics related to system load, process scheduling, memory etc. They might provide useful signals when ClickHouse is the only process on the host with high resource consumption. [#73369](https://github.com/ClickHouse/ClickHouse/pull/73369) ([Nikita Taranov](https://github.com/nickitat)). +* In storage `S3Queue` allow to transfer old ordered tables created before 24.6 to new structure with buckets. [#73467](https://github.com/ClickHouse/ClickHouse/pull/73467) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Add `system.azure_queue` similar to existing `system.s3queue`. [#73477](https://github.com/ClickHouse/ClickHouse/pull/73477) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Function `parseDateTime64` (and its variants) now produces correct results for input dates before 1970 / after 2106. Example: `SELECT parseDateTime64InJodaSyntax('2200-01-01 00:00:00.000', 'yyyy-MM-dd HH:mm:ss.SSS')`. [#73594](https://github.com/ClickHouse/ClickHouse/pull/73594) ([zhanglistar](https://github.com/zhanglistar)). +* Address some `clickhouse-disks` usability issues addressed by users. Closes [#67136](https://github.com/ClickHouse/ClickHouse/issues/67136). [#73616](https://github.com/ClickHouse/ClickHouse/pull/73616) ([Daniil Ivanik](https://github.com/divanik)). +* Allow to alter commit settings in storage S3(Azure)Queue. (Commit settings are: `max_processed_files_before_commit`, `max_processed_rows_before_commit`, `max_processed_bytes_before_commit`, `max_processing_time_sec_before_commit`). [#73635](https://github.com/ClickHouse/ClickHouse/pull/73635) ([Kseniia Sumarokova](https://github.com/kssenii)). +* In storage S3(Azure)Queue aggregate progress between sources to compare with commit limit settings. [#73641](https://github.com/ClickHouse/ClickHouse/pull/73641) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Support core settings in `BACKUP`/`RESTORE` query. [#73650](https://github.com/ClickHouse/ClickHouse/pull/73650) ([Vitaly Baranov](https://github.com/vitlibar)). +* Take into account the `output_format_compression_level` on Parquet output. [#73651](https://github.com/ClickHouse/ClickHouse/pull/73651) ([Arthur Passos](https://github.com/arthurpassos)). +* Adds reading Apache Arrow's `fixed_size_list` as an `Array` instead of treating it as an unsupported type. [#73654](https://github.com/ClickHouse/ClickHouse/pull/73654) ([Julian Meyers](https://github.com/J-Meyers)). +* Add two backup engines: `Memory` (keeps backups inside the current user session), and `Null` (don't keep backups anywhere), which is for testing. [#73690](https://github.com/ClickHouse/ClickHouse/pull/73690) ([Vitaly Baranov](https://github.com/vitlibar)). +* `concurrent_threads_soft_limit_num` and `concurrent_threads_soft_limit_num_ratio_to_cores` could be changed w/o restart of a server. [#73713](https://github.com/ClickHouse/ClickHouse/pull/73713) ([Sergei Trifonov](https://github.com/serxa)). +* Add support for extended numeric types (`Decimal`, big integers) in `formatReadable` functions. [#73765](https://github.com/ClickHouse/ClickHouse/pull/73765) ([Raúl Marín](https://github.com/Algunenano)). +* Support TLS for Postgres wire protocol compatibility. [#73812](https://github.com/ClickHouse/ClickHouse/pull/73812) ([scanhex12](https://github.com/scanhex12)). +* The function `isIPv4String` returned true if the correct IPv4 address was followed by a zero byte, while it should return false in this case. Continuation of [#65387](https://github.com/ClickHouse/ClickHouse/issues/65387). [#73946](https://github.com/ClickHouse/ClickHouse/pull/73946) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Make the error code in the MySQL wire protocol compatible with MySQL. Continuation of [#56831](https://github.com/ClickHouse/ClickHouse/issues/56831). Closes [#50957](https://github.com/ClickHouse/ClickHouse/issues/50957). [#73948](https://github.com/ClickHouse/ClickHouse/pull/73948) ([Alexey Milovidov](https://github.com/alexey-milovidov)). +* Add setting `validate_enum_literals_in_opearators` to validate enum literals in operators like `IN`, `NOT IN` against the enum type and throw an exception if the literal is not a valid enum value. [#73985](https://github.com/ClickHouse/ClickHouse/pull/73985) ([Vladimir Cherkasov](https://github.com/vdimir)). +* In Storage `S3(Azure)Queue` commit all files (in a single butch defined by commit settings) in a single keeper transaction. [#73991](https://github.com/ClickHouse/ClickHouse/pull/73991) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Disable header detection for executable UDFs and dictionaries (could lead to Function 'X': wrong result, expected Y row(s), actual Y-1). [#73992](https://github.com/ClickHouse/ClickHouse/pull/73992) ([Azat Khuzhin](https://github.com/azat)). +* Add the `distributed` option for `EXPLAIN PLAN.` Now, `EXPLAIN distributed=1 ... ` appends remote plan to `ReadFromParallelRemote*` steps. [#73994](https://github.com/ClickHouse/ClickHouse/pull/73994) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Use correct return type for not/xor with Dynamic arguments. [#74013](https://github.com/ClickHouse/ClickHouse/pull/74013) ([Pavel Kruglov](https://github.com/Avogar)). +* Allow changing `add_implicit_sign_column_constraint_for_collapsing_engine` after table creation. [#74014](https://github.com/ClickHouse/ClickHouse/pull/74014) ([Christoph Wurm](https://github.com/cwurm)). +* Support subcolumns in materialized view select query. [#74030](https://github.com/ClickHouse/ClickHouse/pull/74030) ([Pavel Kruglov](https://github.com/Avogar)). +* There are now three simple ways to set a custom prompt in `clickhouse-client`: 1. via command-line parameter `--prompt`, 2. in the configuration file, via settings `[...]`, and 3. also in the configuration file, via per-connection settings `[...]`. [#74168](https://github.com/ClickHouse/ClickHouse/pull/74168) ([Christoph Wurm](https://github.com/cwurm)). +* Autodetect secure connection based on connecting to port 9440 in ClickHouse Client. [#74212](https://github.com/ClickHouse/ClickHouse/pull/74212) ([Christoph Wurm](https://github.com/cwurm)). +* Authenticate users with username only for http_handlers (previously it requires user to put the password as well). [#74221](https://github.com/ClickHouse/ClickHouse/pull/74221) ([Azat Khuzhin](https://github.com/azat)). +* Support for the alternative query languages PRQL and KQL was marked experimental. To use them, specify settings `allow_experimental_prql_dialect = 1` and `allow_experimental_kusto_dialect = 1`. [#74224](https://github.com/ClickHouse/ClickHouse/pull/74224) ([Robert Schulze](https://github.com/rschu1ze)). +* Support returning the default Enum type in more aggregate functions. [#74272](https://github.com/ClickHouse/ClickHouse/pull/74272) ([Raúl Marín](https://github.com/Algunenano)). +* In `OPTIMIZE TABLE`, it is now possible to specify keyword `FORCE` as an alternative to existing keyword `FINAL`. [#74342](https://github.com/ClickHouse/ClickHouse/pull/74342) ([Robert Schulze](https://github.com/rschu1ze)). +* Add the `IsServerShuttingDown` metric, which is needed to trigger an alert when the server shutdown takes too much time. [#74429](https://github.com/ClickHouse/ClickHouse/pull/74429) ([Miсhael Stetsyuk](https://github.com/mstetsyuk)). +* Added Iceberg tables names to EXPLAIN. [#74485](https://github.com/ClickHouse/ClickHouse/pull/74485) ([alekseev-maksim](https://github.com/alekseev-maksim)). +* Provide a better error message when using RECURSIVE CTE with the old analyzer. [#74523](https://github.com/ClickHouse/ClickHouse/pull/74523) ([Raúl Marín](https://github.com/Algunenano)). +* Show extended error messages in `system.errors`. [#74574](https://github.com/ClickHouse/ClickHouse/pull/74574) ([Vitaly Baranov](https://github.com/vitlibar)). +* Allow to use password for client communication with clickhouse-keeper. This feature is not very useful if you specify proper SSL configuration for server and client, but still can be useful for some cases. Password cannot be longer than 16 characters. It's not connected with Keeper Auth model. [#74673](https://github.com/ClickHouse/ClickHouse/pull/74673) ([alesapin](https://github.com/alesapin)). +* Add error code for config reloader. [#74746](https://github.com/ClickHouse/ClickHouse/pull/74746) ([Garrett Thomas](https://github.com/garrettthomaskth)). +* Added support for IPv6 addresses in MySQL and PostgreSQL table functions and engines. [#74796](https://github.com/ClickHouse/ClickHouse/pull/74796) ([Mikhail Koviazin](https://github.com/mkmkme)). +* Implement short circuit optimization for `divideDecimal`. Fixes [#74280](https://github.com/ClickHouse/ClickHouse/issues/74280). [#74843](https://github.com/ClickHouse/ClickHouse/pull/74843) ([Kevin Mingtarja](https://github.com/kevinmingtarja)). +* Now users can be specified inside the startup scripts. [#74894](https://github.com/ClickHouse/ClickHouse/pull/74894) ([pufit](https://github.com/pufit)). +* Add support for Azure SAS Tokens. [#72959](https://github.com/ClickHouse/ClickHouse/pull/72959) ([Azat Khuzhin](https://github.com/azat)). + +#### Bug Fix (user-visible misbehavior in an official stable release) +* Set parquet compression level only if compression codec supports it. [#74659](https://github.com/ClickHouse/ClickHouse/pull/74659) ([Arthur Passos](https://github.com/arthurpassos)). +* Fixed a regression that using collation locales with modifiers throws an error. As an example, `SELECT arrayJoin(['kk 50', 'KK 01', ' KK 2', ' KK 3', 'kk 1', 'x9y99', 'x9y100']) item ORDER BY item ASC COLLATE 'tr-u-kn-true-ka-shifted` now works. [#73544](https://github.com/ClickHouse/ClickHouse/pull/73544) ([Robert Schulze](https://github.com/rschu1ze)). +* Fix cannot create SEQUENTIAL node with keeper-client. [#64177](https://github.com/ClickHouse/ClickHouse/pull/64177) ([Duc Canh Le](https://github.com/canhld94)). +* Fix incorrect character counting in the position functions. [#71003](https://github.com/ClickHouse/ClickHouse/pull/71003) ([思维](https://github.com/heymind)). +* `RESTORE` operations for access entities required more permission than necessary because of unhandled partial revokes. This PR fixes the issue. Closes [#71853](https://github.com/ClickHouse/ClickHouse/issues/71853). [#71958](https://github.com/ClickHouse/ClickHouse/pull/71958) ([pufit](https://github.com/pufit)). +* Avoid pause after `ALTER TABLE REPLACE/MOVE PARTITION FROM/TO TABLE`. Retrieve correct settings for background task scheduling. [#72024](https://github.com/ClickHouse/ClickHouse/pull/72024) ([Aleksei Filatov](https://github.com/aalexfvk)). +* Fix handling of empty tuples in some input and output formats (e.g. Parquet, Arrow). [#72616](https://github.com/ClickHouse/ClickHouse/pull/72616) ([Michael Kolupaev](https://github.com/al13n321)). +* Column-level GRANT SELECT/INSERT statements on wildcard databases/tables now throw an error. [#72646](https://github.com/ClickHouse/ClickHouse/pull/72646) ([Johann Gan](https://github.com/johanngan)). +* Fix the situation when a user can't run `REVOKE ALL ON *.*` because of implicit grants in the target access entity. [#72872](https://github.com/ClickHouse/ClickHouse/pull/72872) ([pufit](https://github.com/pufit)). +* Fix positive timezone formatting of formatDateTime scalar function. [#73091](https://github.com/ClickHouse/ClickHouse/pull/73091) ([ollidraese](https://github.com/ollidraese)). +* Fix to correctly reflect source port when connection made through PROXYv1 and `auth_use_forwarded_address` is set - previously proxy port was incorrectly used. Add `currentQueryID()` function. [#73095](https://github.com/ClickHouse/ClickHouse/pull/73095) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* Propagate format settings to NativeWriter in TCPHandler, so settings like `output_format_native_write_json_as_string` are applied correctly. [#73179](https://github.com/ClickHouse/ClickHouse/pull/73179) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix a crash in StorageObjectStorageQueue. [#73274](https://github.com/ClickHouse/ClickHouse/pull/73274) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix rare crash in refreshable materialized view during server shutdown. [#73323](https://github.com/ClickHouse/ClickHouse/pull/73323) ([Michael Kolupaev](https://github.com/al13n321)). +* The `%f` placeholder of function `formatDateTime` now unconditionally generates six (sub-second) digits. This makes the behavior compatible with MySQL `DATE_FORMAT` function. The previous behavior can be restored using setting `formatdatetime_f_prints_scale_number_of_digits = 1`. [#73324](https://github.com/ClickHouse/ClickHouse/pull/73324) ([ollidraese](https://github.com/ollidraese)). +* Fixed filtering by `_etag` column while reading from `s3` storage and table function. [#73353](https://github.com/ClickHouse/ClickHouse/pull/73353) ([Anton Popov](https://github.com/CurtizJ)). +* Fix `Not-ready Set is passed as the second argument for function 'in'` error when `IN (subquery)` is used in `JOIN ON` expression, with the old analyzer. [#73382](https://github.com/ClickHouse/ClickHouse/pull/73382) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fix preparing for squashin for Dynamic and JSON columns. Previously in some cases new types could be inserted into shared variant/shared data even when the limit on types/paths is not reached. [#73388](https://github.com/ClickHouse/ClickHouse/pull/73388) ([Pavel Kruglov](https://github.com/Avogar)). +* Check for corrupted sizes during types binary decoding to avoid too big allocations. [#73390](https://github.com/ClickHouse/ClickHouse/pull/73390) ([Pavel Kruglov](https://github.com/Avogar)). +* Fixed a logical error when reading from single-replica cluster with parallel replicas enabled. [#73403](https://github.com/ClickHouse/ClickHouse/pull/73403) ([Michael Kolupaev](https://github.com/al13n321)). +* Fix ObjectStorageQueue with ZooKeeper and older Keeper. [#73420](https://github.com/ClickHouse/ClickHouse/pull/73420) ([Antonio Andelic](https://github.com/antonio2368)). +* Implements fix, needed to enable hive partitioning by default. [#73479](https://github.com/ClickHouse/ClickHouse/pull/73479) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Fix data race when creating vector similarity index. [#73517](https://github.com/ClickHouse/ClickHouse/pull/73517) ([Antonio Andelic](https://github.com/antonio2368)). +* Fixes segfault when the source of the dictionary contains a function with wrong data. [#73535](https://github.com/ClickHouse/ClickHouse/pull/73535) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Fix retries on failed insert in storage S3(Azure)Queue. Closes [#70951](https://github.com/ClickHouse/ClickHouse/issues/70951). [#73546](https://github.com/ClickHouse/ClickHouse/pull/73546) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fixed error in function `tupleElement` which may appear in some cases for tuples with `LowCardinality` elements and enabled setting `optimize_functions_to_subcolumns`. [#73548](https://github.com/ClickHouse/ClickHouse/pull/73548) ([Anton Popov](https://github.com/CurtizJ)). +* Fix parsing enum glob followed by range one. Fixes [#73473](https://github.com/ClickHouse/ClickHouse/issues/73473). [#73569](https://github.com/ClickHouse/ClickHouse/pull/73569) ([Konstantin Bogdanov](https://github.com/thevar1able)). +* Fixed parallel_replicas_for_non_replicated_merge_tree being ignored in subqueries for non-replicated tables. [#73584](https://github.com/ClickHouse/ClickHouse/pull/73584) ([Igor Nikonov](https://github.com/devcrafter)). +* Fix for std::logical_error thrown when task cannot be scheduled. Found in stress tests. [#73629](https://github.com/ClickHouse/ClickHouse/pull/73629) ([Alexander Gololobov](https://github.com/davenger)). +* Do not interpret queries in `EXPLAIN SYNTAX` to avoid logical errors with incorrect processing stage for distributed queries. Fixes [#65205](https://github.com/ClickHouse/ClickHouse/issues/65205). [#73634](https://github.com/ClickHouse/ClickHouse/pull/73634) ([Dmitry Novik](https://github.com/novikd)). +* Fix possible data inconsistency in Dynamic column. Fixes possible logical error `Nested columns sizes are inconsistent with local_discriminators column size`. [#73644](https://github.com/ClickHouse/ClickHouse/pull/73644) ([Pavel Kruglov](https://github.com/Avogar)). +* Fixed `NOT_FOUND_COLUMN_IN_BLOCK` in queries with `FINAL` and `SAMPLE`. Fixed incorrect result in selects with `FINAL` from `CollapsingMergeTree` and enabled optimizations of `FINAL` . [#73682](https://github.com/ClickHouse/ClickHouse/pull/73682) ([Anton Popov](https://github.com/CurtizJ)). +* Fix crash in LIMIT BY COLUMNS. [#73686](https://github.com/ClickHouse/ClickHouse/pull/73686) ([Raúl Marín](https://github.com/Algunenano)). +* Fix the bug when the normal projection is forced to use, and query is exactly the same as the projection defined, but the projection is not selected and thus error is prompted. [#73700](https://github.com/ClickHouse/ClickHouse/pull/73700) ([Shichao Jin](https://github.com/jsc0218)). +* Fix deserialization of Dynamic/Object structure. It could lead to CANNOT_READ_ALL_DATA exceptions. [#73767](https://github.com/ClickHouse/ClickHouse/pull/73767) ([Pavel Kruglov](https://github.com/Avogar)). +* Skip `metadata_version.txt` in while restoring parts from a backup. [#73768](https://github.com/ClickHouse/ClickHouse/pull/73768) ([Vitaly Baranov](https://github.com/vitlibar)). +* Fix segmentation fault when Casting to Enum with LIKE. [#73775](https://github.com/ClickHouse/ClickHouse/pull/73775) ([zhanglistar](https://github.com/zhanglistar)). +* Fix for S3 Express bucket not working as disk. [#73777](https://github.com/ClickHouse/ClickHouse/pull/73777) ([Sameer Tamsekar](https://github.com/stamsekar)). +* Allow merging of rows with invalid sign column values in CollapsingMergeTree tables. [#73864](https://github.com/ClickHouse/ClickHouse/pull/73864) ([Christoph Wurm](https://github.com/cwurm)). +* Fix getting error when querying ddl with offline replica. [#73876](https://github.com/ClickHouse/ClickHouse/pull/73876) ([Tuan Pham Anh](https://github.com/tuanpach)). +* Fixes occasional failure to compare `map()` types due to possibility to create `Map` lacking explicit naming ('keys','values') of its nested tuple. [#73878](https://github.com/ClickHouse/ClickHouse/pull/73878) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* Ignore window functions during GROUP BY ALL clause resolution. Fix [#73501](https://github.com/ClickHouse/ClickHouse/issues/73501). [#73916](https://github.com/ClickHouse/ClickHouse/pull/73916) ([Dmitry Novik](https://github.com/novikd)). +* Fix implicit privileges (worked as wildcard before). [#73932](https://github.com/ClickHouse/ClickHouse/pull/73932) ([Azat Khuzhin](https://github.com/azat)). +* Fix high memory usage during nested Maps creation. [#73982](https://github.com/ClickHouse/ClickHouse/pull/73982) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix parsing nested JSON with empty keys. [#73993](https://github.com/ClickHouse/ClickHouse/pull/73993) ([Pavel Kruglov](https://github.com/Avogar)). +* Fix: alias can be not added to the projection if it is referenced by another alias and selected in inverse order. [#74033](https://github.com/ClickHouse/ClickHouse/pull/74033) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* Ignore object not found errors for Azure during plain_rewritable disk initialization. [#74059](https://github.com/ClickHouse/ClickHouse/pull/74059) ([Julia Kartseva](https://github.com/jkartseva)). +* Fix behaviour of `any` and `anyLast` with enum types and empty table. [#74061](https://github.com/ClickHouse/ClickHouse/pull/74061) ([Joanna Hulboj](https://github.com/jh0x)). +* Fixes case when the user specifies keyword arguments in the kafka table engine. [#74064](https://github.com/ClickHouse/ClickHouse/pull/74064) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Fix altering Storage `S3Queue` settings with "s3queue_" prefix to without and vice versa. [#74075](https://github.com/ClickHouse/ClickHouse/pull/74075) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Add a setting `allow_push_predicate_ast_for_distributed_subqueries`. This adds AST-based predicate push-down for distributed queries with the analyzer. This is a temporary solution that we use until distributed queries with query plan serialization are supported. Closes [#66878](https://github.com/ClickHouse/ClickHouse/issues/66878) [#69472](https://github.com/ClickHouse/ClickHouse/issues/69472) [#65638](https://github.com/ClickHouse/ClickHouse/issues/65638) [#68030](https://github.com/ClickHouse/ClickHouse/issues/68030) [#73718](https://github.com/ClickHouse/ClickHouse/issues/73718). [#74085](https://github.com/ClickHouse/ClickHouse/pull/74085) ([Nikolai Kochetov](https://github.com/KochetovNicolai)). +* Fixes issue when after [#73095](https://github.com/ClickHouse/ClickHouse/issues/73095) port can be present in the forwarded_for field, which leads to inability to resolve host name with port included. [#74116](https://github.com/ClickHouse/ClickHouse/pull/74116) ([Yakov Olkhovskiy](https://github.com/yakov-olkhovskiy)). +* Fixed incorrect formatting of `ALTER TABLE (DROP STATISTICS ...) (DROP STATISTICS ...)`. [#74126](https://github.com/ClickHouse/ClickHouse/pull/74126) ([Han Fei](https://github.com/hanfei1991)). +* Fix for issue [#66112](https://github.com/ClickHouse/ClickHouse/issues/66112). [#74128](https://github.com/ClickHouse/ClickHouse/pull/74128) ([Anton Ivashkin](https://github.com/ianton-ru)). +* It is no longer possible to use `Loop` as a table engine in `CREATE TABLE`. This combination was previously causing segfaults. [#74137](https://github.com/ClickHouse/ClickHouse/pull/74137) ([Yarik Briukhovetskyi](https://github.com/yariks5s)). +* Fix security issue to prevent SQL injection in postgresql and sqlite table functions. [#74144](https://github.com/ClickHouse/ClickHouse/pull/74144) ([Pablo Marcos](https://github.com/pamarcos)). +* Fix crash when reading a subcolumn from the compressed Memory engine table. Fixes [#74009](https://github.com/ClickHouse/ClickHouse/issues/74009). [#74161](https://github.com/ClickHouse/ClickHouse/pull/74161) ([Nikita Taranov](https://github.com/nickitat)). +* Fixed an infinite loop occurring with queries to the system.detached_tables. [#74190](https://github.com/ClickHouse/ClickHouse/pull/74190) ([Konstantin Morozov](https://github.com/k-morozov)). +* Fix logical error in s3queue during setting file as failed. [#74216](https://github.com/ClickHouse/ClickHouse/pull/74216) ([Kseniia Sumarokova](https://github.com/kssenii)). +* Fix native copy settings (`allow_s3_native_copy`/`allow_azure_native_copy`) for `RESTORE` from base backup. [#74286](https://github.com/ClickHouse/ClickHouse/pull/74286) ([Azat Khuzhin](https://github.com/azat)). +* Fixed the issue when the number of detached tables in the database is a multiple of max_block_size. [#74289](https://github.com/ClickHouse/ClickHouse/pull/74289) ([Konstantin Morozov](https://github.com/k-morozov)). +* Fix copying via ObjectStorage (i.e. S3) when source and destination credentials differs. [#74331](https://github.com/ClickHouse/ClickHouse/pull/74331) ([Azat Khuzhin](https://github.com/azat)). +* Fix detection of "use the Rewrite method in the JSON API" for native copy on GCS. [#74338](https://github.com/ClickHouse/ClickHouse/pull/74338) ([Azat Khuzhin](https://github.com/azat)). +* Fix incorrect calculation of `BackgroundMergesAndMutationsPoolSize` (it was x2 from real value). [#74509](https://github.com/ClickHouse/ClickHouse/pull/74509) ([alesapin](https://github.com/alesapin)). +* Fix the bug of leaking keeper watches when enable Cluster Discovery. [#74521](https://github.com/ClickHouse/ClickHouse/pull/74521) ([RinChanNOW](https://github.com/RinChanNOWWW)). +* Fix mem alignment issue reported by UBSan [#74512](https://github.com/ClickHouse/ClickHouse/issues/74512). [#74534](https://github.com/ClickHouse/ClickHouse/pull/74534) ([Arthur Passos](https://github.com/arthurpassos)). +* Fix KeeperMap concurrent cleanup during table creation. [#74568](https://github.com/ClickHouse/ClickHouse/pull/74568) ([Antonio Andelic](https://github.com/antonio2368)). +* Do not remove unused projection columns in subqueries in the presence of `EXCEPT` or `INTERSECT` to preserve the correct query result. Fixes [#73930](https://github.com/ClickHouse/ClickHouse/issues/73930). Fixes [#66465](https://github.com/ClickHouse/ClickHouse/issues/66465). [#74577](https://github.com/ClickHouse/ClickHouse/pull/74577) ([Dmitry Novik](https://github.com/novikd)). +* Fixed `INSERT SELECT` queries between tables with `Tuple` columns and enabled sparse serialization. [#74698](https://github.com/ClickHouse/ClickHouse/pull/74698) ([Anton Popov](https://github.com/CurtizJ)). +* Function `right` works incorrectly for const negative offset. [#74701](https://github.com/ClickHouse/ClickHouse/pull/74701) ([Daniil Ivanik](https://github.com/divanik)). +* Fix insertion of gzip-ed data sometimes fails due to flawed decompression on client side. [#74707](https://github.com/ClickHouse/ClickHouse/pull/74707) ([siyuan](https://github.com/linkwk7)). +* Partial revokes with wildcard grants could remove more privileges than expected. Closes [#74263](https://github.com/ClickHouse/ClickHouse/issues/74263). [#74751](https://github.com/ClickHouse/ClickHouse/pull/74751) ([pufit](https://github.com/pufit)). +* Keeper fix: fix reading log entries from disk. [#74785](https://github.com/ClickHouse/ClickHouse/pull/74785) ([Antonio Andelic](https://github.com/antonio2368)). +* Fixed checking grants for SYSTEM REFRESH/START/STOP VIEW, now it's not required to have this grant on `*.*` to execute a query for a specific view, only grant for this view are required. [#74789](https://github.com/ClickHouse/ClickHouse/pull/74789) ([Alexander Tokmakov](https://github.com/tavplubix)). +* The `hasColumnInTable` function doesn't account for alias columns. Fix it to also work for alias columns. [#74841](https://github.com/ClickHouse/ClickHouse/pull/74841) ([Bharat Nallan](https://github.com/bharatnc)). +* Fix FILE_DOESNT_EXIST error occurring during data parts merge for a table with an empty column in Azure Blob Storage. [#74892](https://github.com/ClickHouse/ClickHouse/pull/74892) ([Julia Kartseva](https://github.com/jkartseva)). +* Fix projection column name when joining temporary tables, close [#68872](https://github.com/ClickHouse/ClickHouse/issues/68872). [#74897](https://github.com/ClickHouse/ClickHouse/pull/74897) ([Vladimir Cherkasov](https://github.com/vdimir)). + +#### Build/Testing/Packaging Improvement +* The universal installation script will propose installation even on macOS. [#74339](https://github.com/ClickHouse/ClickHouse/pull/74339) ([Alexey Milovidov](https://github.com/alexey-milovidov)). diff --git a/scripts/aspell-dict-file.txt b/scripts/aspell-dict-file.txt index eb495712286..10d2f134699 100644 --- a/scripts/aspell-dict-file.txt +++ b/scripts/aspell-dict-file.txt @@ -1117,5 +1117,10 @@ ReceiveMessage RelationMessage --docs/integrations/data-ingestion/clickpipes/mongodb/faq.md-- resumable +--docs/cloud/guides/security/01_cloud_access_management/04_manage-database-users.md-- +keygen +keypair +--docs/cloud/guides/security/05_audit_logging/03_byoc-security-playbook.md-- +detections --docs/integrations/data-visualization/dot-and-clickhouse.md-- Hashboard diff --git a/scripts/aspell-ignore/en/aspell-dict.txt b/scripts/aspell-ignore/en/aspell-dict.txt index e8344540201..1f13e4ee5a0 100644 --- a/scripts/aspell-ignore/en/aspell-dict.txt +++ b/scripts/aspell-ignore/en/aspell-dict.txt @@ -218,6 +218,8 @@ CloudNotSupportedBadge CloudStorage Cloudera Cloudflare +CloudTrail +CloudWatch CodeBlock CodeLLDB Codecs @@ -369,6 +371,7 @@ Encodings Encrypter Enum Enums +Entra Eoan EphemeralNode EscapingRule @@ -850,6 +853,7 @@ ODBCDriver OFNS OLAP OLTP +OIDC OOMs ORCCompression OSContextSwitches @@ -1083,6 +1087,7 @@ ReferenceKeyed Refreshable RegexpTree RelationMessage +RelayState RemoteRead ReplacingMergeTree ReplacingReplicatedMergeTree diff --git a/scripts/sed_links.sh b/scripts/sed_links.sh index b11c041eb04..f7e48d96294 100755 --- a/scripts/sed_links.sh +++ b/scripts/sed_links.sh @@ -15,6 +15,10 @@ if [[ "$OSTYPE" == "darwin"* ]]; then sed -i '' 's|(/sql-reference/statements/select#replace)|(/sql-reference/statements/select)|g' docs/guides/developer/dynamic-column-selection.md sed -i '' 's|(/sql-reference/statements/select#except)|(/sql-reference/statements/select)|g' docs/guides/developer/dynamic-column-selection.md sed -i '' 's|(/cloud/reference/cloud-compatibility.md)|(/whats-new/cloud-compatibility)|g' docs/sql-reference/dictionaries/_snippet_dictionary_in_cloud.md + sed -i '' 's|(/cloud/security/secure-s3)|(/cloud/data-sources/secure-s3)|g' docs/engines/table-engines/integrations/s3queue.md + sed -i '' 's|(/cloud/security/cloud-access-management/overview#initial-settings)|(/cloud/security/console-roles)|g' docs/sql-reference/statements/grant.md + sed -i '' 's|(/cloud/security/secure-s3#access-your-s3-bucket-with-the-clickhouseaccess-role)|(/cloud/data-sources/secure-s3#access-your-s3-bucket-with-the-clickhouseaccess-role)|g' docs/sql-reference/table-functions/s3.md + sed -i '' 's|(/cloud/security/secure-s3#access-your-s3-bucket-with-the-clickhouseaccess-role)|(/cloud/data-sources/secure-s3#access-your-s3-bucket-with-the-clickhouseaccess-role)|g' docs/sql-reference/table-functions/s3Cluster.md sed -i '' 's||``|g' docs/operations/server-configuration-parameters/settings.md else # Linux @@ -24,5 +28,9 @@ else sed -i 's|(/sql-reference/statements/select#replace)|(/sql-reference/statements/select)|g' docs/guides/developer/dynamic-column-selection.md sed -i 's|(/sql-reference/statements/select#except)|(/sql-reference/statements/select)|g' docs/guides/developer/dynamic-column-selection.md sed -i 's|(/cloud/reference/cloud-compatibility.md)|(/whats-new/cloud-compatibility)|g' docs/sql-reference/dictionaries/_snippet_dictionary_in_cloud.md + sed -i 's|(/cloud/security/secure-s3)|(/cloud/data-sources/secure-s3)|g' docs/engines/table-engines/integrations/s3queue.md + sed -i 's|(/cloud/security/cloud-access-management/overview#initial-settings)|(/cloud/security/console-roles)|g' docs/sql-reference/statements/grant.md + sed -i 's|(/cloud/security/secure-s3#access-your-s3-bucket-with-the-clickhouseaccess-role)|(/cloud/data-sources/secure-s3#access-your-s3-bucket-with-the-clickhouseaccess-role)|g' docs/sql-reference/table-functions/s3.md + sed -i 's|(/cloud/security/secure-s3#access-your-s3-bucket-with-the-clickhouseaccess-role)|(/cloud/data-sources/secure-s3#access-your-s3-bucket-with-the-clickhouseaccess-role)|g' docs/sql-reference/table-functions/s3Cluster.md sed -i 's||``|g' docs/operations/server-configuration-parameters/settings.md fi diff --git a/static/images/cloud/reference/private-gov-architecture.png b/static/images/cloud/reference/private-gov-architecture.png new file mode 100644 index 00000000000..da7d09b401e Binary files /dev/null and b/static/images/cloud/reference/private-gov-architecture.png differ diff --git a/static/images/cloud/security/compliance/hipaa_1.png b/static/images/cloud/security/compliance/hipaa_1.png new file mode 100644 index 00000000000..7b7e9365a70 Binary files /dev/null and b/static/images/cloud/security/compliance/hipaa_1.png differ diff --git a/static/images/cloud/security/compliance/hipaa_2.png b/static/images/cloud/security/compliance/hipaa_2.png new file mode 100644 index 00000000000..fb5f3672e52 Binary files /dev/null and b/static/images/cloud/security/compliance/hipaa_2.png differ diff --git a/static/images/cloud/security/compliance/hipaa_3.png b/static/images/cloud/security/compliance/hipaa_3.png new file mode 100644 index 00000000000..e203baa038a Binary files /dev/null and b/static/images/cloud/security/compliance/hipaa_3.png differ diff --git a/static/images/cloud/security/compliance/hipaa_4.png b/static/images/cloud/security/compliance/hipaa_4.png new file mode 100644 index 00000000000..9ff1fd9ba2e Binary files /dev/null and b/static/images/cloud/security/compliance/hipaa_4.png differ diff --git a/static/images/cloud/security/compliance/pci_1.png b/static/images/cloud/security/compliance/pci_1.png new file mode 100644 index 00000000000..dd1453f05fd Binary files /dev/null and b/static/images/cloud/security/compliance/pci_1.png differ diff --git a/static/images/cloud/security/compliance/pci_2.png b/static/images/cloud/security/compliance/pci_2.png new file mode 100644 index 00000000000..9c9598dbdf5 Binary files /dev/null and b/static/images/cloud/security/compliance/pci_2.png differ diff --git a/static/images/cloud/security/compliance/pci_3.png b/static/images/cloud/security/compliance/pci_3.png new file mode 100644 index 00000000000..cd622b5c633 Binary files /dev/null and b/static/images/cloud/security/compliance/pci_3.png differ diff --git a/vercel.json b/vercel.json index e89a0bfb252..5baed255e27 100644 --- a/vercel.json +++ b/vercel.json @@ -3502,6 +3502,68 @@ "permanent": true }, { + "source": "/docs/cloud/reference/10_account-close.md", + "destination": "/docs/cloud/reference/11_account-close.md", + "permanent": true + }, + { + "source": "/docs/cloud/security/personal-data-access", + "destination": "/docs/cloud/manage/personal-data-access", + "permanent": true + }, + { + "source": "/docs/cloud/security/audit-logging", + "destination": "/docs/cloud/security/audit-logging/console-audit-log", + "permanent": true + }, + { + "source": "/docs/cloud/security/cloud-access-management/overview", + "destination": "/docs/cloud/security/manage-database-users", + "permanent": true + }, + { + "source": "/docs/cloud/guides/sql-console/config-sql-console-role-assignments", + "destination": "/docs/cloud/guides/sql-console/manage-sql-console-role-assignments", + "permanent": true + }, + { + "source": "/docs/cloud/security/secure-s3", + "destination": "/cloud/data-sources/secure-s3", + "permanent": true + }, + { + "source": "/docs/manage/security/cloud-endpoints-api", + "destination": "/docs/manage/data-sources/cloud-endpoints-api", + "permanent": true + }, + { + "source": "/docs/cloud/security/shared-responsibility-model", + "destination": "/docs/cloud/security/console-roles", + "permanent": true + }, + { + "source": "/docs/cloud/security/cloud-authentication", + "destination": "/docs/cloud/security#cloud-console-auth", + "permanent": true + }, + { + "source": "/docs/cloud/security/private-link-overview", + "destination": "/docs/cloud/security/connectivity/private-networking", + "permanent": true + }, + { + "source": "/docs/cloud/security/connectivity", + "destination": "/docs/cloud/security/connectivity/private-networking", + "permanent": true + }, + { + "source": "/docs/cloud/guides/sql-console/configure-org-service-role-assignments", + "destination": "/docs/cloud/security/cloud_access_management", + "permanent": true + }, + { + "source": "/docs/cloud/security/secure-s3", + "destination": "/docs/cloud/data-sources/secure-s3", "source": "/docs/operations/system-tables/crash-log", "destination": "/docs/operations/system-tables/crash_log", "permanent": true