Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,23 @@ endif::[]
* A Databricks workspace in the same region as your S3 bucket. See the https://docs.databricks.com/aws/en/resources/supported-regions#supported-regions-list[list of supported AWS regions^].
* Unity Catalog enabled in your Databricks workspace. See the https://docs.databricks.com/aws/en/data-governance/unity-catalog/get-started[Databricks documentation^] to set up Unity Catalog for your workspace.
* https://docs.databricks.com/aws/en/optimizations/predictive-optimization#enable-predictive-optimization[Predictive optimization^] enabled for Unity Catalog.
+
[NOTE]
====
When you enable predictive optimization, you must also set the following configurations in your Databricks workspace. These configurations allow predictive optimization to automatically generate column statistics and carry out background compaction for Iceberg tables:

```sql
SET spark.databricks.delta.liquid.lazyClustering.backfillStats=true;
SET spark.databricks.delta.computeStats.autoConflictResolution=true;

/*
After setting these configurations, you can optionally run OPTIMIZE to
immediately trigger compaction and liquid clustering, or let predictive
optimization handle it automatically later.
*/
OPTIMIZE `<catalog-name>`.redpanda.`<table-name>`;
```
====
* https://docs.databricks.com/aws/en/external-access/admin[External data access^] enabled in your metastore.
* Workspace admin privileges to complete the steps to create a Unity Catalog storage credential and external location that connects your cluster's Tiered Storage bucket to Databricks.

Expand Down Expand Up @@ -189,7 +206,7 @@ The following example shows how to query the Iceberg table using SQL in Databric
[,sql]
----
-- Ensure that the catalog and table name are correctly parsed in case they contain special characters
SELECT * FROM `<catalog-name>`.redpanda.`<table-name>`;
SELECT * FROM `<catalog-name>`.redpanda.`<table-name>` LIMIT 10;
----
+
Your query results should look like the following:
Expand Down