ZeroBus - File Mode Prototype DAB

CODEOWNERS

@@ -11,6 +11,7 @@ conversational-agent-app   @vivian-xie-db @yuanchaoma-db
 database-diagram-builder   @alexott
 downstreams                @nfx @alexott
 feature-registry-app       @yang-chengg @mparkhe @mingyangge-db @stephanielu5
+filepush                   @chi-yang-db
 go-libs                    @nfx @alexott
 ip_access_list_analyzer    @alexott
 ka-chat-bot                @taiga-db
@@ -19,4 +20,4 @@ runtime-packages           @nfx @alexott
 sql_migration_copilot      @robertwhiffin
 tacklebox                  @Jonathan-Choi
 uc-catalog-cloning         @esiol-db @vasco-lopes
-.github                    @nfx @alexott @gueniai
+.github                    @nfx @alexott @gueniai

filepush/.gitignore

@@ -0,0 +1 @@
+.vscode

filepush/README.md

@@ -0,0 +1,179 @@
+---
+title: "Managed File Push"
+language: python
+author: "Chi Yang"
+date: 2025-08-07
+
+tags:
+- ingestion
+- file
+- nocode
+---
+
+# Managed File Push
+
+A lightweight, no‑code file ingestion workflow. Configure a set of tables, get a volume path for each, and drop files into those paths—your data lands in Unity Catalog tables via Auto Loader.
+
+## Table of Contents
+- [Quick Start](#quick-start)
+  - [Step 1. Configure tables](#step-1-configure-tables)
+  - [Step 2. Deploy & set up](#step-2-deploy--set-up)
+  - [Step 3. Retrieve endpoint & push files](#step-3-retrieve-endpoint--push-files)
+- [Debug Table Issues](#debug-table-issues)
+  - [Step 1. Configure tables to debug](#step-1-configure-tables-to-debug)
+  - [Step 2. Deploy & set up in dev mode](#step-2-deploy--set-up-in-dev-mode)
+  - [Step 3. Retrieve endpoint & push files to debug](#step-3-retrieve-endpoint--push-files-to-debug)
+  - [Step 4. Debug table configs](#step-4-debug-table-configs)
+  - [Step 5. Fix the table configs in production](#step-5-fix-the-table-configs-in-production)
+
+---
+
+## Quick Start
+
+### Step 1. Configure tables
+Define the catalog and a **new** schema name where the tables will land in `./dab/databricks.yml`:
+
+```yaml
+variables:
+  catalog_name:
+    description: The existing catalog where the NEW schema will be created.
+    default: main
+  schema_name:
+    description: The name of the NEW schema where the tables will be created.
+    default: filepushschema
+```
+
+Edit table configs in `./dab/src/configs/tables.json`. Only `name` and `format` are required.
+
+For supported `format_options`, see the [Auto Loader options](https://docs.databricks.com/aws/en/ingestion/cloud-object-storage/auto-loader/options). Not all options are supported here. If unsure, specify only `name` and `format`, or follow [Debug Table Issues](#debug-table-issues) to discover the correct options.
+
+```json
+[
+  {
+    "name": "table1",
+    "format": "csv",
+    "format_options": { "escape": "\"" },
+    "schema_hints": "id int, name string"
+  },
+  {
+    "name": "table2",
+    "format": "json"
+  }
+]
+```
+
+> **Tip:** Keep `schema_hints` minimal; Auto Loader can evolve the schema as new columns appear.
+
+### Step 2. Deploy & set up
+
+```bash
+cd dab
+databricks bundle deploy
+databricks bundle run configuration_job
+```
+
+Wait for the configuration job to finish before moving on.
+
+### Step 3. Retrieve endpoint & push files
+Fetch the volume path for uploading files to a specific table (example: `table1`):
+
+```bash
+databricks tables get main.filepushschema.table1 --output json \
+  | jq -r '.properties["filepush.table_volume_path_data"]'
+```
+
+Example output:
+
+```text
+/Volumes/main/filepushschema/main_filepushschema_filepush_volume/data/table1
+```
+
+Upload files to the path above using any of the [Volumes file APIs](https://docs.databricks.com/aws/en/volumes/volume-files#methods-for-managing-files-in-volumes).
+
+**REST API example**:
+
+```bash
+# prerequisites: export DATABRICKS_HOST and DATABRICKS_TOKEN (PAT token)
+curl -X PUT "$DATABRICKS_HOST/api/2.0/fs/files/Volumes/main/filepushschema/main_filepushschema_filepush_volume/data/table1/datafile1.csv" \
+  -H "Authorization: Bearer $DATABRICKS_TOKEN" \
+  -H "Content-Type: application/octet-stream" \
+  --data-binary @"/local/file/path/datafile1.csv"
+```
+
+**Databricks CLI example** (destination uses the `dbfs:` scheme):
+
+```bash
+databricks fs cp /local/file/path/datafile1.csv \
+  dbfs:/Volumes/main/filepushschema/main_filepushschema_filepush_volume/data/table1
+```
+
+Within about a minute, the data should appear in the table `main.filepushschema.table1`.
+
+---
+
+## Debug Table Issues
+If data isn’t parsed as expected, use **dev mode** to iterate on table options safely.
+
+### Step 1. Configure tables to debug
+Configure tables as in [Step 1 of Quick Start](#step-1-configure-tables).
+
+### Step 2. Deploy & set up in **dev mode**
+
+```bash
+cd dab
+databricks bundle deploy -t dev
+databricks bundle run configuration_job -t dev
+```
+
+Wait for the configuration job to finish. Example output:
+
+```text
+2025-09-23 22:03:04,938 [INFO] initialization - ==========
+catalog_name: main
+schema_name: dev_chi_yang_filepushschema
+volume_path_root: /Volumes/main/dev_chi_yang_filepushschema/main_filepushschema_filepush_volume
+volume_path_data: /Volumes/main/dev_chi_yang_filepushschema/main_filepushschema_filepush_volume/data
+volume_path_archive: /Volumes/main/dev_chi_yang_filepushschema/main_filepushschema_filepush_volume/archive
+==========
+```
+
+> **Note:** In **dev mode**, the schema name is **prefixed**. Use the printed schema name for the remaining steps.
+
+### Step 3. Retrieve endpoint & push files to debug
+Get the dev volume path (note the prefixed schema):
+
+```bash
+databricks tables get main.dev_chi_yang_filepushschema.table1 --output json \
+  | jq -r '.properties["filepush.table_volume_path_data"]'
+```
+
+Example output:
+
+```text
+/Volumes/main/dev_chi_yang_filepushschema/main_filepushschema_filepush_volume/data/table1
+```
+
+Then follow the upload instructions from [Quick Start → Step 3](#step-3-retrieve-endpoint--push-files) to send test files.
+
+### Step 4. Debug table configs
+Open the pipeline in the workspace:
+
+```bash
+databricks bundle open refresh_pipeline -t dev
+```
+
+Click **Edit pipeline** to launch the development UI. Open the `debug_table_config` notebook and follow its guidance to refine the table options. When satisfied, copy the final config back to `./dab/src/configs/tables.json`.
+
+### Step 5. Fix the table configs in production
+Redeploy the updated config and run a full refresh to correct existing data for an affected table:
+
+```bash
+cd dab
+databricks bundle deploy
+databricks bundle run refresh_pipeline --full-refresh table1
+```
+
+---
+
+**That’s it!** You now have a managed file‑push workflow with debuggable table configs and repeatable deployments.
+

filepush/dab/databricks.yml

@@ -0,0 +1,36 @@
+# databricks.yml
+# This is the configuration for the file push DAB dab.
+
+bundle:
+  name: dab
+
+include:
+  - resources/*.yml
+
+targets:
+  # The deployment targets. See https://docs.databricks.com/en/dev-tools/bundles/deployment-modes.html
+  dev:
+    mode: development
+    workspace:
+      host: https://e2-dogfood.staging.cloud.databricks.com
+
+  prod:
+    mode: production
+    default: true
+    workspace:
+      host: https://e2-dogfood.staging.cloud.databricks.com
+      root_path: /Workspace/Users/${workspace.current_user.userName}/.bundle/${bundle.name}/${bundle.target}
+    permissions:
+      - user_name: ${workspace.current_user.userName}
+        level: CAN_MANAGE
+
+variables:
+  catalog_name:
+    description: The existing catalog where the NEW schema will be created.
+    default: chi_catalog
+  schema_name:
+    description: The name of the NEW schema where the tables will be created.
+    default: filepushschema
+  resource_name_prefix:
+    description: The prefix for the resource names.
+    default: ${var.catalog_name}_${var.schema_name}_

filepush/dab/resources/job.yml

@@ -0,0 +1,46 @@
+# The main job for schema dab
+# This job will trigger in the schema pipeline
+
+resources:
+  jobs:
+    filetrigger_job:
+      name: ${var.resource_name_prefix}filetrigger_job
+      tasks:
+        - task_key: pipeline_refresh
+          pipeline_task:
+            pipeline_id: ${resources.pipelines.refresh_pipeline.id}
+      trigger:
+        file_arrival:
+          url: ${resources.volumes.filepush_volume.volume_path}/data/
+    configuration_job:
+      name: ${var.resource_name_prefix}configuration_job
+      tasks:
+        - task_key: initialization
+          spark_python_task:
+            python_file: ../src/utils/initialization.py
+            parameters:
+              - "--catalog_name"
+              - "{{job.parameters.catalog_name}}"
+              - "--schema_name"
+              - "{{job.parameters.schema_name}}"
+              - "--volume_path_root"
+              - "{{job.parameters.volume_path_root}}"
+              - "--logging_level"
+              - "${bundle.target}"
+          environment_key: serverless
+        - task_key: trigger_refresh
+          run_job_task:
+            job_id: ${resources.jobs.filetrigger_job.id}
+          depends_on:
+            - task_key: initialization
+      environments:
+        - environment_key: serverless
+          spec:
+            client: "3"
+      parameters:
+        - name: catalog_name
+          default: ${var.catalog_name}
+        - name: schema_name
+          default: ${resources.schemas.main_schema.name}
+        - name: volume_path_root
+          default: ${resources.volumes.filepush_volume.volume_path}

filepush/dab/resources/pipeline.yml

@@ -0,0 +1,15 @@
+# The table refresh pipeline for schema dab
+
+resources:
+  pipelines:
+    refresh_pipeline:
+      name: ${var.resource_name_prefix}refresh_pipeline
+      catalog: ${var.catalog_name}
+      schema: ${resources.schemas.main_schema.name}
+      serverless: true
+      libraries:
+        - file:
+            path: ../src/ingestion.py
+      root_path: ../src
+      configuration:
+        filepush.volume_path_root: ${resources.volumes.filepush_volume.volume_path}

filepush/dab/resources/schema.yml

@@ -0,0 +1,7 @@
+# The schema dab
+
+resources:
+  schemas:
+    main_schema:
+      name: ${var.schema_name}
+      catalog_name: ${var.catalog_name}

filepush/dab/resources/volume.yml

@@ -0,0 +1,8 @@
+# The file staging volume for schema dab
+
+resources:
+  volumes:
+    filepush_volume:
+      name: ${var.resource_name_prefix}filepush_volume
+      catalog_name: ${var.catalog_name}
+      schema_name: ${var.schema_name}

filepush/dab/src/configs/tables.json

@@ -0,0 +1,10 @@
+[
+  {
+    "name": "example_table",
+    "format": "csv",
+    "format_options": {
+      "escape": "\""
+    },
+    "schema_hints": "id int, name string"
+  }
+]

filepush/dab/src/debug_table_config.py

@@ -0,0 +1,63 @@
+# Databricks notebook source
+# MAGIC %md
+# MAGIC ## Paste the table config JSON you would like to debug from `./configs/tables.json` and assign to variable `table_config`
+# MAGIC For example,
+# MAGIC ```
+# MAGIC table_config = r'''
+# MAGIC {
+# MAGIC   "name": "all_employees",
+# MAGIC   "format": "csv",
+# MAGIC   "format_options": {
+# MAGIC     "escape": "\"",
+# MAGIC     "multiLine": "false"
+# MAGIC   }
+# MAGIC   "schema_hints": "id int, name string"
+# MAGIC }
+# MAGIC '''
+# MAGIC ```
+# MAGIC Only `name` and `format` are required for a table.
+
+# COMMAND ----------
+
+table_config = r'''
+  {
+    "name": "employees",
+    "format": "csv",
+    "format_options": {
+      "escape": "\""
+    },
+    "schema_hints": "id int, name string"
+  }
+'''
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC ## Click `Run all` and inspect the parsed result. Iterate on the config until the result looks good
+
+# COMMAND ----------
+
+import json
+import tempfile
+from utils import tablemanager
+from utils import envmanager
+
+if not envmanager.has_default_storage():
+  print("WARNING: Current catalog is not using default storage, some file push feature may not be available")
+
+# Load table config
+table_config_json = json.loads(table_config)
+tablemanager.validate_config(table_config_json)
+table_name = table_config_json["name"]
+table_volume_path_data = tablemanager.get_table_volume_path(table_name)
+
+assert tablemanager.has_data_file(table_name), f"No data file found in {table_volume_path_data}. Please upload at least 1 file to {table_volume_path_data}"
+
+# Put schema location in temp directory
+with tempfile.TemporaryDirectory() as tmpdir:
+  display(tablemanager.get_df_with_config(spark, table_config_json, tmpdir))
+
+# COMMAND ----------
+
+# MAGIC %md
+# MAGIC ## Copy and paste the modified config back to the `./configs/tables.json` in the DAB folder