Querying Databricks Apache Iceberg Tables with the borderless Lakehouse

Enterprise organizations often leverage multiple data platforms, including Databricks, to execute heavy Apache Spark jobs alongside BigQuery to power highly concurrent analytics workloads. Historically, bridging these systems often meant building complex pipelines to copy data between environments. This approach increases storage costs, introduces latency, and fragments security controls.


With Google Cloud’s borderless Lakehouse, you can now query Databricks-managed Apache Iceberg tables straight from BigQuery. This zero-copy approach removes the need to duplicate data: BigQuery reads the same files Databricks writes, and new changes appear after Lakehouse's next scheduled metadata refresh (as often as every 5 minutes).



















How Lakehouse and Databricks work together




Managed Iceberg tables automatically generates Iceberg-compatible metadata alongside native Delta logs. Data engineering teams can store Parquet files once in Google Cloud Storage (GCS) using Databricks. Rather than loading this information into BigQuery, Lakehouse reads the Iceberg metadata directly, applying BigQuery’s vectorized engine directly to open-format files in GCS.


Although Google Cloud Storage (GCS) serves as the primary storage layer in the above example, Lakehouse extends this borderless federation directly to external cloud environments like Amazon S3 and Azure Blob Storage. By pairing the open Apache Iceberg REST Catalog with cross-cloud connections, Lakehouse lets BigQuery query data across clouds without moving it. Each federated catalog lives in a Google Cloud location you choose, ideally the region closest to your data. This lets BigQuery read open-format Parquet files where they live, delivering multi-cloud analytics without replication pipelines. Remote-cloud egress charges still apply, but Cross-Cloud Interconnect (AWS) and Lakehouse caching can reduce them.


Beyond simply querying external data, Lakehouse framework establishes true read and write (R/W) interoperability across engines. Because the platform is built on an open Apache Iceberg foundation, data teams can effortlessly execute multi-statement SQL commands directly inside BigQuery.


Centralized security and governance




Handling access controls across disparate systems can introduce vulnerabilities. Lakehouse addresses this through delegated access. Administrators can configure a BigQuery Cloud Resource Connection tied to a service account that manages IAM rights for GCS buckets. Granting users access to the Lakehouse table entity lets you enforce BigQuery controls including row-level, column-level security, and dynamic data masking on external Iceberg tables while Databricks handles ingestion.


Catalog federation and credential vending




Previously, connecting these platforms meant manually configuring BigQuery queries to point at static metadata paths. Modern setups leverage the open Apache Iceberg REST Catalog API. Databricks Unity Catalog functions as the central metadata layer, allowing a BigQuery administrator to establish a Lakehouse runtime catalog with a federated connection. When running a query, Lakehouse contacts Unity Catalog for current table state, and short-lived tokens are issued through credential vending to grant BigQuery temporary read access to GCS.


Partition pruning




BigQuery uses each table's Iceberg metadata to plan the query, then reads only the relevant data files in place. Iceberg's hidden partitioning lets you filter on logical columns, such as timestamps, without knowing the directory layout. Note that Databricks managed Iceberg tables don't support partition transforms; Databricks recommends liquid clustering instead.


Filters in a standard WHERE clause let BigQuery use the Iceberg manifests to skip partitions that can't match. Compare partition columns with literal values so pruning can apply. These manifests maintain detailed column-level statistical bounds, enabling BigQuery to bypass irrelevant manifest records and unneeded Parquet files immediately. Consequently, queries scan only necessary byte ranges, accelerating query execution while significantly reducing on-demand slot consumption and storage scan costs.


Fast queries with Iceberg metadata and Lakehouse caching




Iceberg manifests already identify which data files to read, so BigQuery doesn't need a metadata cache for Iceberg tables. For cross-cloud tables, such as those in Amazon S3, Lakehouse caching (Preview) keeps retrieved data blocks in Google Cloud so repeat queries read them locally. BigQuery reads the Iceberg manifest files directly when it plans each query; there's no metadata caching to enable. This dynamic partition pruning bypasses unneeded files, significantly reducing runtime and bytes scanned.


Maintenance and lifecycle management




Modern lakehouses demand active upkeep; without routine care, performance degrades into an inefficient repository. Fortunately, both BigQuery and Databricks supply native capabilities to preserve table hygiene and accelerate analytical workloads.


Keeping your tables optimized




To maintain optimal query performance, active maintenance of your lakehouse storage footprint remains necessary.




*


Trigger regular compaction pipelines via Databricks APIs to merge tiny Parquet files into larger, optimized layouts and use Predictive Optimization.


*


Execute the Databricks VACUUM command to clear unreferenced files outside active snapshots, avoiding extra GCS storage costs.






Compaction and the small file problem




Frequent write operations, continuous ingestion jobs, and streaming workloads continuously produce thousands of small Parquet files. This fragmentation creates a storage bottleneck that can limit BigQuery’s reading efficiency. Establishing thousands of network connections to GCS and processing metadata footers across numerous tiny files consumes significant compute time, diminishing the performance advantages of vectorized query execution.


Running routine compaction within Iceberg resolves these inefficiencies. Databricks manages this automatically using Liquid Clustering which dynamically adjusts file layouts based on access patterns or through native table management utilities.


To compact small files manually on Databricks, run OPTIMIZE:

code_block




Executing this command aggregates scattered file segments into larger, structured blocks typically between 256MB and 512MB that can help improve BigQuery’s scan throughput. The binpack approach operates rapidly by prioritizing total file size balancing, though alternative sorting strategies can organize dataset ordering along specific column keys.


Managing orphan files and snapshot retention (VACUUM)




Every transactional update, record modification, and table compaction creates fresh snapshots within Iceberg to preserve dataset integrity. Over time, historical Parquet objects detached from active table states become orphaned. Although these legacy files facilitate time-travel analytics, neglecting them can drive up GCS storage charges over time.


Data teams can use the Databricks VACUUM operation to scan and purge abandoned storage files.

code_block




Standard configurations apply a mandatory seven-day threshold (deletedFileRetentionDuration) before executing VACUUM routines. Preserving this delay prevents active analytical jobs from encountering missing file errors while keeping historical recovery options available. Scheduling routine VACUUM jobs, or letting predictive optimization run them, removes unreferenced data files and, for tables with Iceberg reads, old Iceberg metadata. That keeps storage costs low and performance high across your lakehouse.


Conclusion




Connecting Databricks Apache Iceberg datasets to Lakehouse can help you activate your data with BigQuery. By separating processing from underlying storage, embracing open formats, and using standard REST APIs with credential vending, you can run scalable engineering workloads and concurrent analytics without migrating data or sacrificing performance


Pairing Databricks Iceberg reads (UniForm) with Lakehouse federation, Lakehouse caching, and regular OPTIMIZE and VACUUM maintenance helps you build an efficient, zero-copy lakehouse. This approach can help lower storage spend, avoids fragile pipeline maintenance, and prepares your data environment for modern analytics and emerging AI workloads.


Constructing a unified architecture no longer requires moving data files across platforms. You can begin querying Databricks Iceberg tables by registering them with the Lakehouse runtime catalog today for bi-directional federation. Learn more by reading our documentation to get started. 🔗 Google Data


https://cloud.google.com/blog/products/data-analytics/querying-databricks-iceberg-tables-with-google-clouds-lakehouse/?utm_source=dlvr.it&utm_medium=blogger