The Dremio Blog

Iceberg keeps per-file bounds automatically but table-level NDV statistics only when you compute them. Why that decides join order, and how to keep them fresh.

Recovering Polaris means recovering a metastore and object storage separately. What to back up, the order to restore in, and the failure mode nobody plans for.

What causes an Iceberg commit conflict, the retry properties that govern it, the two isolation levels, and how to design conflicts out of a busy table.

Parquet and ORC are columnar and built for reading. Avro is row-oriented and built for writing. How to choose, and why Iceberg quietly uses all three.

Parquet is how columnar data sits on disk and Arrow is how it sits in memory. Every difference between them follows from that one split.

Iceberg is a table format and Parquet is a file format. They are different layers of one stack, not alternatives, and an Iceberg table contains Parquet files.

Apache Iceberg 1.12.0 expands v3 engine support, REST catalog operations, streaming reliability, and v4 groundwork. Learn how to upgrade safely.

Apache Polaris 1.8.0 adds semantic-model privileges, safer commit handling, pagination controls, CLI improvements, and important database upgrade steps.

Apache Iceberg v3 defines geometry and geography as first-class primitive types. Values are encoded as WKB in Avro and Parquet, with optional CRS metadata and spatial bounds that engines can use for pruning. The specification is stable, but engine support is version-specific, so teams should test every reader and writer before upgrading a shared table.

The Apache Iceberg View Spec makes view metadata portable by storing versioned schemas, catalog and namespace context, dependencies, properties, and one or more SQL representations. It does not make every SQL expression universally executable. Portability depends on a catalog and engines that implement the spec and agree on a supported dialect representation.

Iceberg table encryption separates file encryption from key management. Writers create data keys, encrypt files, wrap those keys with a key encryption key in a KMS, and store key metadata with the files. Iceberg can encrypt data files, delete files, manifests, and manifest lists. Table metadata JSON is intentionally not encrypted by the Iceberg encryption specification.

Server-side scan planning lets an Iceberg REST catalog plan table scans close to metadata instead of sending every manifest to the client. A client discovers the capability, submits a scan request, receives completed tasks or an asynchronous plan identifier, then polls for batches and can cancel work it no longer needs. The feature reduces metadata transfer, but it moves planning load and observability into the catalog service.

The Iceberg REST protocol defines an optional transaction commit endpoint that can submit updates for several tables as one catalog operation. That endpoint can provide catalog-level atomicity when the server implements it, but a usable multi-table transaction also requires client support, conflict handling, and clear failure semantics. A REST catalog label alone does not guarantee that an engine will issue coordinated commits.

MERGE INTO is the clearest way to express keyed changes against an Iceberg table, but reliable upserts start before the MERGE. Deduplicate the source to one action per key, define late-arrival rules, and make the batch replayable. For SCD Type 2, close the current row and insert a new version in a controlled transaction, then compact the delete and data files the workload creates.

Iceberg storage cost is not only the number of terabytes in data files. It includes PUT, GET, LIST, and DELETE requests, retained snapshots, delete files, manifests, metadata JSON, compaction rewrites, and recovery windows. The safest cost plan starts with table-aware expiration and orphan cleanup, then applies object-store lifecycle rules only to paths and objects whose ownership is understood.