Managing Data Files in Apache Iceberg
Everything was going great, your data was in your data lake, queries were fast and the SREs were happy. But then things started to slow down. Queries took longer, even specific queries which used to be fast now take a long time. The culprit? Small and unorganized files.
The solution? Apache Iceberg’s RewriteDatafile action. In this talk, Russell Spitzer will dive into how RewriteDataFiles can 1) right-size your files, merging small files and splitting large ones, ensuring that no time is waisted in query planning or in opening files; and 2) reorganize the data within your files, supporting hierarchal sort and multidimensional ordering algorithms, enabling you to make sure your data is optimally set out for your queries. With these two capabilities, any table can be kept at peak performance regardless of ingestion patterns and table size.
Topics Covered
