Apache Iceberg Data Lakehouse Tips

Experience how easy it is to take data from your source data systems, ingest them into Apache Iceberg and serve a BI dashboard from the confines of your laptop with these tutorials. #DataLakehouse #DataLake #DataEngineering #ApacheIceberg

Know Someone Learning Data Engineering, share this with them: Hands-on with Apache Iceberg on Your Laptop: Deep Dive with Apache Spark, Nessie, Minio, Dremio… medium.com/data-engineeri… #DataEngineering

ICEBERG METADATA TABLES This article will walk you through a hands-on exercise to get familiar with the Iceberg metadata tables. Read here: drmevn.fyi/Icebergmetadat… #DataEngineering #ApacheIceberg #DataLakehouse

APACHE ICEBERG MIGRATION GUIDE dremio.com/blog/migration…

DREMIO.ICEBERG.DBT.NESSIE.MINIO.POSTGRES.MONGODB If you want to try a deep end-to-end tutorial that will get you hands-on with a variety of popular data tools, try this one out. dremio.com/blog/experienc… #DataLakehouse #DataEngineering #ApacheIceberg #Dremio #dbt #DataScience #DataAnalytics

RECENT DATA ARCHITECTURE/ENGINEERING/ANALYTICS CONTENT — Apache Iceberg — > What is Data Lakehouse Table Format? dremio.com/blog/apache-ic… > Comparing Iceberg to Other Lakehouse Solutions dremio.com/blog/comparing… > Iceberg Migration Guide dremio.com/blog/migration… > Hands-on with Managed Polaris Catalog dremio.com/blog/getting-h… > Hands-on with Self-Managed Polaris dremio.com/blog/getting-h… — Hybrid Lakehouse — > 3 Dremio Use Cases for On-Prem Data Lakes dremio.com/blog/3-dremio-… > Hybrid Lakehouse Solution: NetApp dremio.com/blog/hybrid-la… > Hybrid Lakehouse Solution: Minio dremio.com/blog/hybrid-la… > Hybrid Lakehouse Solution: Vast Data dremio.com/blog/hybrid-la… > Hybrid Lakehouse Solution: Pure Storage dremio.com/blog/hybrid-la… — Unified Analytics — > Analysts Guide to JDBC/ODBC, REST, and Arrow Flight dremio.com/blog/a-data-an… > Unified Lakehouse dremio.com/blog/the-unifi… #DataEngineering #DataLakehouse #DataScience #DataAnalytics #DataArchitecture

HOW ICEBERG CATALOGS WORK Iceberg tables are one part data stored in several parquet files and a second part metadata files that provide context and understanding of that data as a singular table. The metadata entry point is a file called metadata.json which tracks the tables schemas, partition schemes and snapshots. Everytime the table changes a new metadata.json is created. So when there is possibly dozens or hundreds of these metadata.json files, how does an engine like Dremio, Snowflake or Apache Spark know which is the right one to query the table accurately. This is where a catalog comes in like Nessie and Polaris. A catalog acts like a traffic controller maintaining a list of tables along with the file address where the current metadata.json is stored. These references are updated at the end of a transaction after the new metadata.json is created enabling Atomicity guarantees. A catalog directs queries to the right metadata.json and updates that list when writes are complete. If you enjoyed this post, give it a like and a share! Also check out Dremio.com/blog for a lot more Apache Iceberg education resources. #ApacheIceberg #DataLakehouse #DataEngineering

OPTIMIZING ICEBERG TABLES One the things that make Iceberg queries fast is that the metadata can be used eliminate files that don’t need scanning from the scan plan. This is great but if the data is not clustered properly or spread out across many small files, you can still see less than ideal performance. ** Compaction ** When you have more manifests and data files than you need, you are doing more file operations and slowing down performance. By rewriting these files so you can collapse the data into fewer larger files you have the opposite effect. This can be done the REWRITE_DATA_FILES or REWRITE_MANIFESTS procedures in Spark or the OPTIMIZE TABlE command in Dremio. ** Clustering ** If I only am searching for agent in the northwest region, it’d be nice if all those reps where in the same few files, this is known as clustering. When rewriting data files with Spark, there is a “sort” parameter you can pass so it can cluster the data as it rewrites the files. By compacting and clustering you data, the Apache Iceberg metadata becomes even more powerful in skipping data files when executing queries. Read more in my new article on maintaining Apache Iceberg lakehouses here: dremio.com/blog/guide-to-… #DataLakehouse #ApacheIceberg #DataEngineering

Join us on September 5th at 10am PT for a MinIO x @dremio x @Carahsoft webinar about how modern #datalakes can help government customers solve their modernization initiatives. Register here: hubs.li/Q02Lc2rV0

Join us for "An Apache Iceberg Lakehouse Crash Course" an in-depth series designed to provide a comprehensive understanding of Apache Iceberg, taught by Iceberg expert Alex Merced. hello.dremio.com/webcast-an-apa…

Basics of Lakehouse Engineering - Apache Iceberg, Nessie (2 hour Course) youtube.com/playlist?list=… #DataEngineering #Nessie #Dremio #ApacheIceberg

Do you use an open table format? If so how’s your experience been, vote, reply and share! #ApacheIcberg #DeltaLake #ApacheHudi #DataEngineering #DataLakehouse

Open Tables (Apache Iceberg) + Open (Nessie, Polaris, Gravitino) Catalogs = No Vendor Lock-in Lakehouses Read More: blog.iceberglakehouse.com/open-source-ta… #DataEngineering #DataLakehouse #DataLake @dremio @SnowflakeDB @ApacheIceberg

NEW MEDIA ON OPEN SOURCE APACHE ICEBERG CATALOGS - New Episode of "Datanation" which you can find on iTunes and Spotify - Substack: open.substack.com/pub/amdatalake… #ApacheIceberg #DataLakehouse #Dremio #Snowflake #Databricks #DataEngineering

DATA PROFESSIONAL FOLLOW TRAIN - reply to this tweet, I will follow you - follow me and everyone else who replies - retweet to maximize reach of the train #DataEngineering #DataAnalytics #DataScience #BigData #DataLakehouse #DataLake

@AMdatalakehouse Follow me

🎙️ Dive into the minds of data disruptors! 🚀 Join us on the #DataDisruptors podcast as we unravel the strategies and insights shaping the future of data leadership. Tune in for exclusive conversations that redefine the data landscape. Listen now! 🔗 dremio.com/data-disruptor…

#Dremio paddles harder in the growing #lakehouse market with product improvements blocksandfiles.com/2024/05/03/dre… via @BlocksandFiles #AI #OpenSource

WHAT IS A DATA LAKEHOUSE? #DataEngineering #DataAnalytics #BigData #ApacheIceberg

8-BIT GAME TRAILER: Quest for the Data Lakehouse #DataLakehouse #DataEngineering #ApacheIceberg #Dremio

FREE YOUR DATA AND GET A FREE BOOK Watch to learn more #DataEngineering #DataAnalytics #DataLakehouse #DataLake #BigData