Luca Canali

New blog post 🚀 Are you happy with your CPU performance? 🤔💻 Quickly measure and load-test your CPUs with a simple Rust tool: test_cpu_parallel externaltable.blogspot.com/2025/12/are-yo… #CPU #Performance

New blog post 🚀 Why I’m Loving Spark 4’s Python Data Source (with Direct Arrow Batches) 🔗 db-blog.web.cern.ch/node/200 #ApacheSpark #Python #CERN

New post: Troubleshoot I/O & Wait Latency with OraLatencyMap and PyLatencyMap 👉 db-blog.web.cern.ch/node/199 #Performance #Oracle #Linux

Our latest tech blogger interview features “long-time performance geek” @TanelPoder. He just hit 18 years of blogging, and that deep experience really comes across in his thoughtful responses. writethatblog.substack.com/p/tanel-poder-…

[ANNOUNCEMENT] The Apache Spark 4.0.0™ release is here! 🎉 Congrats to the #ApacheSpark community! This milestone is a testament to tremendous collaboration, resolving over 5100 tickets with contributions from more than 390 individuals. Try it out! ➡️ spark.apache.org/releases/spark…

@jaiminks Yes, you can use sparkMeasure for you use case of running on Databricks and saving the collected metrics to Delta tables. After collecting the metrics, just convert the results to a Spark DataFrame using the method stagemetrics.create_stagemetrics_DF() and write them to Delta.

@LucaCanaliDB can we use sparkmeasure for databricks. I would like to save the output in delta tables.

🚀 A new version of #sparkMeasure is now available. This lightweight tool helps collect and analyze #ApacheSpark job metrics, making performance troubleshooting easier for developers and data engineers. Explore it here 👉 github.com/LucaCanali/spa…

🚀 New blog post: “ATLAS DCS Analysis with Apache Spark and Jupyter Notebooks” Discover how we leverage Spark, Hadoop, and Jupyter Notebooks for scalable time-series analytics. Read more: db-blog.web.cern.ch/node/198 #ApacheSpark #Python #CERN

🚀 New blog post! Kepler’s Mars Orbit: Reproducing his analysis with Jupyter Notebooks, Python & AI-assisted coding for faster insights. 🔗 Read more: db-blog.web.cern.ch/node/197 #Python #DataScience #AI 🚀🔭

The computing behind LHC data processing (WLCG) these days: 💻 1.3M cores crunching data 📀 1 exabyte on disk 📼2 exabyte on tape 🌍 Distributed across 40+ countries #BigData fueling collaboration and discovery. #CERN #LHC

Ran the TPC-DS benchmark at 10TB scale using #ApacheSpark on our Hadoop cluster after recent upgrades. Everything looks good. 🔗 TPCDS workload generator: github.com/LucaCanali/Mis… 🔗 Dashboard: github.com/cerndb/spark-d…

I have uploaded the video from my today's 0xtools eBPF talk here: youtu.be/_nLAdkDhaI0 Github code, slides, discussion: github.com/tanelpoder/0xt…

@TanelPoder Fantastic work Tanel, thank you for sharing. This work and toolset will be big!

TPCDS_PySpark is a TPC-DS workload generator in Python, designed to scale with #ApacheSpark. It collects and reports performance metrics by integrating sparkMeasure, a monitoring library for Spark. 🔗 Repository: github.com/LucaCanali/Mis… 🎥 Demo: youtube.com/watch?v=1EjLCa…

🌟 SparkMeasure is a tool and a library designed to simplify Spark metrics collection, for efficient analysis and troubleshooting of #ApacheSpark jobs. 🔗 Repository: github.com/LucaCanali/spa… 🎥 Watch the demo: youtube.com/watch?v=NEA1kk…

🚀 New Blog Post: "Building an Apache Spark Performance Lab: Tools and Techniques for Spark Optimization." Get tips, tools, and short demos to boost your Spark performance. Ideal for developers! 🛠️#ApacheSpark #Performance Read more: db-blog.web.cern.ch/node/195

New Blog: "Enhancing Apache Spark & Parquet Efficiency" - explore the latest updates on column indexes and bloom filters for better big data processing. Read more at db-blog.web.cern.ch/node/194 #BigData #ApacheSpark #Parquet #CERN

🚀 Just dropped a fresh blog post! Dive into the world of Apache Spark optimization with flame graphs, featuring a hands-on example with Grafana Pyroscope. 🔥📈 🔗 db-blog.web.cern.ch/node/193 #ApacheSpark #FlameGraphs #Pyroscope

New blog post "Performance Comparison of 5 JDKs on Apache Spark" db-blog.web.cern.ch/node/192 #ApacheSpark #Java #Performance