polars data

A float column can hold two different kinds of missing: null for a value that is absent, and NaN for arithmetic that had no valid answer (think 0.0/0.0). Polars keeps them strictly separate. fill_null() leaves NaN untouched, and fill_nan() leaves null untouched. is_nan() on a null returns null instead of false, because nullness propagates through operations. And drop_nulls() keeps NaN rows in place. Want one uniform treatment? Convert first, then fill: fill_nan(None) turns every NaN into a proper null, and fill_null(0.0) handles the rest in one go. In the example below, sensor B keeps its NaN after fill_null(), and sensor C reports null for is_nan(). The "both" column shows the two-step fix.

Reading a terabyte dataset from S3 goes fastest on a cluster of small machines, while heavy joins run fastest on one big machine. We benchmarked single node Polars against distributed Polars on the same total vCPUs, RAM and price, and the bottleneck of your query decides the winner. Read the full post here: pola.rs/posts/single-n…

We've been busy in Q2 2026. Read all the highlights in the latest Polars in Aggregate: pola.rs/posts/polars-i…

Libraries for Your Python Polars Workflows #python opensource.posit.co/blog/2026-06-0…

We've released Polars Cloud client 0.9.0. Some of the highlights: • Expressions now run distributed Expression lowering lets expressions run with distributed algorithms, just like joins and group-bys. Enabled for `select`, `with_columns`, and `filter`, with the biggest gains from aggregating expressions like `mean` or `unique`. • Faster cloud I/O Polars Cloud now ships the adaptive concurrency controller for cloud reads. On TPC-H scale factor 1000 across 32 m6i.xlarge instances, total runtime dropped 17%, with some queries cut in half. No API changes needed. • Breaking: `ClusterContext` now uses `uri=` The `compute_address=` keyword is superseded by `uri=`. Blog post: pola.rs/posts/polars-c… Full changelog: github.com/polars-inc/pol…

We've released Python Polars 1.42. Some of the highlights: • Adaptive cloud I/O concurrency Parquet and IPC reads from S3, GCS, and Azure now dynamically tune the number of in-flight requests, with up to 4x improvement on I/O-bound queries. • Contradictory filter elimination The query optimizer now detects filters that can never match, such as value > 100 AND value < 50, and skips the scan entirely. • is_sorted() for DataFrame and Expr Check whether a DataFrame is sorted by one or more columns, or use it as an expression inside select() and filter(). Blog post: pola.rs/posts/polars-1… Full changelog: github.com/pola-rs/polars…

How well do LLMs migrate pandas to Polars by themselves? We tested how well Claude translates a pandas corpus to Polars. Results were promising but not perfect. To improve this, we built a Polars skill that helps the agent. Read the full post here: pola.rs/posts/llm-pola…

Distributed Polars is up to 7.8x faster than Spark on the PDSH benchmark, averaging to a 3x speedup on all queries. We ran the PDS-H benchmark (TPC-H derived, 1 TB-scale dataset) against PySpark 4.0.1 on two setups. - Distributed (32 × 4 vCPU workers): Polars averaged 3.2x faster, ranging from 1.6x to 7.8x per query. - Single node (128 vCPUs, same total resources): Polars averaged 6.4x faster, with one query finishing 38x quicker. Read the full benchmark post here: pola.rs/posts/polars-p…

Run Polars' distributed engine on your own infrastructure. Deploy a distributed Polars cluster on any Kubernetes setup (EKS, AKS, GKE, or minikube) and get a query dashboard with past queries, advanced query profiling, Open-lineage support, and more. Sign up and install with a single Helm command. Connect via `ClusterContext` and run distributed queries. Read all about it at pola.rs/posts/polars-d…

We've released Python Polars 1.41. Some of the highlights: • Faster Parquet metadata decoding Parquet metadata is now decoded with a hand-written, specialized Thrift parser instead of the generic auto-generated one. Speedup scales with table width: 1.6× for 100-column tables, up to 3.3× for 10,000-column tables. • Nested common subplan elimination The query optimizer now eliminates duplicate subplans at all nesting depths. • LazyFrame.gather() Row selection by integer index is now available in lazy mode, without collecting first. Blog post: pola.rs/posts/polars-1…

We're big @DataPolars fans at marimo. The Polars team has been working on something new: it's early, it's fancy, and it ships with a few features that are directly relevant to marimo. Livestream with @thijsnieuwdorp on May 22, 2026. Link below ⬇️

Polars supports a full Iceberg roundtrip on the streaming engine. You can scan an Iceberg table with scan_iceberg(), transform it lazily, and write the result back with sink_iceberg(). Useful for workflows like data redaction or compliance cleanup: scan the table, redact the matching user's PII, and overwrite the table with the cleaned result. That overwrite is committed as a new Iceberg snapshot, and after you validate it you can expire older snapshots as part of your cleanup workflow.

Handling schema changes in Polars. Our latest blog post maps the four shapes of schema change (a new column appears, an expected one disappears, a type drifts, or one breaks) to the Polars solution that handles each, across CSV, multi-file Parquet, Delta Lake, and Apache Iceberg. Read the full breakdown here: pola.rs/posts/schema-e…

We've released Python Polars 1.40. Some of the highlights: • Streaming grouped AsOf join AsOf joins with a `by` argument are now supported in the streaming engine, extending last release's streaming AsOf support to grouped time-series joins. • Basic over() in the streaming engine Elementwise window expressions using over() can now run in the streaming engine. • More expressions lowered to streaming cov(), corr(), interpolate(), skew(), kurtosis(), and entropy() are now natively supported in the streaming engine. Link to the complete changelog: github.com/pola-rs/polars…

We've been busy in Q1 2026. 12 releases. 778 PRs. 95 contributors (thank you!). The streaming engine now covers more join types, all major formats have a streaming scan implementation, Delta and Iceberg both have full read/write support, and Polars Cloud gained a query profiler that helped us run a TPC-H benchmark 54% faster at 64% lower cost. Read all the highlights in the latest Polars in Aggregate: pola.rs/posts/polars-i…

Polars loves sorted data! If your data is already sorted, you can get a performance boost up to 18x when joining your datasets. Read all about it in our latest blog post: pola.rs/posts/streamin…

Realtime query profiling of Polars In this post we use the query profiler in Polars Cloud to optimize the infrastructure configuration for a specific query. This results in a 54% faster and 64% cheaper query with only five runs. Read all about it here: pola.rs/posts/query-pr…