TL;DR
Get business pricing on networking and server gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Polars says version 2.0 changes lazy query execution to use its streaming engine by default and enables initial out-of-core, or spill-to-disk, support. The release also expands SQL support and adds a Map data type; performance comparisons published by the project are based on its own specified benchmark tests.
Polars has released version 2.0, making its streaming engine the default for lazy queries and enabling initial spill-to-disk support for workloads that exceed available memory. The update also gives SQL a first-class place in the project and adds a native Map data type, changing how some existing queries behave as well as what the library can handle.
Under the new default, calling collect on a LazyFrame uses the streaming engine. Polars says this can bring memory and performance improvements on many queries. The change can affect observable row order: the project says operations including joins, group-bys and unpivots do not guarantee row order by default in the streaming engine. Users who need order preserved can set maintain_order=True for supported operations.
Version 2.0 also enables out-of-core execution, which can spill supported operations to disk as memory use rises. According to Polars, spilling starts at about 80% of RAM, and the default disk budget is 64 GB. Current support includes sort, window functions and many expressions. The release does not yet extend this support to joins and group-bys; Polars says those are planned for later.
The release adds a native Map dtype for data represented as key-value mappings, with methods for retrieving values, checking keys and accessing keys or values. Polars also reports optimizer and engine work, including join reordering, common-subplan elimination and dynamic predicates or bloom filters. The project says stricter handling of data types and explicitness should provide faster feedback during development.
Safer Execution for Large Queries
The default streaming engine and initial disk spilling address a practical limitation for people running larger data jobs: a query may be able to proceed without all intermediate data fitting in memory. Polars says the changes improve resilience for high-memory workloads, although the benefit depends on the query using operations that currently support streaming or spilling. Joins and group-bys remain outside the current out-of-core coverage, so the release does not remove memory constraints for every workload.
The default change also makes migration testing important. Code that depends on a particular row sequence after a join, group-by or unpivot may produce a different observable order unless that requirement is specified. SQL improvements may broaden the library’s use for teams that prefer SQL to its expression API, but the performance case should be judged against workloads and hardware beyond the project’s published tests.
large memory spill to disk data processing laptop
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
How Polars Tested SQL Speed
Polars presents version 2.0 as a major version focused on making its existing engine available to more workloads, rather than solely as a feature release. Its SQL comparison used queries generated with DuckDB 1.5.6’s TPC-H and TPC-DS query generators, against DuckDB 1.5.6, a DuckDB 2.0 alpha build and DataFusion 54.0.0. Tests ran on two Amazon machine types: one with 16 vCPUs and 32 GB of memory, and another with 192 vCPUs and 384 GB.
In the project’s methodology, each query ran five times in a hot setting, and the fastest run was used. Polars says it was fastest on all but one of the reported benchmarks under its default settings. It also reports that performance on the 192-thread machine carried a constant overhead that hurt smaller queries; limiting Polars to 32 cores made it competitive or faster across the tested benchmarks. These are vendor-reported results, not an independent evaluation, and the project has published a repository for reproducing the tests.
The results have exclusions: according to Polars, DataFusion timed out on TPC-DS query 72, timed out once on query 67 and ran out of memory on TPC-H query 18 on the smaller machine. Those queries were excluded from the results for all engines. The project says it has diagnosed the high-thread overhead and hopes to address it in a later release.
“Calling collect on a LazyFrame will now default to the streaming engine.”
— Polars, in its version 2.0 release report
high performance data analysis software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Limits of the New Defaults
The release report does not specify a publication date, so the timing of the announcement cannot be dated more precisely from the supplied material. It also does not quantify performance or memory changes for a broad set of independent workloads. Polars’ benchmark comparisons are project-run tests; results may differ with other data, hardware, query patterns or configuration.
It remains unclear when out-of-core support for joins and group-bys will arrive. The roughly 80% RAM spill threshold may also need tuning, according to Polars. Users should check whether their particular operations preserve row order under streaming execution and whether their workload is among those that currently support spilling.
As an affiliate, we earn on qualifying purchases.
More Spill Support Planned
Polars says it plans to extend out-of-core execution to joins and group-bys, which would cover additional memory-intensive workloads. The project has not provided a delivery date in the release report. It also says it hopes to fix the overhead it observed when running on 192 threads in a subsequent release.
For now, users moving to version 2.0 can test queries that rely on row ordering, review whether their operations are supported by disk spilling and reproduce the SQL benchmarks using the repository Polars shared. Those checks can show how the new defaults and performance characteristics apply to each deployment.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is the main change in Polars 2.0?
LazyFrame.collect now uses the streaming engine by default. Polars also enables initial spill-to-disk support and expands its SQL features.
Does Polars 2.0 preserve row order?
Not by default for some streaming operations, including joins, group-bys and unpivots, according to Polars. Users who need a specified order can use maintain_order=True where supported.
Which operations can spill to disk?
The release report lists sort, window functions and many expressions as currently supported. Joins and group-bys are not yet supported for out-of-core execution and are on the project’s roadmap.
Did Polars prove it is faster than DuckDB and DataFusion?
No independent proof is supplied. Polars reports that it was fastest on all but one of its stated TPC-H and TPC-DS benchmarks, but those are project-run tests with specified hardware, methods and exclusions. Results on other workloads may differ.
Source: hn
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
