Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- > as soon as their ready.
So strange that it's now relatively normal to see a typo and think 'oh cool, a human wrote this, I can take this seriously' rather than 'oh dear, they can't spell, I can't take this seriously'.
by dtj1123 - The decision to default to the streaming engine is really interesting. My intuition is that this would be slower than other data frame operations that are more parallelizable with batch processing, because streaming engines necessarily process rows sequentially. Is my intuition off/am I overestimating how much auto-parallelization polars does?by arn3n
- Streaming here has a different meaning than perhaps what you're used to. It's not referring to online processing where you maintain aggregates/state while an endless stream of data comes in.
The name was chosen early on to contrast with the old execution model, which was essentially all-data-in-memory, column-at-a-time. That engine still exists, we use it as a fallback mechanism for things that aren't supported yet in the new engine (or if you explicitly ask for `engine="in-memory"`).
The new execution model first constructs a computational graph of nodes which communicate in streams of in-cache batches (morsels) of data, meaning the full dataset will never be held in memory if not necessary. This was called the streaming engine for that reason in an early prototype and the name stuck. In hindsight I do admit the naming choice is somewhat confusing.
by orlp - Bothering me like crazy that "Use instead: .cat.to(dtype) for int → categorical, .cat.physical() for categorical → int." doesn't give the requisite code example!by bbstats
- https://docs.pola.rs/releases/upgrade/2/#disable-casting-fro...
You can find it in the migration guide. Let me know if you miss anything, if you'd like you can make an issue and I'll make sure to get to it before the 2.0 release!
by thijsn - >Enums/Categoricals <> integers.
>pl.Series([None, 1, 0, 2], dtype=pl.UInt32).cast(pl.Enum(["a", "b", "c"])) ># ComputeError: casting from u32 to enum is not supported.
>Use instead: .cat.to(dtype) for int → categorical, .cat.physical() for >categorical → int.
Always show the correct way of doing things. I have no idea what the correct is here, and I don't really see what benefit this change in API brings.
by wodenokoto - Happy to see activity around Polars. This has been my go-to library for data processing due to the enhanced ergonomics compared to Pandas and SQL.
But they were a bit quiet lately, and I started looking more and more into DuckDB recently… until the recent acquisition of DuckLab by AWS
by Kydlaw - I've been using clickhouse-local for quite some time, instead of DuckDB. There is also chDB.
After using pandas for 10 years, I favor SQL now, for some reason.
by f311a - Moving towards streaming and generally out-of-core is great
We recently added a Polars backend to GFQL (cypher graph queries on dataframes, no DB needed), both CPU and GPU mode, and super impressive. Noticeable improvements vs pandas/cudf, and enabled GFQL to beat out popular systems on more categories like low-latency, not just big datasets: https://www.graphistry.com/blog/cypher-on-polars-cpu-gpu-gra...
by lmeyerov - I love polars. Did a lot of evangelizing in work to get people to give up pandas in favor of it.
- Both have terrible syntax that make SQL look like the most readable thing ever.by mgaunard
- I guess I am a casual pandas user only. Reading a guide on migrating/differences, it's hard to see why polars would be obviously better.by duskdozer
- Taken out of context, your post looks like a conservationist who got fed up with pandas being a flagship species and made it their lifelong mission to replace them with polar bears.
This is not a criticism. As someone who doesn’t use Python, I simply found it amusing.
by latexr - I gave up pandas in favor of polars after someone at work did the same and I am very happy with it. Pandas API is just so much worse and much slower.by anotherpaul
- Is there a reason besides performance that maintain_order=False by default? I ask because polars is used in many scientific data analysis pipelines, and non-deterministic behaviour is a well-documented source of bugs in scientific computing (e.g. https://pmc.ncbi.nlm.nih.gov/articles/PMC6919963/). The new default requires users to keep the implementation details of the API in their head while determining whether code is correct or not. This is tricky with scientific computing because the correct answer is not known in advance, so bugs can slide by and silently give incorrect results.
- Perhaps to get better results on benchmarks.by winniewinnie
- This behavior has repeatedly frustrated me. I am writing some new transformation, want to see the results, and my first few sentinel rows are nowhere to be seen because they have been shuffled.
I do not think of a dataframe as a set, but an ordered collection of rows. My source csv had the rows in this order and I want that maintained unless I choose maximum performance.
by 0cf8612b2e1e - Is "non-deterministic" the right description for this? I read it as describing an implementation where ordering is not preserved, but deterministically. Is that a misreading?by sanderjd
- I once persuaded the dplyr maintainers not to do an update that might re-order rows after a filter(). I think the human tendency to think of database rows as existing in a fixed, given order, which will only be changed explicitly, is deep.by dash2
- This is a tricky field, the problem is not actually the non-determinism of the processing algorithms, but implicit ordering of the data.[1] The implicit ordering of the data is a footgun that -- as seen in the paper -- has already claimed victims. Using algorithms that don't need to upkeep the ordinality requirement in every operation will definitely move the library to a better direction and make future data modeling better and more explicit.
[1] Aha, now I see why language models use this so frequently and why it might be overrepresented in the data. This is a perfect way to move the blame from the person you're responding to, if they're mistaken. They probably have a super, super overtuned "politeness" gym using sentiment analysis that tries to reword answers to not blame the misunderstandings of the person. Then this blame shifting unfortunately gets re-used as this super, super common phrase.
by Permik - Unfortunately (for your case) the ordering of group_by, join, and unique all run in parallel hash aggregation across the threads so the output order comes about by how it gets partitioned across cores. Which is why you can get different order of rows depending on the machine even when you have the same set of data and polars version (this has happened to me). To fix you can set maintain_order=True or probably better an explicit sort whereever you save or compare / diff the output.by jorelfermin
- If the correctness of my program depends on the ordering of data (for operations that would otherwise be commutative), that seems like that should be something explicit rather than implicit.
I'm not sure if I agree that "hidden setting actually keeps your data correct" is something that should be the default.
by nemothekid - It's standard sql behavior, users always specify the ordering they want as part of the query.
- For me, the superpower of polars is production stability.
Pandas tends to push all problems to runtime, with all sorts of hidden heuristics. Particularly around column types and missing values. It's very hard to know if you've tested all the edge cases. The only way to test your code is to throw all variations of data at it. Fine if you're sitting at a notebook and have the patience to validate and "clean" the data on its behalf. Not so fine if you get paged at 3am because your data pipeline failed when it expected an int column but got float.
Polars is more strict by default and front-loads costs through its planner. The resulting apps are noticeably more stable in production. You can test code and reasonable assurance that it will work on data in the wild.
I don't really have any interest in the API ergonomics or syntax - both are fine. It's all about how they deal with data variation at runtime. Can you write general code that doesn't break on variants? Pandas, not a chance. Polars, absolutely!
Bonus round: polars has a Rust API too, the compiler can effectively prove that your program handles every edge case. It's common to write rust polars apps that run unattended for years.
by perrygeo - Do people actually use languages where it doesn't do any compile time checks on the API in 2026? Why would developers put up with the lack of that. I'm not in this ecosystem but what you are describing sounds like the bare minimum to me that should be table stakes.by sigseg1v
- > We don’t aim to make a big feature release of Polars 2.0. In fact we hope it to be a boring experience for you. The reason we bump this major version is that we can get rid of design decisions made in the past that currently block us and then we want to change defaults to more sensible settings that will benefit a greater audience
I know this take reveals me as a very dull person, but I love seeing projects take semver seriously like this! Version bumps should really be about removing deprecated cruft rather than shiny new features.
I've used polars for a while now, and their focus on stability was a big part if convincing me to make the jump initially!
by benrutter - "Tranquil development" (vs. "hype-driven shipping") :)by Bluestein
- > Version bumps should really be about removing deprecated cruft rather than shiny new features.
Can there be deprecated cruft without new features? :-D
by nicce - That being said Polars is one of the few Python libraries from the hundreds I use that I need to read the notes of every minor release (eg 1.44 -> 1.45), because they tend to frequently deprecate, remove or change features.by dist-epoch
- Aren't major versions supposed to indicate breaking changes..?
That's how I thought semantic versioning worked
by altern8