Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- Polars seems nice but in my experience using it, the "lazy" APIs would still immediately materialize a ton of stuff in memory and had very spotty support on what data formats and storage integrations were possible with scan_* functions (though that was half a year ago and the support is slowly improving). It's frustrating, I mean really frustrating, to think I could solve a lot of my "scan through heinous amounts of data without any memory hungry things like window aggregations without blowing out my memory" with Polars and then watch my scan_thisorthat() call result in instant memory usage ballooning.
DuckDB on the other hand is wonderful and truly doesn't use any more memory than it really needs to.
by viccis - Did you ever report these issues to polars developers?
I reported many bugs to both pandas and polars over the years and both teams tend to address relatively big ones. (Still have some outstanding pandas bugs that I think are a big deal but the devs disagree.)
- That sounds like there's something going wrong, because that definitely shouldn't be the case! Although it sound like you've given up on it: if you have any example to send me in the right direction I can look into that for you. I'd love to know if there's something I can relay to the team.
Thijs, DevRel @ Polars
by thijsn - Besides the performance benefits, I use Polars at work because it’s just (subjectively) nicer to work with. The “pl.col” API lets you create arbitrary generated/virtual columns anywhere you want, declaratively. You can throw in these column expressions in wherever without actually computing their values and storing that in memory. Very powerful stuff.by stephenlf
- Sup Eddie, the actual motivating example is we finally nixed Pandas from Data Ingestion, now reading excel files takes 2 seconds instead of 2 minutes. However unfortunately your
> TODO: rewrite this entire service
remains.
by akdor1154 - Rewriting pandas to polars is relatively trivial for most tasks these days. Especially if you wrote your pandas code correctly.
I still prefer (and use) Pandas for EDA. I think matplotlib integration is a better choice for most viz.
Also, I'm probably in the top 3-5 worldwide for number of folks I've trained with pandas. I offer Polars training and there is little demand for it.
- > Sup Eddie, the actual motivating example is we finally nixed Pandas from Data Ingestion, now reading excel files takes 2 seconds instead of 2 minutes. However unfortunately your
I shall pass this information along to my sleep paralysis demons. They'll be glad to hear it.
> > TODO: rewrite this entire service
Part of me enjoys the historical significance of this. However, I feel like keeping it would violate my own princples: https://eddie.codes/posts/source-code-comments/
by __eddie__ - Only in the last few years did I start using SQL properly. Before that my pipelines would live in python. Now I offload as much to the db as possible, and keep my python simple glue. I'm very happy with this compared to other methods in pandas or polars.
If I still need to do db-like things in python I think duckdb is better.
by evolve-maz - I use polars or duckdb now exclusively. Better syntax, better performance. But pandas is deeply entrenched - I try to get my team off it but it’s an uphill battle. It’s not going anywhere anytime soon.by Vaslo
- I'm in the middle of wrapping up the edits for Effective Pandas 3rd Edition. (I also wrote a Polars book and just wrapped up a weeklong training session on pandas this week.)
Pandas is not perfect, it has a bunch of warts. But it is good enough for most. (And many of those folks are using Excel or tableau or power bi... These were the types I was training this week).
If you have medium data, migrating from pyarrow backed pandas to duck or Polars is trivial.
- Out of curiousity, why wouldn't someone just migrate directly to Polars?by jamincan
- Am I crazy or did the OP swap the contents of the posts around accidentally?
https://eddie.codes/posts/pandas-should-go-extinct/ <=> https://eddie.codes/posts/source-code-comments/
by crazysim - Yep. Was very confused about the title and URL slug; reached the end waiting for Pandas to be mentioned.
- Sorry, fixed it now!
Published two posts at the same time and total PEBCAK
by __eddie__ - In my opinion a better argument to stop using pandas is the very unintuitive API pandas have. Additionally, a slight change in the query can force you to restructure the whole query (change all lines), while in Polars (and tidyverse in R) it's just a simple one-line change.
- If you learn to write pandas correctly you end up writing it very similar to polars (or tidyverse).
I agree that the API has warts, though typically it is more concise than polars.
- And for a similar dplyr-like API on duckdb I recommand the excellent https://ibis-project.org/ !by d2ou
- People are usually surprised to hear that polars can be slower for fairly pedestrian operations, especially with smaller datasets. For example take a 5000 x 3 dataframe of float64 and sort by one column and you'll find polars takes about 2.5x as long. If you set POLARS_MAX_THREADS to 1 then it's faster. Though this all depends on the machine. Polars tends to shine with larger datasets or where it can heavily take advantage of query planning.
Don't skip your profiling
by mulmboy - (Without profiling or looking into this at all) I'd guess this has to with thread creation, inter-core communication/latency, and possibly having to merge results or otherwise interleave operations. SMT is another likely candidate.
Regardless, CPUs are really good at single-thread.
by winwang - > People typically start with Excel and graduate to Pandas somewhere in the GB range. Pandas serves them well into the 10s of GBs range, and then they start hitting memory issues, slow computation, or become frustrated with Pandas’ baroque API.
Assumes that a project moves beyond 10s of GBs. I guess 99.9% of projects that import pandas fall well below this threshold.
by sjtrny - Most folks just need to learn how to use pandas well and that will open enough doors. Then they can move to polars or duck if needed.
- At my work I had convinced the ML pipeline engineers to switch from pandas to polars for even small ETL pipelines and there were notable performance gain with better CPU/memory utilization.
If a library is performant at large datasets, it is likely performant at small ones too.
by minimaxir - True, but also if Polars and DuckDB offer a similar experience with the ability to scale beyond that range, why not use them (for new projects)?
This isn't a call to arms to rewrite everything in the new shiny, just consider the new shiny for new shiny things
by __eddie__ - Cute title; I thought I was about to read a contrarian ecologist saying "a species that is so very specialized in its diet and finicky in its reproductive behavior doesn't deserve, evolutionarily speaking, to survive". (Seriously, it's seriously freakin' difficult to get pandas to reproduce in captivity). And while my knee-jerk reaction would be "maybe, but we should still preserve them because we can", I was prepared to see if the author had a serious argument to present.
Instead it's a cute bait-and-switch title, and the article tells you upfront that it's actually about the Python `pandas` library. Which I think I've encountered maybe once in my entire career (I'm not in the data-science field), so I don't have much meaningful to say about the article itself. I just want to commend the author on fooling me with the title. This is the kind of "clickbait" I can respect and actually wish there was a little bit more of sometimes. A nice chuckle, then a real article.
by rmunn - I was expecting a diatribe about the (UK) police: "Next week, right, I'm going to blow up a panda in Croydon".
- Yeah I wanted the controversial panda article. Can someone write that instead?
- Apologies for not rewarding your trust ;)
Would be very interested to read an article on the literal interpretation of my title.
Also just wanted to say thank you for this comment - made my day :)
by __eddie__ - > Seriously, it's seriously freakin' difficult to get pandas to reproduce in captivity
It is impossible, and has been tried many times by many people, to reproduce Octopus vulgari in captivity. Their reproductive behaviour looks also quite weird for us (the mother always die). Yet they thrive in the wild unless over fished.
by otherme123 - There is indeed a whole book about ecosystems and charismatic megafauna called "Do We Need Pandas?: The Uncomfortable Truth About Biodiversity"[1]. And there is debate among naturalists [2]
[1] https://www.goodreads.com/book/show/9656950-do-we-need-panda... [2] https://www.theguardian.com/environment/2009/sep/23/panda-ex...
by grndn - It's been a while since I've seen an actual data science post submitted to Hacker News: both because AI has superset a lot of DS tasks (e.g. vector embeddings), but also because not much new has happened in DS. Polars has been around for a bit and as noted it is much better than pandas, but otherwise the DS ecosystem has been somewhat stagnant.
I'd write more tutorials about how to use data science tooling but one consequence of AI is that all the old data sources I used to analyze such as social media and Reddit are now completely locked down (I am surprised NYC Taxi is still being updated, though). Therefore in the meantime, I'm working on making better data science tooling...although unclear to what end due to the data issue above.
by minimaxir