Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- I’m confused about the choice for Parquet and DuckDB here. PostGIS is arguably a better match for what this project is doing, and would let you skip most of if not all uses of Shapely and Pyproj.by sgarland
- You might want to consider looking at Wikipedia's internal article quality assessments (these include Featured Articles, Good Articles, B-class, C-class, Start, and Stub). I use these as a rule of thumb for how popular the topic is, it's a solid proxy for both significance and the richness of the available content.by zdmgg
- > Overall the evaluation of success was one of the most challenging parts of the project. As a developer, I’m used to building features that either work or don’t and there is often an objective way to measure how well a feature performs. For messy real world data it was hard to evaluate how good or bad the pipeline was. Furthermore, it was easy to start optimising for a specific parameter or route and find later that this work led to severe degradations in other areas.
> Verification becomes hard to reason about because there is no ground truth for points of interest, there are no red/green unit tests for taste. I’m sure these are familiar challenges to data scientists and that there are frameworks and evals for working on them. This will require more iteration and manual overrides. Hopefully with feedback and collaboration from the community. But for now I’ve shipped V1…
I suspect LLMs may be able to help us quantify our taste because they can keep track of so many data points all at once, where we have to lossily abstract these details away.
by jt2190 - You can’t unit test for all the aspects that make up taste, it’s true.
But if you break off parts of that - eg. by looking at what is codified out there as “good” design, what’s considered best practice etc - you can create tools the agent can call on that let it get critiques of its own work.
What’s really cool about this is those tools can be code, written by agents and committed to your repo. Put together a script that for example makes sure your brand colors are enforced (eg. https://github.com/cadamsdotcom/CodeLeash/blob/main/scripts/...) and then put it in your pre-commit checks (https://github.com/cadamsdotcom/CodeLeash/blob/main/.pre-com...), and the agent will get feedback on its use of tasteless defaults and adjust accordingly (partly because you blocked commits that contain said tasteless defaults!)
by cadamsdotcom - Taste is mostly the part of the spec you forgot to write down, plus the part you couldn't write down even if you tried.by TimXare
- https://pureinference.com/insights/taste-is-the-new-skill
I wrote about this a few months back. Rick Rubin is famous for this. I do think it is something that can be trained though, it just needs a lot more context. Taste builds over time through lots of unit tests, through lots of content writing, through an accumulation of product decisions. It’s hard to put it in the individual spec, but it can be teased out of 100 project specs. And when you get to that scale the AI starts to do it pretty well.
by timroman - This is exactly it - the ultimate skill now is to be Rick Rubin with an LLM. Not a comfortable transition as a coder.by themgt
- > Rick Rubin told Anderson Cooper he has no technical ability. Doesn't play instruments. Can't work a mixing board.
If you watch his interview on Rick Beato's channel, this myth will fall apart. He plays guitar, had his own punk rock band and his guitar playing is featured on some high-profile records he produced. Also, he has a lot of practical experience with all kinds of studio equipment.
by sesm - It makes me smile when runners use "X is a marathon, not a sprint" to hint at an effort that accumulates over time and an optimal use of energy.
I do it too because it's a common expression, and a marathon is of course longer than a sprint, but both have in common that properly raced, they are absolutely brutal efforts that leave you without a single additional drop at the end. The effort length and instantaneous power output changes, of course. Maybe "it's a marathon build, not the race" would be more precise at the loss of nearly all its expressive power (but with a lot more pedanticism points) :-p .
Nice project !
by chantepierre - "The effort length and instantaneous power output changes, of course."
but that's what the phrase is meant to convey, right?
Don't run through consumable X (energy/money/etc) like there's no tomorrow - even though there's <some big important milestone> now, we've got dozens more of those that we need to meet, so you're better off getting this one done at 75% than committing 100% to it and failing on all the others.
by another-dave - Exactly one of the reasons I never went down with all the TDD dogma of only writing code to fix broken tests.
There is a reason conference talks are always about plain algorithms and data structures.
by pjmlp - yup and I find it weird that people still remain so defensive of the Church of TDD even against empirical studies that show its limited benefitsby zuzululu
- > TDD dogma of only writing code to fix broken tests.
Isn't red-green-refactor pretty ingrained in TDD?
Only write code to make a failing test pass; then refactor while making sure the tests still pass?
Then write a test that fails, repeat?
by e12e - The biggest flaw I've seen with TDD is the fact that correctness does not compose upward. Every time two units come into contact, you've got an entirely new kind of unit. The tests from constituents do not cover emergent properties of the new things. You will repeat this same exercise the entire way up to the top, and the moment you come into contact with the customer (they want to change everything), the house of cards comes crumbling down and you have to start your agonizingly-slow process all over from the bottom again.
The only thing that the business seems to care about is top-down UI testing. This is also convenient because you can leave it until the very end after the customer has already seen several prototypes.
I do think TDD makes sense in isolated scopes (prove this specific custom parser works at the edges), but as the general policy for the entire product it's definitely not a viable practice. Much of the time if comes off as an ego trip to see just how cleverly we can mock something so that we can say we technically tested it.
by bob1029 - > but it ended up merely in a supporting role
This has been my experience, as well, but it’s a really big support. It just needs adult supervision. I can’t understand how vibe-coded apps, actually work.
As far as “taste,” goes, I test my stuff constantly, checking for even minor “friction points,” sometimes, refactoring back to design, in order to resolve issues that many folks would ship. I’m pretty anal, and want my work to be the best experience possible.
I can’t see any LLM coming close to being able to evaluate the user experience, like I can.
- Tools like Playwright and Maestro can already give you a small taste of what that would look like.
But overall I agree, LLMs are currently awful at being beta testers. They miss the most basic stuff that any human would immediately catch as being poor UX, and for all their visual prowess they are terrible at auditing UI.
by paytonjjones - > I can’t understand how vibe-coded apps, actually work.
With a better process. e.g. plan->revision cycles, better instructions/docs like an ADR system.
I don't think vibe-coding is relegated to "build me reddit but with blockchain" and then it's done.
I think it instead describes the workflow where the software impl stays opaque but you evaluate the end product as an end user to step the product forward. It basically centers you as the tastemaker.
I'd say I vibe-code all of my personal projects now since December where AI had a breakthrough where it required less babysitting and developed good "taste" like smart sum types without being prompted to do so.
I've accumulated my own best practices like a heavy plan->revise cycle where plans ultimately promote into ./plans/impl/YYYY-MM-DD-{slug}.md, and an ADR system in ./docs/design/*.md that encodes arch/design invariants that accumulate over time, and new decisions/principles are folding back into it as they are discovered (by the AI).
During the plan revision cycles, the LLMs may ask me a multiple choice question about which decision branch to take, and lately I've just been responding with "take the ideal option" with good results -- either way it will take a well-reasoned position that I can't really argue with.
Meanwhile, my role is mainly to evaluate the end product and steer it directionally. How much I decide to prescribe and inject myself into technical decisions is a function of how serious the project is, but it's easy to notice that LLMs are simply better and better at arriving at well-reasoned decisions, and my interjections are more and more limited to technical/directional taste rather than necessity.
by hombre_fatal - Unrelated to code, but along the same lines. I've been keeping track of the Reckless Ben case to fuel my unhealthy indignation, and we just had a like-for-like comparison between a human and an LLM.
Human: well-scoped argument that does just enough to get the job done with minimal risk.
AI: Extremely clever and correct legal argument that almost any lawyer would have said not to file (at least as written). It tries to burn the world and seriously risks pissing off the judge.
by zamalek - Interesting video, thanks for sharing it.by jdlshore
- Language count is a decent notoriety signal though pretty coarse. The OP/author should take a look at QRank: https://qrank.toolforge.org/
> QRank is a ranking signal for Wikidata entities. It gets computed by aggregating page view statistics for Wikipedia, Wikitravel, Wikibooks, Wikispecies and other Wikimedia projects
from https://github.com/brawer/wikidata-qrank/blob/main/doc/desig...
by Gosper - Cool! Thanks for sharing.by hei-lima
- OP here, that looks really neat, thanks for the link!by kalli
- You can't unit test for taste if you haven't written down what you mean by taste. If you can externalize it, then you can.
Follow this line of thinking, and the AI-friendly answer is easy: we just have to externalize everything we know, so Claude can implement what I want.
Except that I can't fully externalize myself. Debugging a system takes more resources than running the system. If I could write down everything I know and hand it to a machine, I'd do that, but it impossible.
People aren't books or hashmaps. If you want to build something, you need to use the tools, not teach the tools to use you.
[edit: I'm trying to figure out if there's something to be done about this. Email me if you want to chat -- tr at tern dot sh]
by trjordan - > You can't unit test for taste if you haven't written down what you mean by taste. If you can externalize it, then you can.
I'm not so sure. For instance, you can write down what it means for a program to be free of XSS and other injection vulnerabilities. Now, how would you unit test for that property?
by tmoertel - I remember reading an interview with a fireman who described a time when his buddy evacuated a team because he "felt" that a floor would collapse imminently.
He couldn't articulate why but they trusted his gut and it did collapse.
A lot of software engineering relies on that kind of intuition and on a good team you can integrate it and benefit from it and avoid all manner of floor collapses.
by pydry - Randomized trial. Half of them pledge to use AI freely and liberally, half of them to never use it, compare via surveys and off-AI tests after X months. Could even flip it so then the non-users used it for X months and vice versa, see if losses/gains are stable.by Dumblydorr
- >You can't unit test for taste if you haven't written down what you mean by taste. If you can externalize it, then you can.
If you can externalize it, you only captured the small part of taste that can be externalized in concrete rules.
You can of course pretend anything else doesn't exist, like a person denying anything that can't be measured by their instruments.
by coldtea - What's kind of funny is this is how I implemented "gates" for the ticketing system I built for Claude, because Beads would just close tickets without validation. I have tickets that are literally "Human validation" tier, so it will work on the next available thing until I personally tell the model to close it. So, in that spirit, yeah, you can unit test for taste, if you implement external validation.
Unit test runs, waits for human input before passing or failing, which might seem out of the norm, but we already have QA do manual testing.
- The bigger problem I have as a worker is that, once I externalize it (by writing a skill or whatever), it becomes a work-for-hire whose copyright is owned by my employer. Technically this is true of a few other things I do for work, like my .emacs and .bashrc files, small scripts I keep in ~/bin on my workstation, etc., but no employer cares to assert this unless they're being assholes for some unrelated reason. Agent skill files, especially ones that seem to semi-reliably do what they say on the tin (the white whale!), are not like that at all, and I can see them pursuing you if you try to use them at a future employer.by ElevenLathe
- You absolutely cannot unit test for taste.
I had this experience doing a port from Big Query to Postgres using Opus. I had unit tests to guarantee parity with the original code, and Opus insisted on building this bespoke query builder (e.g. `def _where(very_complicated_params)`) on top of sqlglot.
Even with the original code being straightforward and legible and repeated instructions to match, I had to fight with it to get close.
In the end, I ended up doing things the "old fashion way" where I copied chunks code into Claude proper and gave explicit instructions for each piece.
I clearly had externalized the requirements, and yet that wasn't sufficient. The only way to unit test further would be to use an AST to evaluate the output against metrics I couldn't even encode.
by fny