Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- It's weird that people think AI breaks the concept of work sample testing. Work sample testing isn't "about" programming, and predates the profession of programming. You can (and some companies do) work-sample test sales account managers, customer support, accountants, whatever.
AI changes the underlying job you're testing for. So, obviously, the tests you might have been using pre-AI won't work anymore; they're testing something that isn't really the job anymore. Update your tests so they're about the real work again, that's all. For coding, that probably means assuming (or requiring) candidates use AI to do your assessment.
What AI really does mess with is conversational/interactive interviewing. We do all our interactive scripted interview on Slack, but I can imagine us having to end that practice and return to face-to-face.
by tptacek - I've seen you comment along those lines before, and I think you're both right, and terribly wrong.
By all measures, the way you describe any hiring processes you've designed and had input on designing the metric and the work sample, probably is both high in signal, and not very arbitrary in scoring. They sound genuinely and refreshingly good. Maybe one day I'll interact with one :)
But man, that is very much not how work sample tests work in _a lot_ of places out there.
Places with actual, formalized rubrics for what constitutes a "pass" or a "good" score are very rare, it's almost always just based on vibes of whatever person is reading your code. If there _are_ formalized rubrics, then they have "suggested allowed time" that is incongruous with what the requirements for a "pass" actually are.
All of that is downstream, fundamentally, from the design of the tests not being taken very seriously by the people doing that. And because they're not actually thinking about it very deeply (or they're not allowed to because of time constraints or whatever); "sit down and redesign this task from first principles thinking about what AI enables now" is just not something that happens, and you just get people increasing the scopes of the project blindly.
by klausa - Sneaky age filter? You must be young enough to remember your SAT scores.by tb99
- It’s an approximation for an IQ testby seibelj
- I remember mine (both from when I was 12 and when I was 17) close to 30 years later.by nostrademons
- Between 2005 and 2015 the maximum score was 2400 instead of 1600. Assuming anyone who got <1600 during that period wouldn't admit it, you now have three well defined buckets.
But of course this is a lot of unnecessary steps compared to the usual method: length of work and education history +18 years.
by hamdingers - Sneaky age filter? You must be young enough to remember your SAT scores.
I can remember mine just fine.
If you're really looking for smart people, use "Answer this word problem in two or more paragraphs. Write your answer on the sheet of paper provided. In cursive."
by reaperducer - Sneaky immigration filter? Most wont have an SAT score at allby boredatoms
- This by default includes a whole bunch of people who didn't take any kind of standardized tests (most notably, immigrants).
The (albeit small) country I'm from doesn't do any. Reasoning was that standardized tests create an environment where teaching is merely done to create good test scores, not to actually teach.
by OptionOfT - The SAT doesn’t test course material. It is literally just applying middle school math and English proficiency.by gmadsen
- They are obviously using the SAT as a safer alternative to the legally dubious practice of IQ testing which can lead to running afoul of the ADA and EEOC. I'm not sure it's much safer, but I am positive it's less safe than doing timed leetcode. At least leetcode problems can be painted as relating to the job.
Additionally, the SAT is a shitty IQ test that is constantly crammed for and cheated on. I remember my SAT test. I was the only person in the room not openly cheating. The teacher proctor didn't care. Higher scores mean better students, more funds, higher home prices, bonuses, and a litany of secondary effects. That's not even including people that pay professional test-takers to do it for them.
The software industry needs to let go of their obsession with finding 10X ROCKSTAR L33T programmers. They never will though. It has gotten worse every few years for decades, and the problems are almost entirely managerial.
- I hate to be the one to tell you this, but every job interview is a derivative IQ test. That includes leetcode.
- > Higher scores mean better students, more funds, higher home prices, bonuses, and a litany of secondary effects.
Sounds like an IQ test
by slashdave - Canonical is doing IQ testing (with very crappy tests, see my other comment). Not sure if that is because SAT doesn't exist everywhere.by LtWorf
- IQ testing isn't legally dubious. The idea that it is is an Internet myth. There are a couple household-name corporations that administer general cognitive tests for candidates for some roles.
More companies don't do it because it doesn't work well.
by tptacek - I would never ask for them since its so cringe. But SAT scores correlate to IQ at .81, and IQ is one of the few things that strongly correlates to knowledge work performance positively. There is probably a lot of alpha from knowing candidates SAT scores. Its more useful than knowing the college they went to.by cm2012
- > and IQ is one of the few things that strongly correlates to knowledge work performance positively
i think you mean that it correlates to pay. nobody knows what you mean by "knowledge work performance." reviews of your peers also correlated with pay. often it is not the smartest person who is the most popular. so... do you see how you said something kind of meaningless?
- >I would never ask for them since its so cringe. But SAT scores correlate to IQ at .81
The combination of these two phrases is the equivalent to "I hate Trump, but ..." in another context.
by TMWNN - Is it really that hard to assess someone's intelligence after talking with them for half an hour?
That would be an interesting experiment, post-interview ask each interviewer to guesstimate the candidate's SAT score.
- Nothing is worse than a smart bad programmer. There's no limit to the technical debt they can pile up.by KevinMS
- That's r = 0.81 right? so R2 would then be ~.66. Meaning that 66% of the variation in IQ scores can be statistically explained by variation in SAT scores.
It's really high for psych stuff. If you even get r=0.5, you've got a great result there.
But it is important to note, I feel, that SAT maps to only about 2/3 parts of the IQ score, and IQ score is also a quite fuzzy measure here for things like knowledge work job performance.
I do agree though, you get quite a bang for your buck just reporting these numbers.
But, if you explicitly tie money and compensation to the SAT score, man, that is setting up some very perverse incentives around it. If it adopts widely to do so, then you're gonna get some really strange interaction effects there.
by Balgair - Company is Alpha Vantage:
https://beaverhand.com/apply/alpha-vantage-gtm-team-various-...
- > Please note that we will also rely significantly on both solicited references (where you introduce us) as well as unsolicited or "back-end" references (where we do our own). For the latter, please rest assured that we will never contact a current employer without first getting your permission.
References already give me goosebumps. Having them reach out to people who haven't given you permission to be a reference sounds like a recipe for disaster.
by OptionOfT - airbnb asked me why i didnt go iit in india and interviever soured after that. it was a chinese guy.by dominotw
- No way! These guys stole my money a while back. But it was too small (less than $100) to pursue through criminal/civil.by energy123
- > Low ego: There can be significant economic upside to this role as things progress, but the nature of this role day-to-day may involve things that can be less exciting / more mundane (it's a job, after all!)
What does this mean?
by dataviz1000 - The job description touts investment from some members of the "PayPal Mafia." For some odd reason that fact that and the SAT requirement combine to make this whole thing feel kind of normal.by F7F7F7
- I had a friend with an MS show up for first-day of work for a job that asked for SAT scores on the application. HR said "we never got documentation for your SAT scores, can you provide that?" He was on the phone with his mother, having her go through a filing cabinet when he realized that he didn't want to work for a company that was this serious about SAT scores when hiring someone with a post-graduate degree.by aidenn0
- That does seem wild. Out of curiosity, did he also have to submit GRE scores, which would be closer in time and more representative of his current knowledge/skills?by apparent
- As a manager, there are several qualities that I value highly in an engineer, and they all happen to begin with the letter C: Competent, Consistent, Curious, Caring, and Clear Communicators.
While SAT scores might act as a proxy for competency and possibly curiosity, they're not going to tell you much about whether the person is consistently reliable, whether they care about others and cooperate well, or whether their vocabulary or literary analysis skills have any correlation with their ability to read the room and tailor their communication to their audience.
If I were giving these job posters the benefit of the doubt, I would guess they're including this requirement for the same reason that musicians request particular colors of M&Ms in their riders. They want to weed out people (or bots) who aren't paying attention. Nevertheless, there are better ways to do that than demanding (and presumably filtering by) teenage performance metrics.
by jawns