90% of US working hours are spent on tasks not covered well by AI benchmarks

Join the discussion

• Hacker News
• Curious how food service is at 0 tasks covered by AI. A few places have been switching to using those robots on wheels to run food to tables. And Doordash and Uber Eats have some robot and drone delivery in select markets now.
  by pcthrowaway
• AI benchmarks are usually focused on the model’s ability to make, build, or solve something while the critical part of most tasks in the real world is in the ability to catch whether there is a mistake. Until we have more benchmarks focused on the ability of a model to detect an error I think we’ll have this discrepancy.
  by kayashaolu2
• > Our findings suggest that 90% of American working hours are spent on tasks with little or no coverage from existing benchmarks. This makes it very hard to know how good AI models are at the work Americans actually do.

  > ...

  > But even after controlling for AI-exposure, O*NET tasks from occupations with larger wage bills are significantly overrepresented among benchmark prompts:

  So, in short: they want AI to take the good jobs. AI is not coming for jobs with tasks like "gather and empty trash" or "service, clean, or supply restrooms," or "mix water and detergents or acids in containers to prepare cleaning solutions according to specifications." We software engineers and lawyers would be smart to start retraining (and adjusting our living standards) now!

  And this points to one of the ways AI will help improve society and living standards for all: we'll have cleaner, better-maintained bathrooms, as labor one wasted on engaging intellectual tasks is redirected towards menial, low-pay tasks that still need a human touch.

  Reason to be optimistic, amirite?

  by 10111001
• The fact that they think ANY part of psychiatry is replaceable by AI in light of all the issues it has caused leads me to believe this study is actually over-estimating at least some jobs.
  by hajile
• One thing I've been thinking about is Baumol's Cost Disease. It's usually applied in at a macro scale across an economy: when a job somewhere becomes much more efficient, it makes all other jobs comparatively more expensive.

  I've started to notice this on a micro scale as well. I am 20x more productive at coding and scripting, but I don't spend more of my time coding. My time is now filled with things like presentations, Jira, QA, etc. Ironically, despite all of the automation, the things that are painstaking and time consuming disproportionately require more time now.

  by legitster
• What if what we call "white collar work", for the most part sitting at a desk, interacting with a computer, isn't really the sort of work suitable for human flourishing at all?

  When you think about it humans were never really well suited to such a lifestyle. Of course the transition may be bumpy but maybe the end point, of machines doing these machine-like tasks, isn't so bad for us.

  What work does suit humans? Anything involving interaction with the world or other humans, perhaps.

  by pmg101
• I think “covered well” is being used here as a synonym for “covered at all”.

  For example, the “social workers & care -> counsel parents with child rearing problems” category is considered “well covered” because there are 13 prompts.

  It’s self-evident that the spectrum of work carried out by a social worker undertaking that work cannot be well represented by thirteen prompts.

  I suspect if you really strictly defined “covered well”, to the extent that any normal person would use that phrase, the headline would be above 99.9%.

  by OtherShrezzing
• Perhaps a version of the Doorman Fallacy, ie. replacing a doorman with an automatic door seems like an obvious win, but it turns out the doorman was performing a number of other vital functions (customer experience, for example) which weren't considered.
  by b800h