New AI tutor achieves 0.71-1.30 SD effect size in Dartmouth course [pdf]

Join the discussion

• Hacker News
• Too bad the educational use case doesn't make any money. Good LLMs are a game changer for people motivated to learn.
  by kubb
• Not sure if its education, but there is huge money in the college admissions process, e.g., SAT prep.
  by jasondigitized
• I don't want to learn from hallucinations where it will change its answers based on me questioning their teachings. I use it for conversations in a language I'm learning, but I quickly learned that asking it grammar questions for example is not a wise decision.
  by TheLML
• Wikipedia doesn’t make much money but is still helpful. LLMs don’t need to make a whole bunch of money to be helpful.
  by Robotbeat
• 0.71 SD effect size is impressive but i would be more convinced if they ran the control group against a TA who actually showed up to office hours.
  by luciana1u
• Very nicely typeset.
  by albinahlback
• Honestly whether or not this was effective seems less important to me than the adoption numbers.

  Text book reading in this course was 10-15% at baseline ... but this AI thing got 90% voluntary usage ungraded.

  Even if its worse per-hour than a textbook, you're now teaching 6x as many students _something_ instead of teaching a small minority everything.

  So really it just becomes an optimization problem at that point because most students are at least in the funnel/in the running to learn something.

  The paper kind of proves this itself ... they tweaked the quize formats mid-semester and where able to iterate which you can't do on a textbook that nobody opens in the first place

  by Rperry2174
• I'd argue the results are even better: just reading a textbook doesn't really teach you much. You have to do exercises, but they're expensive to create and grade. LLMs with a proper harness (see paper) tackle both.
  by baq
• Shocking that a well executed AI tutor improves outcomes.

  Hasn't computer assisted interactive learning already been proven for years? Why does there seem to be so much skepticism about enhancing it with AI?

  Is this just something like, astoundingly slow adoption or poor execution? Being held back by paper textbook makers? Teachers unions dragging their feet?

  How can interactive AI driven individually paced learning _not_ be obviously dramatically more effective?

  by ilaksh
• Lots of people in education will happily tell you how the past 15 years of tech integration has been a net negative.

  There ARE technologies that have improved things, but so much high-cost useless tech has been shoved into every level of education that many educators are incredibly leery of new tech.

  The issue is that while the underlying technology is useful, the way it gets integrated is frequently not. An administrator cuts a deal for a product they never have to use to an ed-tech giant for a huge amount. Because the ink is dry and a huge sum of money has been spent admins pressure educators to use the technology as much as possible regardless of outcome.

  In that context it makes a lot more sense why there is pushback and FUD among educators.

  by dghlsakjg
• its like anything else. benifits students that are already motivated to learn.

  very few are actually motivated to learn and are just there to get a job or its just next thing that they have to do in life.

  by dominotw
• Selection effects are extremely important in education. Dartmouth students have already had a large selection effect. If you try to apply this more broadly then it might not work.

  Motivation is also a huge part of the problem. I'm wondering if the novelty of the AI tutoring gets more people to try it and whether it would wear off?

  It's surprising to me that many students at Dartmouth don't read the textbook. You'd think college admissions would select for that?

  It seems promising but, as they say, more research needed.

  by skybrian
• In mice!

  Jk, but the skepticism is inevitable. I think we can be dubious about how AI mobilizes global capital while also appreciating tutoring as one of its best targeted use cases.

  by glenstein
• > the platform was adopted by 90.2% of enrolled students

  Then it is "effectiveness", not "efficacy". Prefer simpler and more specific words when possible, to reduce effort for the reader.

  by zkmon
• A lot of pessimism in the comments, but I am just happy that we are seeing some work towards bridging the 2 Sigma gap for regular education vs. elite private tutoring. I can't imagine that people assume it's the physical presence of the tutor that is making the difference, it has to come down to the personalisation and expertise which is exactly what AI can provide in a form. And yea it might not be "there" yet. But if we don't start trying and studying then it'll never get there.
  by klustregrif
• Tell me if I am oversimplifying, but I never understood the noise about the two sigma problem. Like, of course if you have a private tutor to immediately answer any question that pops into your head at the immediate moment you get confused, you are going to learn vastly more efficiently than in a large classroom where once you get confused you are likely to stay confused. To say nothing of how the pace will likely either drag way behind what you'd like, or accelerate too fast ahead of it.

  The environment is just obviously two sigma better. This just... seems obvious to me? In the same way that I will get stronger much faster if I have a physical trainer to tell me exactly what I am doing wrong when I do it? And it seems obviously unsolvable other than by getting everyone a private tutor (or AI..?).

  Asking from a place of curiosity.

  by johnfn
• This is super, but students will have access to AI during the test in real life, so it's ironically less realistic to remove it (thinking of the "... GPT-4 actually harmed subsequent performance by 17% when the tool was removed ..." part).

  I'm more curious how students perform on the test with vs. without AI.

  by RA_Fisher
• On a Dartmouth related note, I still can’t believe they abandoned Blitzmail for Office 365. What a loss.
  by riddlemethat
• Even if the research is flawed I'm happy they are trying this. They are taking advantage of LLMs to have less rigid tests and also give feedback.

  I think there is more potential applications possible with combining LLMs with reference/text books. Like how about an assistant that points you to the correct books/chapter/paragraph for the concept you need to understand better for a project you are working on? Or clarify any confusion you are having?

  Like a human tutor but infinitely patient and non-judgy + search engine.

  by bobajeff
• While there's some skepticism in the thread, I'm not particularly surprised if this is true. Children who can get human tutoring do a lot better. An LLM that can answer questions and patiently explain likely offers some benefit.

  What creeps me out about bringing LLM into early education is that it's a period where kids learn to socialize and cope with problems, and I do worry about forming substitute relationships with chatbots that are engineered for sycophancy / enablement. But I guess that's a problem either way, because almost every student will try an LLM at some point.

  by zerobees
• From my understanding the actual AI that was barely used. What was used was a quiz with an AI grader.
  by sarchertech
• Interesting, congrats.

  Are you planning on opening access to Phosphor?

  by constantius
• maybe they did already via the "formerly known as" comment in the paper?: https://www.spongium.org
  by thadk
• Wow, putting effort into training material, thoughtfully designing it, and relating the material to the final exam, will increase performance on said exam. So much AI so much wow. Like seriously. Most university statistics courses suck big time, so literally any effort put into them will Improve the field. I’m happy the authors want to improve education but they don’t seem to understand that preparing questionare style material is a confounding factor which could very well explain the better performance too… instead of cramming AI into the next thing. I’m generally opposed to AI on basic textbooks. You don’t want hallucinations imprinted on students who have no idea and can’t judge the quality of the generated text. Some things require effort, reading intro to statistics is one of them, and it’s for a reason, the effort IS the learning
  by usernametaken29
• Interesting article, wonder where we're going with this though, I find it's very difficult to keep LLMs on track and critical enough to be useful.

  Just want to say that:

  >In our deployment, student-reported reading completion baselines for MATH 010 were approximately 15%, with instructors estimating 10%. Individual student reports of reading compliance ranged from "literally no one does that" to "is this being recorded?"

  is hilarious

  by NeutralForest
• Do you have a larger study planned for the Fall? It definitely seems promising.

  I'm curious how well you feel this worked because the subject was Statistics (objective grading) versus something more subjective like Civics or Literature.

  PS - I'd say this qualifies for Show HN, too!

  Do you

  by boulos
• They were using Sonnet 4.6 for some fre form responses so that could be applied to something subjective.
  by ilaksh
• I currently study Multivariate Calculus by using very new and nodern method: I read the text book, while solving the examples of the general for,ulas, or try to come up with my own. Then I do a s$ht-ton of exercises. I only use LLM’s to quickly clarify confusing topics or notation, but not really much else. I cancelled my Claude subscription. Now I use just Mistral and local Vibethinker-3B, but they work just fine.

  Earlier I used Claude by giving it the course material and asking it to generate me exercises (our cpurse work went way over my head) and yeah i learned to differentiate a gradient or Jacobian, but it was very shallow - I knew the formulas, but not what they meant or how to apple them correctly. After I just filled glaring holes I had in Univariate Calculus by readong and doing, I actually started to understand something.

  Lon story short, in my experience Learning with LLM’s is ok with very unfamiliar material that is not too complex (there’s obvious problems of LLM’s themselves being pretty ghastly with maths sometimes), but at least it os not better than the traditional method of just putting your nose on the grimd stone.

  by delis-thumbs-7e
• In the Matrix movies you can upload complex knowledge to a person in seconds. If LLMs could help with this even a little it would be useful. But alas, people like myself who are slow learners still get overwhelmed when trying to learn too much or too fast without rest breaks.
  by raegis
• The article explicitly calls out selection bias (this is entirely based on 90% that opted into using the tutor, there was no control group), I wish the headline did as well. "Engaged students score 0.71 - 1.30 SD better in tests" sounds like a much simpler explanation.
  by or_am_i