Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- To corroborate this - I found pretty similar rates of AI-flagged papers over time from running pangram on ArXiv (abstracts) in a physics subfield, which is currently at about 25% as of April (write up here: https://peterse.github.io/2026/06/15/The-rising-tide-pt1.htm...).
Its not clear from your writeup what threshold needs to be reached to be classified as "machine written". A preprint where half the text is human and half is 100% AI should be a different category than a preprint where 100% of the text is AI-assisted.
Also its cool that you're making the detector available. When you say "cheap to run", do you know how this compares to pricing for a commercial detector pangram or GPTZero?
by epq22 - Something I've thought about a lot is that there is having someone with some domain knowledge or reason to care a lot about a particular issue spend a bunch of tokens and cycles on it until something useful comes out the other end. The most obvious ones are the math problems that have been coming out and help push the frontier of various areas of math. Another example is taking all of the public NYC open data ecosystem and crunching it to get some value which I have spent a lot of time and tokens on but not found a great medium to share.
The question is just how to organize these outputs and conclusions in a way that is consistently reproducible and also how to correct errors or remove LLM nonsense where it refuses to take a position on something.
Before it made sense to do this in papers but it feels like we need something like a paper format.. that is fully reproducible ideally and optimized for aggregating knowledge in a better way. I.e. before a person spent months on one of these and there was just more filtering, and the output itself was a clear signal of time spent and effort that no longer exists.
by ianm218 - Yes, I’ve thought about this too. The strength of these models is that there is a lot more knowledge encoded in them than the average scientist has in mind at any given time. That means they can explore many more possible combinations of concepts.
If we imagine a set of all human ideas that these models have access to, then the set of possible discoveries would be something like the superset of all possible combinations of those ideas. I think all LLM discoveries are bounded by that space.
Looking at the recent OpenAI math discoveries, that seems to be pretty much what happened. Existing ideas were used as building blocks, the model found a valuable combination, and the result was something new that had real value.
- One challenge with this approach is: could it be possible that the detector is simply learning to recognize words and jargon used more in the literature post 2022 as 'AI'?
For example, LLMs love to talk about LLMs (and the people who write with LLMs love to write about LLMs). Could "large language model" itself therefore be flagged as an AI-like phrase by this approach? It didn't exist much in the literature before 2022, does now, and certainly does more in AI-generated text: but, it is not actually a great way to distinguish modern AI generated text from human written text.
A helpful control would be to show that on some cohort of papers that can be declared reasonably clean of LLM generated text post 2023 there are very low rates compared to the arxiv.
For example, while papers in the journals Nature and Science are unlikely to be entirely LLM free at this point, if those were tested through 2026, we should see a line significantly lower than the arxiv's growth.
- You raise a valid point. Just by intuition I'd say if this were true, it would probably just be a small fraction of the actual flagged articles. I will still look into how I can mitigate this when I update the detector.
The difficulty with this is then: How do you get a clean post 2023 dataset? I have no straightforward idea for this. You can't use other AI detectors to build it because then you'd never outperform them.
- I work with several researchers (primarily working on consensus protocols etc).
We were discussing research in general and I asked them: "Do you prefer the writing of the papers or the research?"
They, almost unanimously, agreed that they preferred the research. This makes sense as if they preferred writing they probably would have chosen another profession.
I say this b/c having LLMs available to turn research diagrams, code etc into a paper (or at least the starting point of a paper) will probably lead to MORE quality research papers. This is b/c I'm sure there was some friction in a researcher's mind of "I would love to do the research on this but don't want the trouble of writing the paper".
Put another way: on a 2D plot with one axis being the skills as a researcher and the other being hatred of writing, LLMs may "unlock" the people high on both axes to get more papers out.
Post Script: I agree that this could also lead to more BAD papers but the net may turn out to be positive in the long run.
by alexpotato - Just to play devils advocate. These kind of papers are very verbose and boilerplate. I can imagine using AI to write 90% but then the actual novel content and explaining what’s important could be handwritten.
Perhaps that’s what’s happening.
by bilsbie - If 90% of it is boilerplate, then you really need to question if you have something worth publishing as an academic paper.by breezybottom
- Yes, this is the thing. In the scientific enterprise, the writing is mostly wasted time. Why would you write it up if machines can do it competently?
Sure, some people are artists - but most aren't.
- I've looked at this same problem in an academic environment and have come to conclude that there is no way to reliably detect AI writing using only text. The reason for this is that no detector can take two identical inputs and classify one as synthetic (LLM) and one as organic (Human) and this situation can easily happen at the sentence or even paragraph level. Posed as a question, if a human writes a paragraph that happens to be exactly the same as a paragraph written by an LLM how do you classify that paragraph? There are a finite number of words and a finite way of combining them within an academic setting/field which means that we can't build a perfect classifier with just text. In practice, there are obvious tells and LLM-isms but these also change with time and each model has different tells so that even if a paper is full of LLM like writing we have no way to disambiguate between organic text that appears synthetic or synthetic text that appears organic.
A more interesting question, to me, is looking at a corpus of essays and analyzing how writing has changed with the introduction of LLMs. We can look at changes in vocabulary, linguistic features, style embeddings, regular embeddings, typos, errors, and references over time. When looked at in this way it is clear that academic writing has changed at the population level but what has led the change is harder to track down.
by Eextra953 - > There are a finite number of words and a finite way of combining them within an academic setting/field which means that we can't build a perfect classifier with just text.
Maybe for very short phrases, but otherwise I disagree. Phrasing very quickly runs into a combinatorial explosion. In the words of Noam Chomsky, "Virtually every sentence that a person utters or understands is a brand-new combination of words, appearing for the first time in the history of the universe."
In my opinion, the difficulty in LLM/human text discrimination isn't that a person might coincidentally write exactly the same text as an LLM would, but rather that 1) LLMs aren't hard locked to a single phrasing (so this is a tougher problem than matching to a single static document, e.g. plagiarism detection) and 2) text has relatively low information density, so you need quite a bit of it to gather enough data to run a statistical test with a reasonably narrow confidence interval.
by Calavar - When some closed models are "retired", interestingly we might not even be able to extract all their tells. We'd be limited to people writing about them on forums (where they don't often specify version, and there's a lot of memes and confirmation bias) and I don't know, doing some statistical Bayesian guessing that given the likelihood it's FooGPT 10.5, maybe we should update our beliefs about its characteristics etc.by stereolambda
- > If a tool marks 40% of new papers as machine-written but also marks 20% of papers written before ChatGPT existed, the real story is the 20% nobody mentioned.
When 65% of the papers you read have the characteristics of being AI written, whether or not you use AI to write, your writing will be influenced by the AI style. I imagine this must be particularly the case for newbie researchers who are still developing their writing style
by willquack - I might be missing something but what’s the real story of the 20%.
To me it sounds like 1. Either your tool is just not that good and reliable as you thought, 2. AI is trained on human written articles, so some of that human written content informed the now established “AI slop”.
There are people who shipped “slop” before AI.
by serial_dev - even if your writing is influenced by "the AI style," it would likely only show up in patches, a word or rhetorical flourish maybe, but AI-generated text doesn't "sound like AI" in patches, it does so consistently across the entirety of the output. it's very hard for a person to maintain the syntactical patterns of AI writing over an extended period of time unless they know exactly what those are -- probably more knowledge than any person currently has, really -- and are applying a level of detail to each clause comparable to forging a painting.
there's also the fact that "the AI style" has changed over time. for instance, the word "delve" is notorious as an "AI sign," which it was up until mid-2024, at which point it dropped off sharply and has now basically disappeared from LLM output. so if someone happened to pick that up due to reading it everywhere, their writing is now less characteristic of AI, not more.
- I am not a native English speaker. This is not surprising to me. I think most of the papers we write would be flagged by AI detectors.
It is not because we ask LLM to write us a paper about X. It is because we are bad at writing in a scientific style, and american editors expect us to do it. With LLMs, we can write in basic sentences and tell the LLM the idea and it converts that to nice writing.
If you write each paragraph and have an LLM make that paragraph more scientific, it is entirely your paper, but it is flagged as LLM generated. If you have an LLM write the paper but speak good enough English, you can make it look human even though it is not human.
- most papers like this are flagged by AI detectors because, by your own admission, AI was used to generate the final output. what exactly did you expect here?
- My English is fine, and I did hand-write my MSc thesis in English, but scientific style writing is very tedious. Personally I don't think I'd even consider writing a paper "by hand" if I were to write another one.
There really is no point, as long as you verify the content matches your intent and edit out anything poorly written.
Frankly, I've read plenty of papers by native English-speakers over the years that'd strongly benefit from being rewritten by an LLM too...
by vidarh - Yes that is exactly what I suspected. And I see absolutely nothing wrong with this.
- There is a fine line between using an LLM to clean up grammar and spelling and using it for text generation.by bjourne
- It seems that LLMs often insert new ideas even when asked only to polish writing without changing it: https://bsky.app/profile/tomerullman.bsky.social/post/3mq33c...
I don't think this is surprising. Good technical writing is very precise. If you're starting from non-technical writing, I suspect that in most cases you can't make it "sound scientific" without adding new claims or changing the meaning. (Maybe you are being more careful, but this is something that worries me in general.)
- > If you write each paragraph and have an LLM make that paragraph more scientific, it is entirely your paper
"More scientific" is not some merely stylistic thing that faithfully preserves the original meaning of what you wrote. The precise details of each paragraph matters a lot in terms of what and how it communicates. The fact that these details do matter means that, according to my accounting, it is not entirely your paper.
Also, I find striving for "scientific" to be a pretty undesirable thing. Why should papers read like that? What is the benefit? The best papers (in terms of their writing and communication) are unpretentious and conversational. I'm pretty sure I'd prefer your "basic sentences", especially if they were wholly yours. (I understand that there are also external forces at play here as you mentioned.)
by zfnmxt - As with all text-only AI detection schemes, I am concerned about the accuracy of the detection. I'm skeptical of the methodology, specifically the final join of the three detector scores---how can you be sure that that final step does not introduce any biases? There's no source available, so it's difficult to tell exactly how this works or reproduce the research.
I've also uploaded text samples from my own (unreleased) research from pre-LLM era, and it's seemingly scoring pretty high on the LLM-detection scores. On other papers, nearly every sentence is highlighted as red "machine-leaning," but that does not impact the score? Additionally, there are dramatic differences between the scores for identical text with and without LaTeX formatting, despite the fact that it should not matter.
The takeaway from this should be "it is difficult to detect generated text and we should be careful about accepting results simply because they confirm a hypothesis."
--
Relatedly, the text above scores as highly machine-written, despite the fact that I just wrote it with my human hands, I promise :)
by pkage - The most concerning is the use of these tools to detect cheaters in schools.by make3
- There some real game theory mechanics at play when it comes to LLM usage in corporations. Devs are cranking out superficially superior code and documentation by just aiming the Claude Code fire hose at everything they can. Leadership encourages this because from what they can tell, there is no downside.
It's hard to say if this code is structurally better or worse than before, but it's certainly voluminous and as far as leadership could ever tell, with their flawed metrics, that is all that matters. It will be years before we figure out if this is a good idea and worth the cognitive atrophy.
Anyone not using LLMs all day is just not going to be as prolific. I can't imagine that the same factors aren't at play in the scientific research community where it's all about how much you can publish.
by ryandvm - > Leadership encourages this because from what they can tell, there is no downside.
This was my big fear before we saw price increases. Now I'm pinning all my hopes on AI being too expensive to justify further big corporate pushes. (Sigh.) I love having new tools, but I hate being pushed to use ______ tool to meet some managerial metric.
- > It's hard to say if this code is structurally better or worse than before,
Huh?
But that's easy to tell?
It's better then incompetents code, worse then a motivated average dev... But good enough hence the real question is value aka time& money invested/quality.
That's much harder to tell, and I currently think opus/fable generated code is decent enough to be safely in favor - at least on subscription
by ffsm8 - > It's hard to say if this code is structurally better or worse than before, but it's certainly voluminous and as far as leadership could ever tell, with their flawed metrics, that is all that matters.
Hah. I've been working with ChatGPT-5.6 Sol and a lot of guidance to port a particular geometry from 2800 lines of Python-scripted Rhino3D (basically a custom parametric CAD kernel in there, plus use of it) to Python-scripted OCCT/FreeCAD (an existing CAD kernel) and it's up to 36,000 lines and only about half way there. And I've been setting goals and targets for duplication minimization, code size reduction, etc. The results are fine for my purposes, but if I put a positive value on "voluminous" I'd be super impressed; with my actual negative value assignment on lines of code, this is more "absolute crap but still useful to me."
I think "absolute crap but still useful to me" is a pretty high value and not worth neglecting, but I'm definitely scared by leadership who sees the toilet overflowng and assumes that means that value has been maximized.
by addaon - >I can't imagine that the same factors aren't at play in the scientific research community
I can't speak for the entire scientific research community, but I can say that for basic biomedical research (not clinical research), LLMs are mostly ignored. They simply don't have the ability to parse most raw wet lab data.
This obviously has not stopped the creation of a legion of startups, new Asst Profs, etc, claiming that they are using "AI" to crack new frontiers. In practice, the most effective of these use machine learning, rather than LLMs.
by pcrh - IMO, cognitive athrophy is the point. If humans become dependent on AI, those corporations will get enough leverage to change laws in their favor. For example, people don't want a huge data center in the middle of their city, but the corporations will say that the same people can't do anything without AI powered by this data center. That's why I think that making humans deficient and dependent on AI is the entire point.by akomtu