Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • I detect SPAM.
    by BobbyTables2
  • I didn't want to shell out $20 a month for the general thing, so I spent $90 on data collection and built my own for code comments specifically. It runs locally in your browser with a relatively small classification model trained on old-school stylometric features. You can try that before turning to Pangram for uncertain cases, if you wish.[1]

    It's easy to get high accuracy numbers if you're testing on long (50+ words) texts. Much harder when the documens are short, as code comments tend to be.[2]

    [1]: https://xkqr.org/aicomment

    [2]: https://entropicthoughts.com/better-ai-comment-classifier

    by kqr
  • Tools like Pangram can detect whether text is written by AI or a human with a probability non-negligibly different from 0.5. Therefore, the Turing test in its strictest form (that there must not exist such a distinguisher) has not been passed.
    by SAI_Peregrinus
  • Based on their methodology it looks like the accuracy figures (99.82% for Opus 5) are the true positive rate rather than a combined metric that factors in the false positive rate as well. They claim 1 in 10,000 but it would be nice if we had a per-model breakdown for that specific test.
    by mdspan
  • Gotta be honest. Pangram is not that reliable. It flagged my texts as 100% human made whilst at best it was 50%.
    by Codefrontier
  • Pangram is definitely at the forefront of AI/Human detection. To me though, I think that the bigger issue in a post-AI world is that human authorship (if it is relevant and can be proven) is only part of the ultimate problem. The baseline of Common AI output really can't be ignored -- content to be meaningful needs to do something to expand the idea spade beyond AI. Humans can be original and NOT exceed that baseline. Moreover, AI detection penalizes people who use AI as an adjunct to their individual talents to create new things. To me then, the point is not whether Pangram works well -- it's whether it answers the right question.
    by jonathan_aberma
  • I’m surprised at the current sentiment in the comments. Pangram is amazing and has really interesting engineering too. I would have guessed that reliably identifying LLM generated text was not possible without watermarks.
    by jdc-pub
  • People expect a binary response, is it AI generated yes or no. But it's more complicated than that. For example, if you see an emdash, it's probably AI generated. But it can also mean the author used it for fixing grammar or tenses. LLMs can't help but try to help. The same for it's not X, but Y. Sure it's a known pattern, but it's not like people don't use this trope all the time.

    In my experience, Pangram is great for detecting an author who is trying to pass someone else's work as theirs, or if they are tackling a subject they have little to no knowledge in.

    by firefoxd

Explore Birbla archives