Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • this - if trustworthy - would be great for social media and "news" sites
    by stevenalowe
  • exactly, that is one of the use cases I see for these skills
    by skorniienko
  • That's too much effort. Should have been a browser extension that uses ollama so it can run with local models or maybe even openrouter...
    by cawingcrow
  • That's fair. Install is a one time real friction and extension would be much simpler.

    But, there is a reason why it's not an extension. At least for now. Whole product isn't a UI or service, it's a set of skills for your AI agent of choice, which gets content from URL or whatever shared as a source, splits it into claims and does extensive web searches for every claim to compare it with statement from provided content.

    On local models constraint isn't a model itself - but search. Model don't judge from it's trained memory, so even local model will need a backend for search, otherwise it can't provide a verdict.

    by skorniienko
  • I don't even watch most videos anymore. YouTube app -> Share -> Gemini -> Summary. Only if that shows interesting content do I actually watch.
    by leobg
  • And you are not alone, that is one of the purposes why I am building this
    by skorniienko
  • Who checks the checkers?
    by N_Lens
  • At the moment - it's model's judgement upon search results for every claim from provided content. If search results returned inaccurate data, or were fabricated - it might affect a verdict for that claim, or highlight that it's unverifiable or contradictory.

    But there is a room for improvement, what do you suggest? Have a Judge agent which will check results?

    by skorniienko
  • me. I volunteer as ultimate arbiter of Truth.
    by qntmfred
  • I like the idea! How can you let folks know they're being fed nonsense before they even finish the intro?
    by h2aichat
  • can't, at least with this design. Report itself takes minutes to proof-check. Realistic version is some kind of extension/plugin idea from earlier in a thread, but I'm not there yet
    by skorniienko
  • For truth checking actual thinking and logic is required. It doesn't seem to be a proper tool for the task.
    by madmaniak
  • True, and it doesn't do logic. It decomposes content into claims and search for every claim citations. It catches a false premise, not a bad inference. Someone with valid reasoning from true facts to a wrong conclusion goes straight through.
    by skorniienko
  • Washington Post gave up tracking Trump lies last term in 2021 because it became impossible by human hands with 21+ per day and over 30,000 in their database

    but with "AI" now it's possible not only to do non-stop but in REALTIME

    you could even just restrict the source of the check to the paper's own reporting the past fifty years

    * https://www.washingtonpost.com/graphics/politics/trump-claim...

    I'd like to see that backfilled, all the way back to the "long form birth certificate" (remember that horror show)

    by ck2
  • I'm not sure AI helps for that particular task. If you just assume that 100% of everything he says is a lie you'll be 99% correct. Anyone who believes anything he says at this point is living in a constructed reality where facts never matter. Fact checking seems more useful for less pathological cases where it might make a difference.
    by BugsJustFindMe
  • > Long output? Redirect to a file and read it from there:

    Hmm, smells AI generated. Why should an LLM (this is from a SKILL.md) care about $LINES?

    by lynx97
  • Done, fixed in v0.4.1. Reason why agent reads from a file - it can do it in chunks or pass the path to subagent and keep it out from own context. I've tested it on 3hrs long interview YT video - worked smoothly.
    by skorniienko
  • I think this would have a much bigger impact as short-form video content such as TikTok, Reels or Shorts. That's where most people consume this kind of content today, not in a .md file.

    What would really set it apart is adding fact-checking before publishing and letting it post directly to TikTok. That combination would make it something I'd actually use every day.

    by alealvarezarg
  • Not sure that I get your idea about short-form content. It already works with tiktok or yt shorts, especially if it has generated transcription/captions. Otherwise, skill will download video locally and ask your permission to run local models like whisper to transcribe video.

    Checking before you publish - that's interesting usecase I haven't thought about. Nothing stops you to fact-check your own content and decide what to do with it. Point is a report itself not more video content from it

    by skorniienko
  • Can someone please use this skill against the claims in its own repo?
    by egeozcan
  • Done, fixed 2 misleading lines in README. You can see example report in repo examples.

    Thanks for that, btw! At least someone questioned it :)

    by skorniienko
  • What do you use as “ground truth”? The page says “independent sources”, and I’m sure there’s too many to list, but my question is how are they vetted as being truthful and how are two sources with opposite viewpoints reconciled?
    by mysterydip
  • I could imagine running multiple different AIs seeing if they agree if something is true.

    I imagine this is similar to how Twitter's/X's community notes work. Something along the lines of when you have two accounts that disagree traditionally and they agree on something. That's how you know it's likely to be true.

    So from that approach, you actually want to have two AIs that are on the other side of the spectrum of whatever you're trying to find out if it's true e.g. if a conservative AI and a liberal AI both agree that something is false, it's highly likely to be false.

    by autonomousErwin
  • Something I've found surprisingly effective is telling ChatGPT to "use credible sources" - you can then watch its thinking trace and see it do things like ruling out random blogs, considering media publications with a good reputation for fact checking, and double-checking information that seems unlikely.
    by simonw
  • There is no whitelist source. It can't rate sources for truthfulness. When sources conflicts - skill drops verdict to misleading or unverifiable and both are linked. Can't pick a winner at the moment. That's probably the weakest part and still a judgement call for a model.
    by skorniienko