Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • I've run some evals on my puzzle game https://redactle.net/llm-leaderboard

    Deepseek v4.1 flash is able to solve it some of the time. I've found it burns through more reasoning tokens than any other model. Google models like Gemini 3.8 Flash are still dominating and is able to one-shot most evals while being the cheapest.

    I'm curious what other unique evals people are running.

    by pampas
  • Got some fun if slightly janky looking pelicans out of this one: https://tools.simonwillison.net/markdown-svg-renderer?url=ht...

    I ran it on all seven reasoning levels supported by OpenRouter, but the reasoning token counts suggest to me that it doesn't actually support seven different levels. This is one of my biggest problems with OpenRouter - their abstraction layer makes reasoning levels harder to reason about.

      reasoning_level  reasoning_tokens

  none             0
  minimal          6,520
  low              11,873
  medium           5,678
  high             9,779
  xhigh            10,197
  max              13,386
    Update: explained here: https://api-docs.deepseek.com/guides/thinking_mode/

    That says it supports three levels - low, high, max, and maps them out like this:

      minimal   low
  low       low
  medium    high
  high      high
  xhigh     high
  max       max
  ultra     max
    (But it looks like "none" is a valid option too.)
    by simonw
  • Initial impressions: this is a really strong model and the fact that they reduced prices at the same time makes it an awesome backup model to use when your primary subscription runs out and you need to bridge a few days before it resets.

    It also seems to be more willing to just do whatever you ask of it. My favourite benchmark for this is to ask it to download a rom for an old game, that I own. Legal in my juristiction but the US models (except Grok) have a tendency to refuse it.

    by LaurensBER
  • I think it's very clear that DeepSeek is obviously the best AI lab in the world.

    Every model release seems like it packed with wonderful research and advancements.

    by impulser_
  • Already on HuggingFace: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash

    The bad news is that the original v4 flash was 284B, which was large but still somewhat reasonable for running locally. This one is 552B so almost twice that, so the huge gains in benchmark scores make sense - it's not really flash anymore, imo.

    I've no idea about actual performance vs benchmaxxing, though deepseek was fairly trustworthy as far as Chinese models go. If that holds (and if it doesn't think forever, as deepseek 4 sometimes did) it's probably the newest king of the hill amongst open weights models.

    It does include vision, and they do something funky with KV cache so it's very efficient: "[...] these designs reduce the global KV cache footprint to 890 bytes per token — roughly 1/4 of DeepSeek-V4-Flash". I do appreciate the high focus on efficiency, but at this point we sure could use a flash-flash version.

    @edit: I couldn't make sense what the actual parameter count is, with the addition of Engram memory. To my understanding the 4.1 flash is 552B parameters you want in vram or ram, out of which ~16B is active (8B for prefill). It also includes additional 196B Engram memory which you can put on an SSD. I think.

    Assuming that's correct 256 GB memory is insufficient to even load the model at q4 - you'd be 1GB short, assuming you can fill it to 100% (so no mac). You'd also want some for kv cache of course. A 256 GB desktop with some extra VRAM from GPU could run it, but normal consumer boards get real slow once you fill 4 slots so you'll probably want quad channel which is Threadripper or above territory.

    by revolvingthrow
  • I'm surprised more people aren't talking about the cache hit price: $0.003 per million tokens. I have a feeling that the price of 1 million tokens transmitted over the internet is more expensive than cache hit. Are we close to making the chat completion API obsolete because the cost of context transfer over network is going to dominate the task total cost?

    Here's the same token usage priced at different rates: a real long-running coding task, medium codebase, 447 turns.

      Input           1,026,957
  Output          164,667
  Cache read      36,554,368
    GPT-6-astra

      Type      Rate     Cost  Share
  Input   10.000   10.270    19%
  Output  50.000    8.233    15%
  Cache    1.000   36.554    66%
  Total            55.057   100%
    DeepSeek v4.1 Flash, $0.003 cache hit

      Type      Rate     Cost  Share
  Input    0.300    0.308    50%
  Output   1.200    0.198    32%
  Cache    0.003    0.110    18%
  Total             0.615   100%
    DeepSeek v4.1 Flash, $0.006 cache hit

      Type      Rate     Cost  Share
  Input    0.300    0.308    42%
  Output   1.200    0.198    27%
  Cache    0.006    0.219    30%
  Total             0.725   100%
    Hypothetical: same DeepSeek input/output rates, but cache priced so it accounts for 66% of the bill.

      Type      Rate     Cost  Share
  Input    0.300    0.308    21%
  Output   1.200    0.198    13%
  Cache    0.027    0.982    66%
  Total             1.487   100%
    This cache it improvement makes the model x2-x2.5 more efficient on a long horizon tasks in terms of cost.
    by k9294
  • As I also said on Twitter - it really amazes me how fearless Deepseek are. Every single model release is packed with new and crazy clever ideas and somehow, they always commit to training them at near frontier scale.

    I know everybody wants the tell all story of the clever ideas that were developed over the last ~3 years at Anthropic and OpenAI, but what I really want to thumb through is DeepSeek's notebook of "brilliant but didn't quite make the cut" ideas.

    They must be trying some truely bonkers stuff to be able to land this much architecture novelty in their full releases.

    by rao-v
  • It's so refreshing to see DeepSeek's tech report[1] full of juicy details; meanwhile, something like Fable's system card[2] is like 70% "safety", 10% "model welfare" to make sure little Claude isn't distressed, and 20% benchmark numbers.

    [1]: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/...

    [2]: https://www.anthropic.com/claude-fable-5-1-mythos-5-1-system...

    by kouteiheika

Explore Birbla archives