Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • Looks incredibly cool, although it ran at ~4 tok/s on my browser (I assume because I've disabled V8). I believe your numbers though. I always wanted to try using something like this to create an AI assistant for my phone. If you try to create something without AI, you'll quickly run into the problem of using NLP to map input to the equivalent of AI tool calls. On the other hand, shelling out to a full-blown LLM for every input is slow, costly and if you want it local, greatly restricts its computing power and the number of devices it can run on. Needle and similar models solve this problem nicely: they're incredibly cheap and handle NLP (not in a predictable way of course, but with great success rates), so you can quickly run predefined actions or shell out to a proper LLM for more complicated tasks. You have the best of two worlds.

    You may have inspired me to give it a try!

    by a022311
  • Please give it a try and let us know!
    by HenryNdubuaku
  • 310 tok/sec on my browser, hm.
    by stavros
  • Nice. I used Needle as a router in https://rcarmo.github.io/projects/memento/, need to take a look at this
    by rcarmo
  • Insane!
    by HenryNdubuaku
  • This is cool!

    While most of the industry focuses on the frontier of “intelligence” (function), a release like this represents the frontier of the other end of the spectrum (form).

    Both are important if we ever want to see “Opus-level” capability running locally on commodity machines in the future.

    by redrix
  • thanks!
    by HenryNdubuaku
  • agreed, this is where we have the biggest opportunity for innovation.
    by dalemhurley
  • Can you share more about the architectural/design tradeoffs you considered or decided upon? Particularly for me, why is a model that is intended mainly to just make tool calls and marshal the results back focusing on speed? Speed as an inherent result of small size, I get, but speed as a design focus confuses me because it’s simply not going to be dealing with large outputs as a rule, wouldn’t it be better to trade some of that raw speed for better smarts?

    For example, I mocked a dumbed down version of what would be a reasonable intermediate tool call prompt:

    > It's currently 58 degrees. User asks for house to be 8.5 degrees warmer. What temperature to set thermostat to?

    The reply?

    Reasoning: “User asks for temperature to set thermostat to 8.5 -> set_thermostat with temperature=8.5.”

    Sounds like something Siri would do!

    by ComputerGuru
  • Could someone please share how such open source micro-LLMs might have been created?

    Do the creators take something like DeepSeek, and then delete most of the neurons to whittle down the size?

    by profsummergig
  • Another option for something this small and narrowly specialized could be to get traditional LLMs to synthesize the training data. Model collapse is probably less of an issue at this size relative to terabyte sized models.
    by hgoel
  • Technically, you could do that, but we trained this one from the ground up!
    by HenryNdubuaku
  • There's a Manning book on creating your own LLM from scratch which answers your question exactly. There's another book from the same publisher specifically about small language models for specialty purposes.
    by anigbrowl
  • Specifying units seems to be unreliable; I tried adding a description to the set_thermostat temperature:

        "temperature": {
      "type": "number",
      "description": "degrees Fahrenheit"
    },
    Set the living room temperature to 70 degrees Celsius

        {
      "function_calls": [
        {
          "name": "set_thermostat",
          "arguments": {
            "room": "living room",
            "temperature": 70,
            "mode": "cool"
          }
        }
      ],
      "confidence": 0.6045
    }
    Set the living room temperature to 70 degrees Fahrenheit

        {
      "function_calls": [
        {
          "name": "set_thermostat",
          "arguments": {
            "room": "living room",
            "temperature": 70,
            "mode": "heat"
          }
        }
      ],
      "confidence": 0.4536
    }
    Set the living room temperature to 70 degrees

        {
      "function_calls": [
        {
          "name": "set_thermostat",
          "arguments": {
            "room": "living room",
            "temperature": 70
          }
        }
      ],
      "confidence": 0.8517
    }
    Trying "in degrees Fahrenheit" for the tool description had similarly counterintuitive confidences.

    Edit: to be clear, the counterintuitive behavior is that the confidence ended up higher for the wrong units.

    by wky
  • Amazing project. Have you guys thought about running it or adapting it for FPGA-like approaches like this one below that reach Taalas-speed like 20k tok/sec on cheap hardware?

    https://www.mikeayles.com/blog/on-chip-llm-kv260/

    500 tok/sec is great but this would be 40x and it changes the kind of tasks that it could even be used for if you link it to fast tools too.

    by gbxk
  • That's really cool - I was already thinking of compressing `functiongemma-270m-it` down to 1-2 bits so it would work flawlessly in the browser. Your `Fine-tuning` feature is even much more convenient.
    by arthuqa
  • Thanks, give it a splin!
    by HenryNdubuaku
  • Funny result from the web demo. I'm well aware that it's an extremely small and, well, stupid, model, but even so:

    Query: HN

    Result:

    { "function_calls": [ { "name": "lock_door", "arguments": { "door": "front door" } } ], "reasoning": "User wants to lock the door. No specific door mentioned, so use 'front door' as default.", "confidence": 0 }

    I'd expect it to at least ignore (call no tools) for the queries that it doesn't understand. And it seems like it does do that, just not consistently.

    by Tiberium
  • This is exactly why the confidence feature was introduced, the model knows when its wrong, we could hide that part and return a placeholder "sorry I only do function calls", would that be better or you prefer to see everything?
    by HenryNdubuaku
  • Was that the first message you sent it?
    by Schiendelman
  • yeah I got the same, almost like its biased heavily towards that as the 0 ranking -- my prompt was just the word 'potato'
    by hmokiguess
  • I've got an identical output with the prompt "do not lock the door".
    by plingbang
  • It seems to love locking doors. I was hoping this would dim the lights.

    > I'm hungover

    { "function_calls": [ { "name": "lock_door", "arguments": { "door": "front door" } } ], "reasoning": "User wants to lock the door. 'hungover' implies a security door. No specific door named, so use 'front door' as default.", "confidence": 0 }

    by kristjan
  • no, this is the appropriate response to hearing the words "HN" :P
    by jszymborski
  • "confidence": 0, so I guess you could threshold it
    by petu
  • The website says the model is for "tool calling, device use, and structured extraction". Your example just doesn't seem to be very relevant. FWIW, it did a pretty good job for tool calling when I tried it, and I think it could be pretty nice to have this running on locally and integrate with Home Assistant.
    by yoavm
  • My first query:

    > Make it a little warmer in here.

    The reply:

    > "name": "set_thermostat", > "arguments": { > "temperature": 65, > "mode": "cool", > ... > "reasoning": "'warmer' implies need for cooling; set_thermostat with temperature 65 (typical warmth) and mode 'cool'.",

    Maybe I'm doing it wrong?

    by dbeardsl
  • but wouldn't it need to know the current temp for that to work? my understanding was that the model is best at tool calling
    by victor_edka
  • It's not a conversational model. It's meant as a local tool calling model.
    by dannyw
  • Try asking it to set the thermostat to a value. It's a very small model, there's not much reasoning capability
    by owebmaster