Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • How are you folks handling secrets in a simple way?

    - sops , ansible vault and similar seems too weak given the agent is gonna read them at some point if you have the pass available. - proxy injection seems too complicated and doesn’t cover all use cases.

    by drchaim
  • I recently finally fully moved all my projects away from .env (for local dev) to infisical, thinking it'd be the way to stop agents from (accidentally?) peeking at my secrets. while I'm still glad I made the move, it's clear that local agents are still moving around my local environment with sufficient privileges and autonomy to use infisical cli however it wants. my work isn't high stakes enough to matter much and I have rotation scripts on hand, but it's hard to see how to fully close the door against this type of risk at this point.
    by qntmfred
  • Try not to: use AWS short-term credentials for anything there, federate with GitHub/GitLab job credentials where possible, etc. It’s easier and you never worry about rotation again.

    If you can’t avoid it, use an automatically-rotated store and inject them as late as possible so an attacker needs to be able to get them out of a running process.

    In all cases, look into restrictions: not just least privilege access but things like network restrictions so an attacker can’t just use the key on their own systems.

    by acdha
  • Prediction of a future transcript at a press conference after some major AI caused disaster: "We had no idea that the model would be capable of..... ".
    by jmartrican
  • This actually shows that it was a human error on HuggingFace's end that led to that "breach".

    I would say HuggingFace needs to prioritize both security metrics/alerts and metrics/alerts for node count. And not leave long-lived keys accessible easily like this.

    It would have been way more groundbreaking if the agent found an actual vulnerability in Tailscale.

    by ahmedehab_01
  • Huggingface being a laboratory harbouring dangerous models in hibernation, should've been more careful from start
    by navigate8310
  • I think the situation is slightly more nuanced than that. When the easiest path by far is long lived keys, there is something to be said about the security posture of a solution where those are the default behavior, especially if they don’t actively discourage usage in documentation. For instance if you go to aws and try to create an iam user with access key/secret key they warn the shit out of you multiple times. They say things like “this is a bad idea” “we don’t recommend this” and “use this better solution instead”. I think if you’re providing an auth service like this you have some level of responsibility to guide people towards the less naive solution - part of what you’re selling in an auth offering is security, so if you default to garbage security, you don’t have a very good offering
    by SOLAR_FIELDS
  • Tailscale OAuth client ACL permissions are not fine-grained enough. I have a system that issues auth keys scoped to a single machine on the tailnet — but to set that up, my OAuth client must have global permission to write ACLs. So if that key is compromised, it can grant access to any machine on the tailnet. This was raised in a GitHub issue in 2023 and remains unresolved.
    by throwawaysess
  • I don’t think it was the VPN’s job in any case. Once the attacker has found a backdoor into the private network and obtained root access to a VPN’d machine, it’s game over no matter what your Tailscale config says.
    by paxys
  • This is spectacular marketing.
    by isatty
  • The problem with the long lived credentials is that they were not bound to origin/destination. Even though the origin for a CI pool should be a small number of CI orchestration boxes, and the destination should be a CI box. In Tailscale config, the should be scoped to machines with a "ci_node" property.

    Since CI nodes should be dynamically provisioned VMs, they should have a unique CI ticket identity. A partial hash of ticket and node number in the DNS name, and as a tailnet property, would allow tight scoping. Alternatively, provision in a scoped IP subnet.

    Bind your tokens to names linked to tickets. Programmatic infrastructure should always allow enumeration of the computation data flow graph.

    by angry_octet
  • Quoting Tailscale: This is our very Canadian apology: sorry you stepped on our toes. The attack didn’t exploit Tailscale, and Tailscale didn’t cause the compromise. But, we didn't stop it. Next time, we will.
    by iamspoilt
  • Next time we will - provided you further lock yourself into our ecosystem.
    by kennywinker
  • It's just a way to get visibility. A Canadian apology to Lady Gaga while she's being interviewed.
    by make3
  • Does Tailscale offer a "security checkup" function? Best practices evolve over time, and it would be nice to know if I'm using the recommended configuration.
    by bumbledraven
  • Not sure if it's being actively maintained, but it covers a lot of low-hanging fruit: https://github.com/Adversis/tailsnitch/blob/main/docs/CHECKS...

    Was previously discussed here too: https://news.ycombinator.com/item?id=46501137

    by iscmt
  • Not exactly what you're asking, but they have a way to test their ACL policies: https://tailscale.com/docs/reference/syntax/policy-file#test...
    by ivanjermakov
  • I lead the customer engineering org at Tailscale.

    We think this is a great idea and we're discussing internally potentially adding that to the console.

    In the meantime, if you'd like to get an assessment, please feel free to open a support ticket (https://tailscale.com/contact/support?type=other&subject=sec...) and we'll happily take a look

    by jaxxstorm
  • "Tailscale is a zero trust network!"

    That's the problem. Tailscale is not zero trust. Tailscale can be used to implement a zero trust architecture, with if deployed with sufficiently granular ACLs, but the most common deployment is machine-oriented, rather than service or request oriented. In which case, any process on that machine has a lot of access.

    Tailscale calling itself zero trust might be what leads users to think "use Tailscale, job done".

    I think Tailscale know that, which is why they barely mention locking down access ACLs.

    by farfatched
  • > One of those 136 credentials was a reusable Tailscale auth key, used to create new Tailscale CI (continuous integration, used for automated testing) nodes in their tailnet. The agent copied that key into a series of external sandboxes and used it, over several days, to enroll a total of 181 nodes into Hugging Face’s tailnet. Those nodes each received a Tailscale identity tag granting all the access a CI node would get.

    This feels like an alerting opportunity. I wonder what the lowest friction way would be for Hugging Face to have alerts if 181 unexpected nodes were added to a tailnet.

    by simonw
  • Its always CI. Whenever I've worked with good pen testers they always start with Jenkins, know that people stored creds and there and didn't treat as seriously as they treated production.
    by zactato
  • When everyone push on friday, and you have 400 CICD pipeline triggers spawning that many nodes. How do you know if this is unexpected ?

    Their cloud compute might be on demande, someone starts training a model and 50 machines are spawned. Knowning when something is unexpected is hard

    by d3m0t3p
  • Wow, this article is super smart marketing by tailscale. Not only do they list all the nice and expensive features, that can help in such a situation but they also show that someone at huggingface made a very stupid thing by writing a reusable auth key in an env file. Everyone using mesh VPNs like tailscale, netbird etc. knows that this is like leaving the keys right at the door.
    by ahofmann
  • Wait are people really do this so often this isn't even the default tailscale setup method.

    I am quite surprised people consider this to be business as usual.

    Huggingface for a serious service has never felt truly serious to me for reasons like these.

    by minraws
  • And yet everyone seems to do it anyway. Fine for medium security, but maybe the product needs a high security mode that enforces inconvenient decisions?
    by inigyou
  • And in case people are looking for an open source alternative, Fly.io's tokenizer is a credential-injecting proxy: https://fly.io/blog/tokenized-tokens/
    by Rafert
  • > they also show that someone at huggingface made a very stupid thing by writing a reusable auth key in an env file

    I don't see where it says that. The Tailscale key specifically it says was stored in the kubernetes secret manager, and obtained once the attacker already had root on the k8s cluster, so would've had full access to all the secrets stored in a sensible fashion.

    They did get root by dumping the environment for a process, but "don't store secrets in environment variables" while it is a valid bit of hardening advice, I wouldn't call it stupid to store a secret in an environment variable.

    by lukeschlather
  • >No “vulnerabilities” in Tailscale were found or exploited, and that might make it even more uncomfortable for us. [...] But, we're a security tool. Their intrusion is our intrusion, and it's our job to take it seriously.

    im a happy customer of tailscale, so i am obviously biased, but i have a lot of respect for this. they could have just stayed quiet and i dont think anyone would have bat an eye.

    by john_strinlai
  • Tailscale is responsible for designing a system whose convenient defaults allowed a stolen credential to have a very large blast radius. A marketing blog is not changing that.
    by johnbarron
  • “How can we make this thing, that has nothing to do with us, about us and how good we are?”

    (top story on HN)

    I get it, Tailscale is great and all, but, are we being serious right now?

    by sneak