Comments
Hacker News
You’ll often find that problems aren’t solved by the people who would find themselves out of work if they solved them.
> Why did they stop and take no further action?
It costs them nothing to blame the user for clicking links that they don't trust, even though they're hosting the links...
by nubinetwork
"security" is a cost-center, it does not generate revenue. Incidents from lack of "security" need to have a greater impact on revenue before the typical corporate entity spends money.
by nickphx
GitHub today isn't GitHub from the early-mid 2010's. Today it is a Microsoft side-gig, something they bought simply because nobody wanted theirs. But, GitHub makes good money, and turns out they're a great source of training material.
They're just limping along. In reality, I think Microsoft wants to kill the GitHub brand, they're just doing it slowly, feeding poison.
by sscaryterry
I generally believe in hanlons razor (assume stupidity rather than maliciousness), it's likely that github saw the easiest possible solution rather than diving deeper into the cause of the problem to fix it permanently
Interesting article but doesn't seem to understand how security teams work or even how to address a threat outside of an brittle easily bypassed response.
The initial report was investigated and resolved, with the removal of specific repositories. GitHub likely lacks ability to prevent the attacker from repeating the attack (this is potentially a hard technical problem to solve depending on the infrastructure, available telemetry and how the threat actor is performing their attack).
Once that initial report was resolved, the ticket was closed, archived, and the team moved on. No one is going to sit and re-check a blog post to see if it is updated. That just doesn't happen anywhere, not just at GitHub. Likely a separate ticket was created though to see if they could identify the root cause (compromised accounts, spam attack, etc) and potentially address it. That analysis takes time though and AI doesn't provide much value in scaling here (yet).
> All they needed to do was allocate a few days for any regular employee so they could use Copilot to find all these repositories and block them.
That statement in particular shows a poor understanding of malicious repositories. Blocking on a string signature is almost always worthless as that is a brittle signature and very easy for an actor to identify (that the actor is being blocked by said string).
If you're going to block malicious uploads you need something much more concrete than a string. Having a bot search periodically via regular expressions to remove things on a brittle signature runs the risk of blocking a non-malicious repository. Even if the regular expression is perfect, the threat actor would realize relatively quickly what is going on.
If the threat actor is infecting repositories via compromising GitHub accounts, blocking the repository also does little to resolve the problem as the account remains compromised.
Preventing spam or malware problems at scale in any organization the size of GitHub to date not solved well. Social media platforms (using these as an example as they have comparable scale) still struggle with spam and they have more telemetry and experience dealing with such things.
This is like US healthcare. Almost nobody in US healthcare starts of evil. Almost none of the processes start with a clear evil intent. The problem is that the only real reward is money so the processes and people that generate money stay while the processes and people that merely made things better, but didn't increase revenue, slowly fade away.
Does GH get money for taking these actions? Do they have any monetary incentive other than 'their reputation', which clearly isn't changing usage, to improve? Maybe the best question here is why is it that after a lot of black eyes on data usage, reliability and monitoring aren't people switching. What keeps you using GH after stories like this?
I have a suggestion. PyPi and similar package managers should start publishing security warnings about the hosts of projects. That in turn can eventually lead to bans of packages from generally insecure places. Maybe if we start seeing 'WARNING: projects from github.com may contain malware!' after doing pip install XXX MS will start listening.
by jmward01
GitHub's response time on malicious repositories is often lackluster. However, from conversations with folks at GitHub, my suspicion is that this is because they're being starved for resources, not incompetency or maliciousness.
This (IMO) points to a perverse reality: things need to get worse before they can get better. In other words, Microsoft probably needs to feel more pain (in the form of negative revenue pressure) before they take their own platform responsibilities (vis a vis not distributing malware) seriously.
We say this play out recently with improvements to GitHub Actions security, I expect we'll see the same here.
(Edit: to be absolutely clear, I have first-hand experience that GitHub's security folks work extremely hard, and are often doing the kinds of invisible, thankless "deck-swabbing" work that nobody even thinks about. They're just under-resourced.)
by woodruffw
Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- You’ll often find that problems aren’t solved by the people who would find themselves out of work if they solved them.
- > Why did they stop and take no further action?
It costs them nothing to blame the user for clicking links that they don't trust, even though they're hosting the links...
by nubinetwork - "security" is a cost-center, it does not generate revenue. Incidents from lack of "security" need to have a greater impact on revenue before the typical corporate entity spends money.by nickphx
- GitHub today isn't GitHub from the early-mid 2010's. Today it is a Microsoft side-gig, something they bought simply because nobody wanted theirs. But, GitHub makes good money, and turns out they're a great source of training material.
They're just limping along. In reality, I think Microsoft wants to kill the GitHub brand, they're just doing it slowly, feeding poison.
by sscaryterry - I generally believe in hanlons razor (assume stupidity rather than maliciousness), it's likely that github saw the easiest possible solution rather than diving deeper into the cause of the problem to fix it permanently
- Interesting article but doesn't seem to understand how security teams work or even how to address a threat outside of an brittle easily bypassed response.
The initial report was investigated and resolved, with the removal of specific repositories. GitHub likely lacks ability to prevent the attacker from repeating the attack (this is potentially a hard technical problem to solve depending on the infrastructure, available telemetry and how the threat actor is performing their attack).
Once that initial report was resolved, the ticket was closed, archived, and the team moved on. No one is going to sit and re-check a blog post to see if it is updated. That just doesn't happen anywhere, not just at GitHub. Likely a separate ticket was created though to see if they could identify the root cause (compromised accounts, spam attack, etc) and potentially address it. That analysis takes time though and AI doesn't provide much value in scaling here (yet).
> All they needed to do was allocate a few days for any regular employee so they could use Copilot to find all these repositories and block them.
That statement in particular shows a poor understanding of malicious repositories. Blocking on a string signature is almost always worthless as that is a brittle signature and very easy for an actor to identify (that the actor is being blocked by said string).
If you're going to block malicious uploads you need something much more concrete than a string. Having a bot search periodically via regular expressions to remove things on a brittle signature runs the risk of blocking a non-malicious repository. Even if the regular expression is perfect, the threat actor would realize relatively quickly what is going on.
If the threat actor is infecting repositories via compromising GitHub accounts, blocking the repository also does little to resolve the problem as the account remains compromised.
Preventing spam or malware problems at scale in any organization the size of GitHub to date not solved well. Social media platforms (using these as an example as they have comparable scale) still struggle with spam and they have more telemetry and experience dealing with such things.
- This is like US healthcare. Almost nobody in US healthcare starts of evil. Almost none of the processes start with a clear evil intent. The problem is that the only real reward is money so the processes and people that generate money stay while the processes and people that merely made things better, but didn't increase revenue, slowly fade away.
Does GH get money for taking these actions? Do they have any monetary incentive other than 'their reputation', which clearly isn't changing usage, to improve? Maybe the best question here is why is it that after a lot of black eyes on data usage, reliability and monitoring aren't people switching. What keeps you using GH after stories like this?
I have a suggestion. PyPi and similar package managers should start publishing security warnings about the hosts of projects. That in turn can eventually lead to bans of packages from generally insecure places. Maybe if we start seeing 'WARNING: projects from github.com may contain malware!' after doing pip install XXX MS will start listening.
by jmward01 - GitHub's response time on malicious repositories is often lackluster. However, from conversations with folks at GitHub, my suspicion is that this is because they're being starved for resources, not incompetency or maliciousness.
This (IMO) points to a perverse reality: things need to get worse before they can get better. In other words, Microsoft probably needs to feel more pain (in the form of negative revenue pressure) before they take their own platform responsibilities (vis a vis not distributing malware) seriously.
We say this play out recently with improvements to GitHub Actions security, I expect we'll see the same here.
(Edit: to be absolutely clear, I have first-hand experience that GitHub's security folks work extremely hard, and are often doing the kinds of invisible, thankless "deck-swabbing" work that nobody even thinks about. They're just under-resourced.)
by woodruffw