GitHub Outage Tracker: Is GitHub Cooked?

Join the discussion

• Hacker News
• I hear the comments that state we should have sympathy for github. For the individual ops/sre I can have that sympathy because it must a heck of a time internally to manage things. I can't however have sympathy for company itself. They have not set themselves or the community for success. I think it is the responsibility of the company to anticipate the issues or at least react in a way that would build trust. The fact is there is a systemic issue in the company that causes the repeated unreliableness and they have not rooted it out.
  by silver92bullet
• It's not making headlines, but Google and Apple seem to be getting hammered by AI as well. Getting app updates approved to release on prod on Play Store / App store used to take days, now it's taking weeks.
  by imnes
• Getting rid of Actions and Copilot and other secondary services almost halves Github's incident rate: https://i.imgur.com/XPcMIFr.png

  I'm a big fan of Github Actions and I think people are often a little too harsh on it, but it's clear that it's sad that it's come at such a high cost to the platform's stability

  by nightpool
• GitHub Actions is really, really badly designed. The security model is fundamentally broken, the YAML hell is as bad as any, the log streaming lags like hell, they charge self-hosted runners for using their coordination plane, jobs queuing is really slow, and their software for actually running jobs is cursed and designed in a way that is practically hostile to self-hosting.

  GitHub Actions is god-awful. Have you ever used any other CI tools?

  by isityettime
• Actions aren't secondary to most paid GH users; and if they are down it usually means no deployments and no tests, which can often halt work.
  by Normal_gaussian
• An important thing to consider is how much of their uptime without incidents is not the normal working hours. Their incident-free uptime on 9-5 EST, Mon-Fri, is probably like 60%.
  by kevmo
• what's normal working hours? very US centric comment IMO. Europe, India, China, Latam etc. don't fall into your 9-5 EST normal working hour bucket.
  by perfectstorm
• > Their incident-free uptime on 9-5 EST, Mon-Fri, is probably like 60%.

  And it may be even worse in EDT, which is currently in effect!

  by CoastalCoder
• As a daily GitHub user in Australia I still haven’t figured out why everyone’s complaining about uptime. :)
  by thombles
• It’s ironic that the SCM that 2026 software engineers landed on, is one with a complicated distributed usage model and is slow/down because of a centralised service
  by djieidj283
• Apparently it's not exactly git side of GitHub that's failing. It's all the bells and whistles bolted on later causing the trouble.
  by chanux
• Is GitHub Cooked? Decidedly yes.

  Migrating to something else has been raised as a concern during every engagement with clients and prospective clients in the past year.

  "Do you use GitHub?"

  "Yes though we'd like to move to something else, we just don't know what yet."

  As soon as the next big thing shows up, they're done. And they know it.

  by xyzzy_plugh
• I wonder if this is a Microsoft Excel situation where everyone

  1. Hates the status quo

  2. Only uses a handful of the 10,000 features the status quo has

  3. Uses a different handful than everyone else

  Because that model kept Excel as the center of buisnesses around the world for 30 years and counting

  by Bratmon
• If backend GitHub services are anything like GHES then I’m surprised it even managed to scale this much.
  by 404mm
• I think its nothing like the Enterprise version, which is why they hate having to maintain it, same with Atlassian and Azure/Entra AD. Once service to rule them all, and in the cloud bind them.

  For security reasons, I would love a low feature, carefully engineered, offline github-like system. Something tuned for agentic behaviours, with an understanding of hierarchical agent identities.

  by angry_octet
• I feel for the team having to deal with these problems at github today. Its also beyond me how those in charge for years wouldn't have seen this coming.

  Github was bought by Microsoft, whether they wanted to acknowledge that internally or not. Microsoft went deep on LLMs, and specifically on LLMs for coding use cases. They must have recognized that LLM generated code and PRs would effectively DDoS GitHub.

  I can only assume they simply didn't care, likely driven by greed.

  by _heimdall
• > "GitHub has had 1125 incidents since February 2016, implying a monthly incident rate of 24"

  1125 incidents / 126 months ≈ 8.9 incidents per month, not 24

  still terrible, but why such an obvious error in the first sentence...

  by JeremyHerrman
• Prob used Copilot to write the excel formula...
  by 4petesake
• > GitHub has had 1125 incidents since March 2016. Over the last 3 months, they've averaged 24 incidents per month (↓ 5% vs prev 3mo).

  Looks like they fixed it already

  by stevage
• Not sure whether it has been updated since your comment, but the sentence now reads:

      GitHub has had 1125 incidents since March 2016. Over the last 3 months, they've averaged 24 incidents per month
  

  edit: although they also have 1.2 days of downtime (in a day) for their 'worst days' of downtime table, which suggests some auto number crunching is not working as expected.
  by 6LLvveMx2koXfwn
• Ahhh, I think the author mixed up two values here. That value seems to actually be the average over the past 3 months.

      const incidents = e.detail.incidents;
      
      // ...snip...
      
      const now = new Date();
      const threeMonthsAgo = new Date(now);
      threeMonthsAgo.setMonth(threeMonthsAgo.getMonth() - 3);
      
      // ...snip...
      
      var currentFreq = recentIncidents.length / 3; // <- We out here, smoking these guns with our homeboy Claude
      
      // ...snip...
      
      var earliest = null;
      for (var j = 0; j < incidents.length; j++) {
        var dd = new Date(incidents[j].started_at); // <- eventually incidents[j].started_at is "2016-03-01T07:07:37.000Z"
        if (!earliest || dd < earliest) earliest = dd;
      }
      
      // ...snip...
      
      document.getElementById('n-since').textContent = earliest
        ? earliest.toLocaleDateString('en-US', { month: 'long', year: 'numeric' })
        : '?';
      document.getElementById('n-rate').textContent = Math.round(currentFreq * 10) / 10;
  
  

  #n-since is going to be either march or feburary. It'll change depending on your timezone because JS's Date object always shifts the date around to match the same instant but in the system's timezone.

  #n-rate has nothing to do with the #n-since month, it's just the last trailing 3 months. And even then, it's sort of underbaked? It's moving the date back by 3 calendar months not taking into account differing numbers of days, so it'll under-report short months.

  I wouldn't trust the stats here.

  Edit: whoops, author updated the template while I was writing this! It now says "Over the last 3 months", though that's still calendar months.

  by graypegg
• Given that the outages are caused by record traffic, so far they are only cooked in the Yogi Berra sense of “Nobody goes there anymore, it’s too crowded”
  by joshuahedlund
• Many people think we're on the path to having separate webs for AIs and humans. This may be the beginning of that.
  by Bratmon
• Random AI slop code pushers don't pay the bills. Traffic isn't revenue.

  I've started suggesting to my CTO that we move to something else because we do pay the Github bills and yet we are also affected by other people's activity. Avoiding that is not a hard engineering problem for an alternative.

  A world where a record amount of code is pushed to GH but a large amount of paying customers leave would be "cooked"

  by majormajor
• Gets me wondering… if I get used to the fact that some location is busy, I stop going there. Other people might do the same. Could this create a detectable wavy signal over time? People stop going because it’s busy. It stops being busy. People go because it’s not busy. Repeat. Is this a known phenomenon?
  by Waterluvian
• Near the end of the 8.5 years that I worked at GitHub as an enterprise support engineer, I asked in an all hands if a "GitHub Classic" product had been considered. Much like World of Warcraft Classic, I imagined it would be a rewrite focused on matching the simpler feature set of the past.

  I was basically given the same response that blizzard gave that question; "you think you want that but you don't".

  by Fuzzwah
• Fun idea, I mean, I use old.reddit.com, would probably use classic.github.com too...

  Imagine YouTube Classic...

  by kigiri
• I think we need to have a little more sympathy for GitHub. You could justify the jabs when we could all blame any outage on the migration to Azure, but then they shared numbers around the scale they're dealing with now that everyone is constantly building and pushing with AI.

  I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers. Yes, they need to get this figured out, but a little sympathy goes a long way. I personally wish them the best and hope their on-call people can go back to getting normal amounts of sleep soon.

  by kashnote
• > I think it's commendable that they're not limiting access to the site or (intentionally) throttling newcomers.

  I don't think this is commendable at all. I give GitHub a lot of money and I'm tired of it being wasted with downtime.

  by herpdyderp
• Our company pays for GitHub. We're paying for a broken product that stops our work. I don't have sympathy for the companies whom I pay for a product and give me broken shit in return. This is entirely preventable and their own fault.

  A restaurant makes pizzas. They suddenly get 100x more popular. They can't make 100x more pizzas. But they are still taking orders from 100x more people. Not only are they not getting enough pizzas delivered that they took orders for, but in their rush to make and deliver more pizzas, they set the kitchen on fire, which makes an even longer wait for pizzas.

  When the pizza you ordered doesn't get delivered, do you have sympathy for the restaurant? Or do you tell them to stop taking orders they can't fill and try not to set the kitchen on fire?

  Now consider the pizza restaurant has 21 billion dollars in cash, is taking your money, and not giving you pizza.

  by 0xbadcafebee