Join the discussion
Write your take first — we'll ask for email only when you're ready to publish.
- Hacker News
- >We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity. We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.
Crazy.
by hnburnsy - they're still migrating to Azure?by ramon156
- "... these incidents make clear that we must accelerate this work."
It feels like GitHub maybe needs to slow down? 'We must change things faster' is a wild way to start off an eight hour hard-down postmortem.
by StilesCrisis - I think he meant to say "prioritize" so that availability related work sees results sooner — "acceleration" — than it would without an increase in priority.by spyc
- Do you think the load is going away?
The current infrastructure cannot handle the new load requirements. Either the infrastructure must change, or they must start denying users the ability to use the infrastructure.
by nickelpro - Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic.
I feel like I'm mildly stupid in trying to out retries as heresy but I'm not sure.
by Quarrelsome - I was using claude tethered via my phone, and would lose signal every now and then as we went through a tunnel. I was glad for how resilient it was its its eventual retries.by madeofpalk
- > Are retries bad?
For sh*ty providers they are great. Best of all when backsourced to the user by "Try again later."
> As it obscures it when stuff has genuinely gone wrong
Works as designed - at every level.
by chrisjj - You're not wrong - https://devblogs.microsoft.com/oldnewthing/20051107-20/?p=33...by anonymars
- It seems like retries are sometimes best left to the human being in front of the screen. Works well enough.by dapperdrake
- I don't like blind retries. It's different if the server or LB knows it's overloaded and asks clients to retry in X seconds.by frollogaston
- I totally agree with you, I think retries are overused, with the exception of operations that are known to be unreliable and can't be improved.
In my experience, errors which go away within a few seconds are quite rare, and are mainly due to flaws which are usually caught in testing.
I think a very careful cost/risk/benefit analysis should be done when adding automatic retries to things. As well as potentially causing cascading failures, it is a degraded user experience when it doesn't succeed.
As a user I would rather see an error straight away than see many seconds of spinning while something silently retries, and THEN an error.
by GeorgeDewar - Retries are good, conditional on having a client-side circuit breaker that stops retries quickly when nothing is working. Otherwise, they are good in good times and bad in bad times.by taylor-s
- https://brooker.co.za/blog/2022/02/28/retries.html
Seems that retries are good when the error is rare, and bad when the error is common. Typically outages have you transitioning from "everything is fine" to "nothing works", so being able detect that transition early is helpful
by Olreich - Distributing across different services wouldn't be a bad idea....
I still can't help but feel a little grateful for what they do across the free side of things. I know it isn't altruism, and I know nobody needs to defend a billion dollar corporation but...
Name another service that does what they do for FREE (and no ads) at this scale. It isn't easy. Wikipedia has probably more usage, but is a simpler endeavor. (except the moderation part, that's just amazing) Open Street map? Smaller and simpler. Internet archive? Again, smaller and simpler. Linux distro mirrors? Again, smaller and simpler than whatever github is doing for free.
by altcognito - It's been bad because it's crowded out better things. When you put something shit on the market for free you destroy most of the will for customers to pay for a thing, even if it's good, because they can just use the shit free version. This can be an intentional business strategy.by inigyou
- “ Name another service that does what they do for FREE (and no ads) at this scale” and is reliable is the question
- The point is, GitHub is not free for me (we pay) yet we were also down?
IMO definiely time to segregate the free and the paid service. Otherwise those of us who need the service to stay up will have to look for alternatives.
The post mortem conclusion should've been to put usage limits on the free tier so paid users aren't killed by them.
by Maxion - My only push back would be on the FREE part. I coulda bought that 5 years ago. Now I look at GitHub and go, "if the product is free, it's because Im the product" with all their co-pilot stuff.by tcmart14
- > Errors in those services triggered a client-side retry loop that increased traffic during recovery
Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours.
> Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.
The detailed root analysis tries to pass this off as a "bug". You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour is functioning exactly as designed. In this case aggressively to try and hide problems if token service responses become flakey.
by cube00 - So, in all of your software, you have introduced randomness in your retries so that the billions of your clients avoid retry synchronization dances?by _kidlike
- Backend API rate limiting has to surely kick in and force you to wait x amount of time before you try again… Discords bot API actually sends you how long before you retry.
- > You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour
Not to join the parade, but what would a unit test that confirms a cycling behavior across all the instances in-flight even look like? I mean, besides "Not a unit test".
by tclancy - > a wider trend to avoid showing the user any error at all costs
And in fact you can see the degradation of software over the previous decade-plus via Google Trends search for ‘something went wrong’ lol: https://trends.google.com/trends/explore?date=all&q=%22Somet...
by Lammy - They added 3 million CPUs. You reduce the complexity of their systems to a unit test…by tverbeure
- Maybe the retry logic was vibecoded instead of using an existing hardened library. After all, according to Twitter, nobody is looking at the code anymore.by sqquima
- > You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour
That wouldn't be a unit test - that's more like an end-to-end or integration test.
Have you ever worked anywhere that had perfect test coverage? It just doesn't happen, nor is it possible unless you're building a calculator app or todo list.
by bluerooibos - A common pattern in highly available services is that sometimes you should retry immediately (because the node you hit is rolling/broken/overloaded, but the others aren't) and other times you should back off aggressively (because the service is degraded).
If your server indicates with 100% accuracy when to retry immediately vs backoff, AND if all your clients consume that information with 100% accuracy, things go great. But there are lots of situations where one or both of those breaks down.
by jdm2212 - > Errors in those services triggered a client-side retry loop that increased traffic during recovery.
The worst outages I've been part of always have some version of this :(
by jdm2212 - the 'ol thundering herd problem...by k33P1Tr3aL