Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • It seems VERY cooperative.

    If you can afford that, with all the coordination costs that it comes from it, good.

    An alternative is just to let the downstream service own the retry logic. Too many requests? Just error out as soon as possible.

    Each team manages its budget and each other team adapts.

    by siscia
  • Easy. Take a larger cut from the driver for each retry.
    by whatever1
  • It's not related to the article content, but I can't help but notice the "diversity" in authorship grades: a senior staff, a (regular) senior, and a principal. I suppose grades are a key mechanism by which corporates keep their hamster wheels running.

    I'm thankful that, at my $WORK, we're all software engineers by title, from the lowly interns to the CTO. Still, everybody knows who's who and whom to talk to, depending on the gravity of the issue.

    by penguin_booze
  • I'd be interested to hear other strategies in this space. I've done the naive thing of allowing retries everywhere, and gotten into retry storms. When I was next presented with the problem, I tried the other naive thing of only allowing retries from the very top level service, which led me to redoing absolutely tons of work for each failure. What's a nice middle path that doesn't add too much complexity?
    by aftbit
  • I'm suspicious of load shedding not mentioned in the article. Combine that with exp backoff in the caller and you got yourself a pretty robust starting point
    by maxchisto
  • Meanwhile Google keeps giving me “please wait, do not reload page” walls, so I ctrl-r as rapidly as possible. Or is that the human test and response?
    by Scoundreller
  • So, effectively if A → B → C → D and D is failing, C may retry D, but B and A are discouraged from retrying the whole chain.

    This is quite slever. I also really like the concept of an "Error Budget", inspired by SRE and SLO(s) no doubt :)

    by prologic
  • I think the example is still too generic.

    First thing is, if your service has 7 call layers, maybe it's just too deep.

    Second thing is, I found that retry strategy works the best if you define it based on what the nodes are actually doing (instead of treating them as generic nodes). For example, if node D is a database failing a transaction, you may just configure it to retry the transaction instead of doing an application-initialized request resubmit, because the database probably knows better about why the transaction has failed than the application connected to it.

    Third thing is, retry is worth it only when progress has been and/or can still be made. If the resources is no longer available forever, then there's no point of retrying.

    by nirui

Explore Birbla archives