Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • The footnote on exponential vs. log-normal service times is the part I'd push on.. in production I almost never see exponential, and heavy tails change the picture. Curious if you've looked at how robust this is under realistic distributions
    by fabijanbajo
  • It's not surprising if one has the mental model of the probability that the request gets enqueued. Then when you add variable time to process requests it becomes more clear why some requests can take unexpectedly long (there is a >0 probability that a request gets queued behind several of the slowest endpoints, for example). So even if 90% of the endpoints are fast and most of the requests aren't even queued, there will still be some that end up being quite slow.
    by resters
  • Why would anyone think that it would get linearly worse? What's the (wrong) assumption there?
    by nilsherzig
  • Of course, this assumes independent events. World Cup, super bowls, etc break these assumptions.

    Still, queuing theory is so cool.

    by crypttales
  • M/M/c is not the right model for a typical loadbalancer, since a loadbalancer typically does not manage a shared queue but simply passes the jobs to one of the servers. The models are:

    - M/M/1 (vertical scaling, one queue and one fast server): fastest response time

    - M/M/c (thread-pool, one shared queue, c slow servers): c-times slower for low loads, asymptotically similar to M/M/1 for high loads

    - c-times M/M/1 (loadbalancer, c slow servers each which its own queue): always c-times slower than M/M/1.

    Only the response times are different. Throughtput is the same in the ideal case.

    by juergn
  • What's conspicuously missing is the plot of performance when you do have a well tuned queue in front of the service. Yes, having a queue becomes less important the more backend servers you have, but here even with 10 servers the plot shows your latency remains >25% worse than it would be with a queue. Also missing is discussion of how the variance in processing times affects you when you rely on load balancing alone.
    by megamalloc
  • A dead comment says:

    > Of course, this assumes independent events. World Cup, super bowls, etc break these assumptions.

    Yes, this is very true. The model here works for Poisson arrivals and exponential service time (the M/M), which are poor approximations of real-world traffic patterns (which tend to be non-stationary and non-ergodic, and include substantial seasonality). However, the frequency of that seasonality is typically rather low (e.g. daily cycles), and so these stronger assumptions are quite defensible for short time periods.

    A better approach is to do simulation with real traffic patterns, or even with more sophisticated parametric models, and get better answers (e.g. https://stability-sim.systems/). The good news is that kind of simulation is cheaper to do than ever before.

    by mjb
  • the article offers a simplified world model: Poisson arrivals and infinite queue, which is fine as a math model.

    In the real world however, the bursts can be correlated, due to factors like timeouts/retries, thundering herd, correlated bursts.

    so the real economics of load-balanced system is a simple reliability story: being able to reasonably serve the peak traffic, which leads to over-provisioning of those systems.

    using cloud allows some form of scale up/down of resources, but doesn't completely solve the problem. I think the migration away from synchronyous systems towards async systems and letting clients gradually absorb the delays is a better approach (rather than forcing infrastructure to be dynamically scaled up/down and be billed per request-second by your cloud provider)

    by bijowo1676

Explore Birbla archives