Client-side load balancing at a million requests per second

engineering.zalando.com55 pointsby cjbooms7 comments

Comments

Hacker News

> The approval gates came out, replaced by a sequenced market-group rollout (test -> eu-0 -> eu-1 -> eu-2) that uses the smaller regions as an alarm buffer before the critical eu-2 region.

this is interesting. these names dont look like aws regions. are they geographic divisions or groups randomly assigned to users? or based on something like total spend over last N months to make sure high value customers dont quit?

if users are assigned to a stable group that would mean some of them experience way more issues than others. i would do it with a random subset of AZs or individual accounts thats different every time. not sure which one is the case here.

by tancop

What we call "Market Groups" are an artifact of how we deploy our Product Read API. Any deployment can serve Product-Offer data (product with price and stock information from a merchant in a country) for all of our 29 European markets. But rather than have a deployment per country, or one deployment for all countries, we choose to group countries into 3 "Market Groups' named: eu-N. Our edge load-balancer interrogates HTTP headers to determine which Market Group deployment to send the request to, and we can dynamically adjust this edge routing as needed. The idea was to have some resiliency and be able to isolate say a massive sales campaign or bot attack in one country from impacting other countries. But it has been mostly useful for this staged deployment where we can deploy and test changes first in a group of our lower monetary value countries, such as Ireland, and then move on to higher value country groups, such as Germany/Switzerland. So not customers, traffic from countries.

by cjbooms

The approaches described in the article are clever and the results are impressive, so kudos to the team!

However, while reading, it felt weird that they treated Skipper as a third-party dependency, even though it is developed and operated by their colleagues. I mean, they could've tried contributing some updates and improvements to Skipper, maybe reusing Skipper's discovery mechanism, instead of rolling out their own, completely independent LB module.

And yes, it's true that in big companies it can be hard to organise cross-team collaboration, but IMO it's still worth a try.

by kirmerzlikin

I had a similar thought - it seems like most of the concerns could be addressed by adding better request tracing and improving the resiliency of the existing system. I wonder if this is just the result of an organizational tension where the load balancing team isn’t motivated to address this particular use case (which seems likely if the organization is large enough).

by 43920

Hi all. Blog author here, happy to take any questions you might have. It's a bit of a long one, but I put a lot of effort into the story arc and readability so it's (hopefully) an easy top to bottom read, that takes you on our journey. It covers some advanced distributed computing topics, such as replicating the exact same hash-ring as our existing load-balancer in the client application (JVM), what we implemented to fade-in new pods slowly so their caches get a chance to warm, and how we attempted to migrate to Availability Zone (AZ) Aware routing to save on AWS' inter-az transfer fees. Hope you enjoy it!

by cjbooms

Thanks for the article.

I have two questions/comments:

1. The N-ring fade-in is quite neat. I guess without the constraint of hash parity rendezvous hashing [0] could have been an elegant approach since it has support for weights (and generally better statistical properties than consistent hashing based on rings).

2. You mention still having the fallback of your existing load balancer. Is this a temporary thing during rollout or do you intend to keep it long-term. Asking because I generally tend to stay clear of fallbacks in distributed systems, as they introduce bimodality and metastable failures [1] [2].

[0] https://en.wikipedia.org/wiki/Rendezvous_hashing

[1] https://builder.aws.com/content/3EuS9Sakq7L3VLQIF3qzfMfke1Y/...

[2] https://brooker.co.za/blog/2021/05/24/metastable.html

by charleshn

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Hacker News
> The approval gates came out, replaced by a sequenced market-group rollout (test -> eu-0 -> eu-1 -> eu-2) that uses the smaller regions as an alarm buffer before the critical eu-2 region.
this is interesting. these names dont look like aws regions. are they geographic divisions or groups randomly assigned to users? or based on something like total spend over last N months to make sure high value customers dont quit?
if users are assigned to a stable group that would mean some of them experience way more issues than others. i would do it with a random subset of AZs or individual accounts thats different every time. not sure which one is the case here.
by tancop
What we call "Market Groups" are an artifact of how we deploy our Product Read API. Any deployment can serve Product-Offer data (product with price and stock information from a merchant in a country) for all of our 29 European markets. But rather than have a deployment per country, or one deployment for all countries, we choose to group countries into 3 "Market Groups' named: eu-N. Our edge load-balancer interrogates HTTP headers to determine which Market Group deployment to send the request to, and we can dynamically adjust this edge routing as needed. The idea was to have some resiliency and be able to isolate say a massive sales campaign or bot attack in one country from impacting other countries. But it has been mostly useful for this staged deployment where we can deploy and test changes first in a group of our lower monetary value countries, such as Ireland, and then move on to higher value country groups, such as Germany/Switzerland. So not customers, traffic from countries.
by cjbooms
The approaches described in the article are clever and the results are impressive, so kudos to the team!
However, while reading, it felt weird that they treated Skipper as a third-party dependency, even though it is developed and operated by their colleagues. I mean, they could've tried contributing some updates and improvements to Skipper, maybe reusing Skipper's discovery mechanism, instead of rolling out their own, completely independent LB module.
And yes, it's true that in big companies it can be hard to organise cross-team collaboration, but IMO it's still worth a try.
by kirmerzlikin
I had a similar thought - it seems like most of the concerns could be addressed by adding better request tracing and improving the resiliency of the existing system. I wonder if this is just the result of an organizational tension where the load balancing team isn’t motivated to address this particular use case (which seems likely if the organization is large enough).
by 43920
Hi all. Blog author here, happy to take any questions you might have. It's a bit of a long one, but I put a lot of effort into the story arc and readability so it's (hopefully) an easy top to bottom read, that takes you on our journey. It covers some advanced distributed computing topics, such as replicating the exact same hash-ring as our existing load-balancer in the client application (JVM), what we implemented to fade-in new pods slowly so their caches get a chance to warm, and how we attempted to migrate to Availability Zone (AZ) Aware routing to save on AWS' inter-az transfer fees. Hope you enjoy it!
by cjbooms
Thanks for the article.
I have two questions/comments:
1. The N-ring fade-in is quite neat. I guess without the constraint of hash parity rendezvous hashing [0] could have been an elegant approach since it has support for weights (and generally better statistical properties than consistent hashing based on rings).
2. You mention still having the fallback of your existing load balancer. Is this a temporary thing during rollout or do you intend to keep it long-term. Asking because I generally tend to stay clear of fallbacks in distributed systems, as they introduce bimodality and metastable failures [1] [2].
[0] https://en.wikipedia.org/wiki/Rendezvous_hashing
[1] https://builder.aws.com/content/3EuS9Sakq7L3VLQIF3qzfMfke1Y/...
[2] https://brooker.co.za/blog/2021/05/24/metastable.html
by charleshn