Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • > The failure was caused by a timing-dependent race condition in hyper’s HTTP/1 connection handling. When the reader was slower and the socket buffer filled, poll_flush returned Poll::Pending, but the dispatch loop discarded that result. Hyper then treated the response as complete and shut down the socket while data remained buffered internally, causing the client to receive an EOF before the full body arrived.

    https://github.com/hyperium/hyper/issues/4022

    Saved you 3000 words

    by 100ms
  • Hey, you have to justify three engineers full time's worth of salary.
    by moralestapia
  • Reminds me of another “slow client”-related bug in gunicorn: https://github.com/benoitc/gunicorn/issues/3334
    by michalc
  • Nice writeup, but I don't understand how `curl` didn't trigger bug for them (or any other hyper HTTP server out there), given the explanation in the article.

    `curl --http1.1` sends `Connection: Close` so sender (hyper) must attempt to shutdown connection after sending whole body. Surely any network is slower than memory copy into socket kernel buffers, so it must reliably trigger condition "buffer flush can't be done in one go" and thus trigger early TCP shutdown.

    by nopurpose
  • So “fearless concurrency” still only happens when one just decides to not be afraid… :)
    by pseudony
  • This does not appear to be a concurrency bug though?
    by c0balt
  • Would using Rust have prevented this?
    by microgpt
  • Agree. This is warning to people who thought Rust is optional at cloud scale.
    by geodel
  • The Hyper library in question is a Rust library.

    Did you read the article, or are you a "use rust" parrot / bot based on titles?

    by Cthulhu_
  • No. Anyone expecting that hasn't read No Silver Bullet essay.
    by Ygg2
  • Isn't this already Rust?
    by re-thc
  • I get that it’s fun to dunk on Rust when a Rust bug surfaces. But is it a bit petty to bring this out when there’s any type of bug of any severity in any Rust software?

    In this case a small minority of requests were getting truncated responses.

    No one said Rust software is bug free. If someone thinks that they’ve been seriously misled.

    by testdelacc1
  • > We spent six weeks chasing a nearly invisible bug — a race condition that occurred only under specific conditions — in the hyper library that impacted how the Images binding returned processed image data back to the client. In the end, it took four lines of code to fix it.

    That's a long time, must be frustrating.

    by worldsavior
  • It is a long time and it gets frustrating when there is significant time where there is flailing with no visible progress.

    I have had long bug hunts (~a month each) and witnessed ones that took much, much longer. But the longest one I witnessed was drawn out because reproduction was initially unreliable and could take weeks to months. Thankfully, reproduction was by letting a box sit in a corner while tje people involved moved on to other tasks. This kept everybody sane.

    by gmueckl
  • This would have been flagged by Clippy lints `let_underscore_untyped` or `let_underscore_must_use`, which sadly are not enabled by default.
    by Twey
  • Ehh, easy fix

        #[allow(clippy::let_underscore_untyped,clippy::let_underscore_must_use)]
    let _ = self.poll_flush(cx)?;
    by pwdisswordfishq
  • Or just by not writing let _ =
    by microgpt
  • Cloudflare does not notice (until a customer complains) that they are sending broken responses at scale? I would have thought they would notice this from sampling and linting a few replies.. just in case they did something like Cloudbleed again.
    by edelbitter
  • Can you get reasonable results without exposing sensitive info? I'm asking because I genuinely have no idea what it's like at their scale
    by ramon156