How we found a bug in the hyper HTTP library (opens in new tab)
The Images binding’s migration to a local Unix-socket architecture exposed a rare race condition in Rust’s `hyper` HTTP library. Under slow-reader conditions, large image responses were truncated even though they returned `200 OK` and a full `Content-Length`, causing downstream processing or image decoding to fail. After six weeks of investigation, the issue was traced to premature socket shutdown and fixed with four lines of code. ## Images Bindings and the Request Path - Cloudflare’s Images service runs on Workers and uses `hyper` to manage HTTP connections. - The Images binding lets Workers send image data directly to the service, chain transformations, and receive the processed result as a stream. - The response path involved: - The Images service generating the complete encoded image. - `hyper` buffering the response. - Data moving through socket buffers managed by the kernel. - A client or intermediary reading the response. - If the reader was fast, `hyper` could flush the entire response and safely shut down the socket. - If the reader was slower, the socket’s outbound buffer filled, requiring `hyper` to pause and resume writing. ## Moving from FL to Local Unix Sockets - Initially, binding traffic passed through Cloudflare’s FL intermediary service. - In December 2025, the Images team replaced FL with an internal binding running on the same machine. - Unix sockets removed network and FL-processing overhead, including routing and DNS work. - The redesign improved performance and allowed the Images team to release binding changes independently. - The bug appeared within days of the rollout. ## Successful Responses with Truncated Bodies - The first report involved nested image-processing pipelines: - An inner Images binding composited large JPEG and PNG inputs from R2. - An outer URL-based pipeline resized, compressed, and transcoded the result. - The inner pipeline returned `200 OK` and a `Content-Length` for several megabytes, but delivered only a fraction of the body. - One response contained roughly 200 KB instead of the expected 3.3 MB. - The outer pipeline reported an end-of-file error because the body ended before the declared message length. - Depending on the image format, clients saw partially rendered images or completely broken images. ## Reproducing and Isolating the Race - Engineers recreated the nested setup, then removed layers until the failure occurred with the binding alone. - Batch testing produced failures reliably—for example, 19 of 25 requests in one run. - The amount of data received, approximately 200 KB, closely matched the production socket-buffer size. - This indicated that the failure was related to backpressure and socket-buffer exhaustion rather than the customer’s specific configuration. - Investigation eventually identified a race in `hyper` where the connection could be shut down before buffered response data had finished flushing. The incident demonstrates that HTTP success status codes do not guarantee complete response bodies when connection handling is incorrect. Systems streaming large payloads over sockets should test slow-reader and backpressure scenarios, and libraries should only close connections after all buffered data has been written.