11-byte payload can exhaust OpenSSL server memory

An 11-byte payload is enough to exhaust memory on vulnerable OpenSSL servers, according to Okta’s Red Team. The denial-of-service flaw, dubbed HollowByte, requires no authentication and has no assigned vulnerability identifier.

The OpenSSL team has quietly fixed the issue and backported the change to older releases. Because OpenSSL underpins TLS across much of the internet, organizations should move to patched packages promptly.

How the HollowByte attack works

During a TLS handshake, every message begins with a 4-byte header containing a three-byte field that declares the size of the following data. Vulnerable OpenSSL versions allocate memory for that declared size before receiving and validating the payload.

An attacker can open a TLS connection and send an 11-byte message claiming that a far larger body is coming. The server allocates the requested memory, then waits indefinitely for data that never arrives. Repeating the process over many connections can consume substantial memory with relatively little network traffic.

Recommended reading

Florida man charged over Steam game crypto malware

“The worker thread then blocks, waiting indefinitely for data that will never arrive.”

OpenSSL eventually frees the buffers when connections close, but glibc typically retains small-to-medium allocations for reuse instead of returning them immediately to the operating system. By varying the claimed sizes, attackers can prevent reuse, heavily fragment the heap, and drive the server’s Resident Set Size upward. Restarting the process is the only way to fully reclaim the space.

OpenSSL versions containing the fix

OpenSSL is embedded in NGINX, Apache, Node.js, Python, Ruby, PHP, MySQL, PostgreSQL, and most Linux distributions. Okta found that low-capacity NGINX environments could be quickly depleted, while larger systems lost as much as 25% of their memory while attack bandwidth stayed below security-alert thresholds.

The fix is included in OpenSSL 4.0.1 and backported to 3.6.3, 3.5.7, 3.4.6, and 3.0.21. These releases grow buffers only as data arrives, ignoring the size claimed in the header.

OpenSSL classifies the change as a hardening fix, not a security vulnerability. Okta nevertheless recommends upgrading distribution-provided OpenSSL packages immediately.

×