Meta has provided a rare glimpse into the company's storage infrastructure, claiming the system underpinning its AI operations helped slash the time it takes researchers to move AI training data across regions by up to 97 percent.
The tech giant – which looks set to join the hyperscale ranks amid reports it’s launching a cloud computing business unit – provided a detailed public breakdown in which it revealed that while graphics processing unit (GPU) compute has roughly tripled every two years, storage speeds haven't kept up.
In a bid to close the gap and serve storage clusters spanning all of Meta’s products, including Meta AI, Reality Labs, its social media platforms, and its prospective cloud customers needs, the tech giant’s engineers rebuilt its BLOB (Binary Large Object) storage layer that sits atop its existing multi-tenant storage fabric, dubbed Tectonic, which spans exabytes of data across its global data centers.
While suitable for web-app workloads like Facebook and Instagram, Meta’s legacy BLOB architecture was not resource-efficient when it came to supporting AI training. According to software engineers Sidharth Bajaj and Venkatraghavan Srinivasan, the old system had too many stacked service layers and had to execute sequential, stateful metadata lookups across multiple layers before it could resolve a file path to its actual storage location – adding latency and slowing performance.
Meta’s rebuilt storage foundation then revolved around three core changes, the first of which was collapsing the fragmented metadata system into a single unified schema backed by a database called ZippyDB, enabling near-instant path lookups. The second saw the elimination of a “dataplane proxy,” which was replaced by a “fat client” architecture capable of streaming bytes directly from storage servers to clients, boosting power-efficiency goals while resulting in lower latency. The third and final step saw its engineers shift from a global deployment model to a regional one, colocating storage with the GPUs that actually need it.
“With these changes, we have rebuilt the foundations and met the goal of adding zero overhead on top of Tectonic,” a Meta blog post reads. “By eliminating the data proxy, we also stay within budget for the power footprint.”
Taming spikes and hot spots
In addition to generally improving its overall storage stack to make it suitable for AI workloads, Meta engineers leveraged spare GPU memory as a distributed data cache for frequently and concurrently accessed data.
The team reused learnings from Owl, Meta’s system for distributing large data objects, which combines a decentralized peer-to-peer data plane with a centralized control plane. Originally built for speedily distributing app updates or viral content to huge numbers of servers, the engineers applied existing peer-sharing logic from Owl, wiring it into its BLOB-storage client software distribution kit (SDK) so that all data access goes through this cache and reduces the frequency with which GPUs need to ask storage for specific data.
Alongside the Owl integration, Meta introduced a separate “read-plan” metadata cache, which returns the storage address for frequently requested files in one to two milliseconds.
The tech giant said the combination helps it absorb demand spikes, such as when GPUs simultaneously restart and request the same ‘hot’ model weights, while also improving latency as data is served straight from memory rather than disk.
The final revision of Meta’s BLOB storage stack saw it solely focus on fixing bottlenecks, like egress spikes, which ended up causing congestion and timeouts and ultimately stalling GPUs. A dynamic concurrency control software solution was implemented, capable of tuning parallelism based on application-level congestion signals. Instead of applications blindly hammering the storage servers, Meta’s system automatically dials back how many requests an application can send at once when traffic is high.
According to Bajaj and Srinivasan, Meta’s revised BLOB-storage stack can now support AI workloads without causing GPU stalls, saving the tech giant money while providing performance boons.
Beyond fixing GPU stalls, though, Meta's engineers also had to solve actually getting data to its researchers in the first place. With GPUs increasingly scattered across regions, staff were routinely forced to wait hours for datasets to be copied and ingested into the specific part of the world where their training job would run.
To fix this, engineers again went to work, building a tiered caching system loosely modeled on how a computer's CPU pulls data from disk into progressively faster layers of memory.
Memory and flash on the GPU host act as the fastest tier, while regional flash-based BLOB-storage fabrics act as the next layer down. Desired data trickles upward via a “prefetch” mechanism pulling data a researcher is about to use into faster storage ahead of time, with an HDD-backed global BLOB-storage fabric acting as the “ultimate source of truth.”
Its new data-loading paradigm is applied to production workloads today and has slashed ingestion times across all of Meta’s workloads. Average ingestion times dropped from 150 minutes to just 10, a 93 percent dip. At the max level, Meta saw ingestion times previously at 89 hours reduce to just 182 minutes, a staggering 97 percent decrease.
“Modern AI workloads are data hungry, and storage plays an important role in both the computational cost and speed of innovation. Storage bottlenecks directly impact GPU utilization and computational cost, and in a world with geo-distributed GPUs, time spent on cross-region data ingestion directly impacts the speed of iteration in research,” the pair wrote.
“The BLOB-storage architecture at Meta was built to serve Meta’s family of apps, and we needed a step-function improvement in performance to serve AI workloads… By rebuilding the metadata subsystem and by adopting a tiered caching architecture with prefetching/on-demand hydration, we are able to meet the needs of today’s workloads effectively.”