Steady reinvention: A quick historical past of block storage at AWS

We didn’t have to worry much about the network getting in the way since end-to-end EBS latency was dominated by HDDs and measured in the 10s of milliseconds. Even our early data center networks were beefy enough to handle our user’s latency and throughput expectations. The addition of 10s of microseconds on the network was a small fraction of overall latency.

Compounding this latency, hard drive performance is also variable depending on the other transactions in the queue. Smaller requests that are scattered randomly on the media take longer to find and access than several large requests that are all next to each other. This random performance led to wildly inconsistent behavior. Early on, we knew that we needed to spread customers across many disks to achieve reasonable performance. This had a benefit, it dropped the peak outlier latency for the hottest workloads, but unfortunately it spread the inconsistent behavior out so that it impacted many customers.

When one workload impacts another, we call this a “noisy neighbor.” Noisy neighbors turned out to be a critical problem for the business. As AWS evolved, we learned that we had to focus ruthlessly on a high-quality customer experience, and that inevitably meant that we needed to achieve strong performance isolation to avoid noisy neighbors causing interference with other customer workloads.

At the scale of AWS, we often run into challenges that are hard and complex due to the scale and breadth of our systems, and our focus on maintaining the customer experience. Surprisingly, the fixes are often quite simple once you deeply understand the system, and have enormous impact due to the scaling factors at play. We were able to make some improvements by changing scheduling algorithms to the drives and balancing customer workloads across even more spindles. But all of this only resulted in small incremental gains. We weren’t really hitting the breakthrough that truly eliminated noisy neighbors. Customer workloads were too unpredictable to achieve the consistency we knew they needed. We needed to explore something completely different.

Around the time I started at AWS in 2011, solid state disks (SSDs) became more mainstream, and were available in sizes that started to make them attractive to us. In an SSD, there is no physical arm to move to retrieve data—random requests are nearly as fast as sequential requests—and there are multiple channels between the controller and NAND chips to get to the data. If we revisit the bank example from earlier, replacing an HDD with an SSD is like building a bank the size of a football stadium and staffing it with superhumans that can complete transactions orders of magnitude faster. A year later we started using SSDs, and haven’t looked back.

We started with a small, but meaningful milestone: we built a new storage server type built on SSDs, and a new EBS volume type called Provisioned IOPS. Launching a new volume type is no small task, and it also limits the workloads that can take advantage of it. For EBS, there was an immediate improvement, but it wasn’t everything we expected.

We thought that just dropping SSDs in to replace HDDs would solve almost all of our problems, and it certainly did address the problems that came from the mechanics of hard drives. But what surprised us was that the system didn’t improve nearly as much as we had hoped and noisy neighbors weren’t automatically fixed. We had to turn our attention to the rest of our stack—the network and our software—that the improved storage media suddenly put a spotlight on.

Even though we needed to make these changes, we went ahead and launched in August 2012 with a maximum of 1,000 IOPS, 10x better than existing EBS standard volumes, and ~2-3 ms average latency, a 5-10x improvement with significantly improved outlier control. Our customers were excited for an EBS volume that they could begin to build their mission critical applications on, but we still weren’t satisfied and we realized that the performance engineering work in our system was really just beginning. But to do that, we had to measure our system.

At this point in EBS’s history (2012), we only had rudimentary telemetry. To know what to fix, we had to know what was broken, and then prioritize those fixes based on effort and rewards. Our first step was to build a method to instrument every IO at multiple points in every subsystem—in our client initiator, network stack, storage durability engine, and in our operating system. In addition to monitoring customer workloads, we also built a set of canary tests that run continuously and allowed us to monitor impact of changes—both positive and negative—under well-known workloads.

With our new telemetry we identified a few major areas for initial investment. We knew we needed to reduce the number of queues in the entire system. Additionally, the Xen hypervisor had served us well in EC2, but as a general-purpose hypervisor, it had different design goals and many more features than we needed for EC2. We suspected that with some investment we could reduce complexity of the IO path in the hypervisor, leading to improved performance. Moreover, we needed to optimize the network software, and in our core durability engine we needed to do a lot of work organizationally and in code, including on-disk data layout, cache line optimization, and fully embracing an asynchronous programming model.

A really consistent lesson at AWS is that system performance issues almost universally span a lot of layers in our hardware and software stack, but even great engineers tend to have jobs that focus their attention on specific narrower areas. While the much celebrated ideal of a “full stack engineer” is valuable, in deep and complex systems it’s often even more valuable to create cohorts of experts who can collaborate and get really creative across the entire stack and all their individual areas of depth.

By this point, we already had separate teams for the storage server and for the client, so we were able to focus on these two areas in parallel. We also enlisted the help of the EC2 hypervisor engineers and formed a cross-AWS network performance cohort. We started to build a blueprint of both short-term, tactical fixes and longer-term architectural changes.

This was a fantastic win for latency and efficiency, so we decided to do the same thing for EBS storage. By moving more processing to hardware, we removed several operating system queues in the hypervisor, even if we weren’t ready to pass the device directly to the instance just yet. Even without passthrough, by offloading more of the interrupt driven work, the hypervisor spent less time servicing the requests—the hardware itself had dedicated interrupt processing functions. This second Nitro card also had hardware capability to handle EBS encrypted volumes with no impact to EBS volume performance. Leveraging our hardware for encryption also meant that the encryption key material is kept separate from the hypervisor, which further protects customer data.

Moving EBS to Nitro was a huge win, but it almost immediately shifted the overhead to the network itself. Here the problem seemed simple on the surface. We just needed to tune our wire protocol with the latest and greatest data center TCP tuning parameters, while choosing the best congestion control algorithm. There were a few shifts that were working against us: AWS was experimenting with different data center cabling topology, and our AZs, once a single data center, were growing beyond those boundaries. Our tuning would be beneficial, as in the example above, where adding a small amount of random latency to requests to storage servers counter-intuitively reduced the average latency and the outliers due to the smoothing effect it has on the network. These changes were ultimately short lived as we continuously increased the performance and scale of our system, and we had to continually measure and monitor to make sure we didn’t regress.

We weren’t happy that solely a comparatively small variety of volumes and prospects had higher efficiency. We needed to deliver the advantages of SSDs to everybody. That is an space the place scale makes issues tough. We had a big fleet of hundreds of storage servers operating thousands and thousands of non-provisioned IOPS buyer volumes. A few of those self same volumes nonetheless exist at the moment. It might be an costly proposition to throw away all of that {hardware} and change it.

There was empty house within the chassis, however the one location that didn’t trigger disruption within the cooling airflow was between the motherboard and the followers. The great factor about SSDs is that they’re usually small and light-weight, however we couldn’t have them flopping round unfastened within the chassis. After some trial and error—and assist from our materials scientists—we discovered warmth resistant, industrial power hook and loop fastening tape, which additionally allow us to service these SSDs for the remaining lifetime of the servers.

Armed with this data, and lots of human effort, over the course of some months in 2013, EBS was in a position to put a single SSD into each a kind of hundreds of servers. We made a small change to our software program that staged new writes onto that SSD, permitting us to return completion again to your utility, after which flushed the writes to the slower exhausting disk asynchronously. And we did this with no disruption to prospects—we have been changing a propeller plane to a jet whereas it was in flight. The factor that made this attainable is that we designed our system from the beginning with non-disruptive upkeep occasions in thoughts. We might retarget EBS volumes to new storage servers, and replace software program or rebuild the empty servers as wanted.

This capability emigrate buyer volumes to new storage servers has come in useful a number of occasions all through EBS’s historical past as we’ve recognized new, extra environment friendly knowledge buildings for our on-disk format, or introduced in new {hardware} to interchange the previous {hardware}. There are volumes nonetheless energetic from the primary few months of EBS’s launch in 2008. These volumes have seemingly been on a whole lot of various servers and a number of generations of {hardware} as we’ve up to date and rebuilt our fleet, all with out impacting the workloads on these volumes.

There’s another journey over this time that I’d prefer to share, and that’s a private one. Most of my profession previous to Amazon had been in both early startup or equally small firm cultures. I had constructed managed providers, and even distributed techniques out of necessity, however I had by no means labored on something near the dimensions of EBS, even the EBS of 2011, each in know-how and group dimension. I used to be used to fixing issues on my own, or perhaps with one or two different equally motivated engineers.

I actually get pleasure from going tremendous deep into issues and attacking them till they’re full, however there was a pivotal second when a colleague that I trusted identified that I used to be turning into a efficiency bottleneck for our group. As an engineer who had grown to be an skilled within the system, but additionally who cared actually, actually deeply about all elements of EBS, I discovered myself on each escalation and likewise desirous to assessment each commit and each proposed design change. If we have been going to achieve success, then I needed to discover ways to scale myself–I wasn’t going to resolve this with simply possession and bias for motion.

This led to much more experimentation, however not within the code. I knew I used to be working with different sensible of us, however I additionally wanted to take a step again and take into consideration tips on how to make them efficient. One among my favourite instruments to come back out of this was peer debugging. I bear in mind a session with a handful of engineers in one in every of our lounge rooms, with code and some terminals projected on a wall. One of many engineers exclaimed, “Uhhhh, there’s no method that’s proper!” and we had discovered one thing that had been nagging us for some time. We had ignored the place and the way we have been locking updates to important knowledge buildings. Our design didn’t often trigger points, however often we’d see gradual responses to requests, and fixing this eliminated one supply of jitter. We don’t at all times use this method, however the neat factor is that we’re in a position to mix our shared techniques information when issues get actually tough.

By all of this, I noticed that empowering individuals, giving them the flexibility to securely experiment, can usually result in outcomes which can be even higher than what was anticipated. I’ve spent a big portion of my profession since then specializing in methods to take away roadblocks, however depart the guardrails in place, pushing engineers out of their consolation zone. There’s a little bit of psychology to engineering management that I hadn’t appreciated. I by no means anticipated that one of the crucial rewarding elements of my profession can be encouraging and nurturing others, watching them personal and resolve issues, and most significantly celebrating the wins with them!

Reflecting again on the place we began, we knew we might do higher, however we weren’t certain how a lot better. We selected to strategy the issue, not as a giant monolithic change, however as a sequence of incremental enhancements over time. This allowed us to ship buyer worth sooner, and course appropriate as we discovered extra about altering buyer workloads. We’ve improved the form of the EBS latency expertise from one averaging greater than 10 ms per IO operation to constant sub-millisecond IO operations with our highest performing io2 Block Categorical volumes. We completed all this with out taking the service offline to ship a brand new structure.

We all know we’re not executed. Our prospects will at all times need extra, and that problem is what retains us motivated to innovate and iterate.