This white paper from Raritan answers some of the most challenging questions associated with energy management while providing detailed explanations of how to correct some of the most common false assumptions regarding power consumption.  It goes on to discuss methods for measuring the difference between IT equipment power load vs. total facility power load, and calculating Power Usage Effectiveness (PUE).  Lastly, it demonstrates what to do with the knowledge gleaned from these calculations, and how to make the best use of it while planning your data centers energy management strategy.

Learn the importance of accurately calculating your data center’s power needs.  Click here to download this white paper from Raritan on energy requirement calculations and the advantages that come with real-time power monitoring.

The moderating pace of data center energy use is “driven mainly by a lower server installed base than was earlier predicted rather than the efficiency improvements anticipated in the report to Congress,” writes Koomey, whose 2007 report to Congress and the Environmental Protection Agency has framed most subsequent discussions of data center energy usage.

Economic Slowdown and Virtualization Cited

Koomey says companies have been buying fewer servers than anticipated due to the economic slowdown and the benefits of virtualization, which allows users to make better use of their server capacity. The study downplays the potential impact of an industry-wide effort to develop metrics and share best practices on energy efficiency, but indicated that these efforts could begin to have a larger impact in the near future, as more computing workloads shift to cloud computing platforms hosted in cutting-edge facilities.

In compiling the data, Koomey used estimates of installed servers from the research firm IDC. He assumed a data center Power Usage Effectiveness (PUE) rating of between 1.83 and 1.92, based on estimates of the average PUE reported in recent samplings by the EPA Energy Star Program and the Uptime Institute. While some cloud computing data centers have PUEs in the 1.1 to 1.3 range, Koomey noted that less efficient in-house corporate data centers account for the largest number of servers. “That (PUE) assumption will need to be revisited as cloud computing becomes more widely used,” he wrote.

Energy Usage Slows During Building Boom

The New York Times has been investigating data center energy use since early 2010, with a particular focus on Google’s data centers. “Data centers’ unquenchable thirst for electricity has been slaked by the global recession and by a combination of new power-saving technologies,” writes The Times John Markoff. “The decline in use is surprising because data centers, buildings that house racks and racks of computers, have become so central to modern life,” “The slowdown in the rate of growth of electricity use is particularly significant because it comes in the midst of the biggest build-out of new data center capacity in the history of the industry.”

The five-year period reviewed in the  Koomey tracks a time of dynamic growth for the data center industry, in which Google, Yahoo, Microsoft, Facebook and Apple all built enormous cloud data centers, while service providers filled space at wholesale data centers built by Digital Realty Trust and DuPont Fabros Technology.

By their nature, mission critical facilities such as Internet data centers are prime candidates for power monitoring systems. By employing monitoring systems to analyze system-wide historical and real-time power data, facility managers can reduce the cost of electricity and improve its quality and reliability.

Where high accuracy metering, disturbance recording, transient detection or harmonic analysis is needed, there is no substitute for power monitors. So if you’re going to deploy power monitors in a data center, this paper offers several key guidelines to keep in mind.

“The rapid rates of growth in data center electricity use that prevailed from 2000 to 2005 slowed significantly from 2005 to 2010, yielding total electricity use by data centers in 2010 of about 1.3% of all electricity use for the world, and 2% of all electricity use for the US,” Koomey writes. The report, “Growth in Data Center Power Use 2005 to 2010,” was prepared for the New York Times, which summarizes the findings. The full report is available via Koomey’s web site.

The moderating pace of data center energy use is “driven mainly by a lower server installed base than was earlier predicted rather than the efficiency improvements anticipated in the report to Congress,” writes Koomey, whose 2007 report to Congress and the Environmental Protection Agency has framed most subsequent discussions of data center energy usage.

Economic Slowdown and Virtualization Cited

Koomey says companies have been buying fewer servers than anticipated due to the economic slowdown and the benefits of virtualization, which allows users to make better use of their server capacity. The study downplays the potential impact of an industry-wide effort to develop metrics and share best practices on energy efficiency, but indicated that these efforts could begin to have a larger impact in the near future, as more computing workloads shift to cloud computing platforms hosted in cutting-edge facilities.

In compiling the data, Koomey used estimates of installed servers from the research firm IDC. He assumed a data center Power Usage Effectiveness (PUE) rating of between 1.83 and 1.92, based on estimates of the average PUE reported in recent samplings by the EPA Energy Star Program and the Uptime Institute. While some cloud computing data centers have PUEs in the 1.1 to 1.3 range, Koomey noted that less efficient in-house corporate data centers account for the largest number of servers. “That (PUE) assumption will need to be revisited as cloud computing becomes more widely used,” he wrote.

Energy Usage Slows During Building Boom

The New York Times has been investigating data center energy use since early 2010, with a particular focus on Google’s data centers. “Data centers’ unquenchable thirst for electricity has been slaked by the global recession and by a combination of new power-saving technologies,” writes The Times John Markoff. “The decline in use is surprising because data centers, buildings that house racks and racks of computers, have become so central to modern life,” “The slowdown in the rate of growth of electricity use is particularly significant because it comes in the midst of the biggest build-out of new data center capacity in the history of the industry.”

The five-year period reviewed in the  Koomey tracks a time of dynamic growth for the data center industry, in which Google, Yahoo, Microsoft, Facebook and Apple all built enormous cloud data centers, while service providers filled space at wholesale data centers built by Digital Realty Trust and DuPont Fabros Technology.



On May 17th the Data Center Efficiency Task Force issued a 14 page document entitled “Recommendations for Measuring and Reporting Version 2 – Measuring PUE for Data Centers”  – also known as PUE 2.

While PUE was originally created by The Green Grid in 2008, it was adopted in early 2010 by the Data Center Efficiency Task Force, a group consisting of 7×24 Exchange, ASHRAE, Silicon Valley Leadership Group, the U.S. Department of Energy’s Save Energy Now Program, the U.S. Environmental Protection Agency’s ENERGY STAR Program, United States Green Building Council, Uptime Institute and The Green Grid.

We’ll hear thoughts from members of the task force shortly. But first, let’s take a look at what’s new.

The updated PUE version is meant to clarify and reiterate the updated measurement requirements of the PUE metric and presumably prevent “PUE envy.” It reflects and reiterates the “Harmonizing Global Metrics for Data Center Energy Efficiency” definition of PUE which was released by the task force in February of this year.

Energy Rather Than Power

While it is still called PUE, it now specifies energy usage (expressed in KWH) not power (KW) as the primary basis for the metric. This still seems to be one of the sources of confusion to those using and quoting PUE numbers for their data centers.

There are four PUE measurement categories (0-3), of which three upper categories (1-3) specify annualized energy consumption as the basis for the calculations. Even PUE Category 0, which still allows power demand expressed in KW, now requires that the calculation be based on the highest power used by the facility, not just the lowest power measurement taken on a cold night when the chillers are off.

“PUE Category 0: This is a demand based calculation representing the peak load during a 12-month measurement period.”

This should help level the playing field of some seemingly exaggerated PUE claims.

Guidance on Measurement

The PUE 2 document is an enhancement and continues to follow the “3 Guiding Principles” and the “Harmonizing Global Metrics” document issued in February by The Green Grid and the Task Force and also adopted by some international organizations.
Moreover, PUE 2 includes more details on how to properly measure and calculate PUE for data centers located in a mixed use building, in particular building-supplied chilled water or condenser water.

One of the significant changes in PUE 2 which may be easy to overlook, is the following caveat regarding the requirement to reference the PUE category when proclaiming a data center’s efficiency.

“When publishing PUE, the category must clearly be indicated using a subscript e.g. PUE0, PUE1, PUE2, PUE3. A PUE reported without the subscript is not considered to be in compliance with these recommendations.”

It would seem that very few (if any) of those who have previously made public proclamations of their PUE results have indicated which PUE category methodology and protocols they used to derive their claimed results. Going forward they will hopefully include the category in future PUE announcements. It should be interesting to see how many organizations will adhere to this requirement in any new crop of PUE announcements.

When viewed from the rack, row, room, or building level (or even across a network of data centers at the enterprise level), ARC provides a simple way to answer the question.

Typically, most data centers don’t calculate ARC explicitly. Instead, operators set a simple alarm threshold on the Actual Load of each device. For example, if the power load reaches 50% on a device (or more often 40% when de-rating), then the device or the monitoring system will throw an alarm.

However, this simple approach to thresholding based on device power usage doesn’t effectively capture all the conditions of the broader power distribution system. There can be hidden capacity that allows for safe failover, even though simple device-level thresholding suggests otherwise.

Where Can I Add Load?

The goal of system ARC is to identify where you can handle additional load without sacrificing system redundancy. To calculate ARC for power of a device in a dual-feed situation, the calculation is simply:

ARC = {Device Capacity}/2 – {Actual Load}

In most cases, the Device Capacity will be de-rated to allow for some margin. In the case of power capacity, it is common to de-rate apparent power (kVA) capacity by 80%. ARC can also be expressed in real power (kW) if you know or can estimate the power factor of the load. It is even more important to de-rate the capacity in the case kW measurements to allow for potential load problems that could degrade power factor.

For operational alarming, calculate ARC continuously and alarm if it goes negative for any subsystem. For reporting, and determining where to install new equipment, be sure to use the minimum ARC measured over time, or equivalently, calculate ARC above with “Actual Load” replaced by “Maximum Actual Load.”

Below is an ARC-based dashboard in action:

An ARC-based dashboard. Click for large version.

Here, the top panel shows ARC calculated for 6 different data centers, along with a measure of cooling overhead in the same units, where available. The lower panel shows the drill down for one of the sites.

When calculating the overall ARC for devices in parallel, you can add the ARCs of the individual units. For instance:

UPS A has 10 kVA ARC
UPS B has 8 kVA ARC

Together, they have 18 kVA ARC. If we have another UPS pair supplying a different load, their ARC can be added to this set in order to get the system ARC. The rule of summing ARC for parallel systems doesn’t depend on the systems be redundant.

Calculating system ARC from the individual device ARCs in this way assumes that the capacities of both parallel components are the same. This is most often the case, but in the rare instance that it is not, then you have to total the actual load across the devices, and compare it to the (de-rated) capacity of the smaller device. This ensures that the most-limited device can handle the entire load.
Interestingly, it is possible to have a safely redundant system even though one of the individual devices has a negative ARC.

For example:

UPS A has 3 kVA ARC
UPS B has −2 kVA ARC

The net ARC of the system is a small but safely positive 1 kVA. In this case, even though one UPS is nominally overloaded according to the simple one-device threshold, either UPS can fail without dropping any load. Note that a simple single-device alarm threshold would show UPS B in alarm, and trigger a potentially costly load rebalancing.

Exact Redundancy Configuration Needed

In order to evaluate a negative component ARC, one does need to know the exact redundancy configuration. Thus, it is important to evaluate ARC for each subsystem and then roll it up to the data center as a whole.

When looking at the entire power chain, as with any capacity measurement, the system is limited by the weakest link in the chain. The component, or more accurately the collection of parallel components, with the smallest ARC will be the limit of the entire system. In the scenario above, if the PDUs downstream of the UPS have a collective ARC of 20 kVA, the load will still be limited to the 1kVA of the UPS.

Some questions may arise when the load is imbalanced, as in the UPS examples above. Such imbalances may arise because some of the load is not configured redundantly. We hope instead that some loads are switched, and simply not balancing themselves between the two power paths. The ARC calculation doesn’t depend on knowing such details. Of course, any non-redundant load will be dropped if it loses its power source; however, as long as the system ARC is positive you know that any redundant load will be protected regardless of which power source is lost.

In summary, the goal of system ARC is to identify where you can handle additional load without sacrificing system redundancy. With parallel equipment, you can total the ARC of all components if they have the same capacity rating. When looking at ARC along the power chain, the correct system value will be the minimum ARC of any one set of components.

Ted Ritter, an analyst at Nemertes Research Group, says many companies are having a hard time justifying an investment in alternative power sources right now — especially if it means completely replacing the reliable AC power already coming into their buildings, as opposed to merely supplementing it.

But some users are forging ahead with alternative energy projects anyway, figuring on a payoff within 15 years.

For the North County Transit District (NCTD) in San Diego, solar was the most obvious choice for alternative power. The organization’s data center is relatively small, but it’s big enough to enable the agency to handle ticketing for 12 million public transportation users annually and process video from security cameras in transit stations.

Angela Miller, the transit agency’s CIO, says her group felt a need to be a better environmental citizen. As part of a data center redesign, the agency spent about $600,000 on a 30-panel solar array, invested in virtualization technology for server and storage systems, and bought new pods that pull hot air out and help cool equipment inside the racks.

The NCTD sells solar-generated power back to the local utility to earn credits on AC power usage (which is allowed under California law), meaning the solar initiative has become a profit center. The solar panels don’t generate power for the building directly.

It works like this: The local utility sells AC power to the NCTD, then the agency sells the utility the solar energy for a 100% credit. The agency has a five-year plan in place to offset all AC power in its data center. The NCTD generates up to 450 kilowatt-hours of electricity, and it plans to reach 1 megawatt-hour in five years.

“Solar is what has made the [data center redesign] project even have an ROI,” says Miller.

Bob Mobach, a consultant at systems integrator Logicalis Group, helped the NCTD redesign its data center. He says a key to realizing an ROI with alternative power is embracing virtualization. The agency’s data center is about 80% virtualized, and that’s a primary reason why the solar arrays are such a successful power source.

“Virtualization was critical for so many reasons,” says Miller, noting that the new setup is “way more efficient,” makes better use of hardware, gives the data center a smaller footprint and is easier to manage with fewer people. “My actual physical footprint went from not having any more slots in the racks available to having only half of the racks occupied, and yet we’ve increased our applications this year,” she says.

To find out, our team at PTS upgraded the IT systems within our own facility and operations in order to validate the energy efficiency savings estimates.

Our first step was to create a baseline to measure the IT performance, capacity, and energy consumption. Next, we redesigned our IT systems with the goal of reducing energy consumption. We also wanted to increase the capacity without sacrificing performance. Lastly, we measured results to assess confirmation of the expectations.

In the end, we consolidated our sever footprint by 60% and reduced IT energy consumption by 24%, yielding a 26% drop in facility power consumption.

Our conclusion is that these results are not anecdotal in that the energy savings realized as a result of this study are completely scalable with larger, more complex data center and computer room facilities. Additionally, these energy savings may be realized without sacrificing IT performance and systems availability, while improving overall systems capacity.

Facebook’s Latest Innovations in Data Center Design
Senior Electrical Engineer, Facebook  Paul Hsu
Datacenter Mechanical Engineer, Facebook Dan Lee

Below is a side by side slide Paul presented on the difference between a typical data center power conversion vs. the Facebook design.

Dan has a slide with side-by-side comparison of a typical mechanical system vs. the Facebook design.

A couple of other slides share are on the Reactor Power Panel and Battery cabinet.

The results Facebook shared.

No one is under the illusion that the enterprise industry will suddenly get “container fever.” But with data needs on the increase and demand for rapid scale-out of physical resources still present, containers offer a ready-made solution for those who suddenly find themselves in a bind.

The latest to join the fray is Cisco, which this week came out with its own containerized data center (CDC) that the company says can be up and running within 120 days of purchase. Structurally, there is nothing unique about Cisco’s design. It is built on a standard, 40-foot ISO container and holds up to 16 racks of equipment with 25 kW of power for each rack. One novel approach, though, is rack-based water cooling, which provides a more granular level of heat dissipation and allows for more flexibility in system configuration.

For Cisco, HP, IBM and other builders of container systems, there seems to be very little rush to gain any sort of dominance in the industry. In fact, as IDC’s Michelle Bailey told HPCwire this week, selling containers is of secondary concern to getting servers, storage and other systems out the door. Less than 150 containers are expected to make it into customer hands this year, and even if that number doubles in 2012, we’re still talking only the tiniest fraction of the overall enterprise market.

Overseas, focus is shifting not so much toward containerized data centers, but modularized ones. Providers like the UK’s Colt Data Centre Services use a sectional approach that can more easily fit into existing facilities. The company has shrunk its offerings from the original 500 square meters down to 125 square meters, with power densities ranging from 750 watts to 3 kW per square meter. The company says it has devised more than 120 different design configurations, with roughly a four-month delivery window to anywhere in Europe.

Elsewhere, modularity is taking hold at some of the largest data centers in the world. India’s Tulip Telecom recently tapped IBM for its Enterprise Modular Data Center (EMDC) to expand a central facility in Bengaluru. As a side benefit, IBM also has the inside track to provide a number of other systems, such as power, cooling and rack systems.

Originally, the containerized system was envisioned as a quick solution for primarily small firms looking to build an instant data infrastructure. Now that the cloud delivers much of that functionality with little more than a basic network and Internet access, the focus has shifted. Containers are still seen as instant data centers, but are likely to become the solution of choice for larger organizations when new resources are required and only on-site hardware will do.