Five Percent: Conserve Energy

Climate Change Is Important: Energy Conservation is the First Step

April 11, 2009

Cloud Computing: Truly Green Data Centers

Category: Conservation,Save Electricity,Technology – Tom Harrison – 10:54 am

cloud-computing-greenThere has been a lot written about how much power is consumed by the computers that drive the Internet. A lot has been written about “green” data centers. But I think there’s a far more significant trend when, combined with more efficient computers and data centers, will make a 10x or greater reduction in power demand possible: cloud computing.

Data centers, brown or green are huge buildings — they are truly incredible places, with thousands of computers owned by multiple companies. I have negotiated the contracts for “co-location” in a number of data centers: you pay for floor space, bandwidth, and power and get a facility that has great connectivity, power that never goes out, and a carefully cooled environment for the computers. This blog, and most other websites are located at such data centers. Little sites like this one share a “slice” of a server with a number of others. Large sites like the ones we have at the Internet companies I have worked at have our own computers and other equipment “co-located” in data-centers.

I have been paying attention to power since 1998 when we moved the first servers for the Direct Hit search engine to a co-lo. Our first boxes were about 8″ tall, and bolted to special racks, and I think we got 7 to a rack, along with a few other needed parts. These days computers are many times more powerful, yet are 1/4 as tall — “pizza boxes” — thin computers that use a lot of juice, and put out that much more heat.

One of my companies had a space with ~100 servers at a facility in Waltham, MA but had to move — the data center “ran out” of power (the city/utility could not get variances for new power lines to their location). Over time the demand for space has decreased as servers get smaller, but the need for electrical power increased both because the servers needed more, but also the cooling systems (and backup generators) all needed to get larger.

One way to think of a data center is a bunch of hair dryers running continuously in a room whose temperature must be kept lower than 65° with air conditioning. Not a pretty picture.

Green Data Centers, Green Servers

google-solarRecently, data centers started figuring out that using good air-control methods, they could cool only the area in which servers were located, and (duh!) in cooler locales could draw in cold air from outside rather than using air-conditioners to make it. Facilities in hot locations are finding that the payback rates on improved insulation to keep the heat out are very quick. A few other tricks reduce the amount of energy needed both to power the computers, then cool the air they heat up.

And yes, data centers are also mounting solar panels on their roofs and coming up with other ways to reduce their consumption of power off the grid. But they aren’t doing this because it makes them look good — they are using solar and wind power, because big buildings with big roofs are perfect for this. It’s cost-effective.

These are “green” data centers.

Another factor, of course is the computers themselves. Servers are real power hogs — the ones we use in our current data center are between two and five years old use between 150W and 250W. Computers available today have more processing power, but draw a lot less power — here’s one from Silicon Mechanics that reports using only 122 Watts. This is good progress; more processing power (and therefore fewer computers) for less power.

These are “green” computers.

But green data centers and more efficient computers are only a small part of what will drive the biggest efficiency gains in the next few years. Here’s why.

The Incredible Expense of Redundancy

My company runs a number of eCommerce web sites. We do not make any money if the site is not working. So it always works (I just crossed my fingers and knocked on wood) — but the complexity and cost associated with this requirement is large indeed.

Every Black Cord is a Power Cable

Every Black Cord is a Power Cable

Presently, my company, DigitalAdvisor has a number of servers located at a data center in Boston for some of our large sites. We have enough servers to handle double our “peak load” — the number of requests we get at the highest volume time. (A website can have the good fortune of getting a link in some widely read publication — volume can increase by two or three time in a matter of minutes. And of course you plan for bad things to happen.) For our high-availability database servers, we have “hot spare” servers that we can switch to if the main one fails. We have two of everything, and its all plugged in, and most of it is sucking power … all the time. In one cabinet, we have a dozen servers running full time, drawing around 2500W. Oh, plus several more we run out of our office server room (which has a special air-conditioner we use in the summer).

But all of these computers are tremendously under-utilized most of the time. We run at between 1/8 to 1/4 of our available capacity for about 80% of the time during a normal day. Total running expense (not including the expense of the hardware itself) for one cabinet is around $2500/month, of which most is electricity: about 1/2 for running the servers, and the other half for cooling them. That’s a lot of electricity wasted — efficient processors or not.

The Efficient, Elastic Computing Cloud

However, based on my experience, the single most significant “green” change to hosted computing will be a result of cloud computing.

We’re now completing our move to Amazon Elastic Computing Cloud (EC2). Yep, this is the same Amazon that you buy stuff from, but they have this other, rather incredible business. It has been in the works for years, but just in the last few months has gotten the features and service levels to be appropriate for sites demanding extremely high availability like ours. Why are we moving? Not because it was a green choice, but because it cost so much less.

We get our own server “machine instances” — they work just like the servers we have at our Boston facility but they are “virtual” — we don’t have any hardware to buy or place to go to look at them. If we need a new machine instance for additional capacity, we start one (from a template that we have configured and saved) and the meter starts running — we pay by the hour.

Because we need to run only as many servers as we need at the moment, we waste very little. As demand increases we add server instances, and we take them offline as it falls. If a server fails, we just fire up a new instance of it (automatically in <2 minutes). If we need to test something, we run an instance for an hour or so, do our release and then away it goes. We don't need redundant power, redundant servers, spare capacity, complicated fail-over processes, servers used only for backup, development, staging, and periodic processing. And our equipment never gets outdated. We still have all the capacity we need and ability to recover from failures. Our running costs will be about 70%-80% lower, and we’ll have no server hardware investment. We’ll get a more reliable, flexible, managed, and even geographically dispersed installation. (In fairness, it’s a little more complicated to set up … although in some ways, its a lot easier).

Just because we can’t actually see our hardware, there is hardware behind all of this — computers, CPUs, and hard disks. And they do use electricity. However, even these servers are more efficient, because they use “virtualization” — our machine instance may be one of many on a single actual physical machine in one of Amazon’s data centers. Virtual servers perform just like a single actual machine, but the hardware (and operating systems) are designed to support many virtual servers at once. Through virtualization, Amazon uses up the full capacity of any one real server before they turn on (or buy) more. Servers use about the same amount of power when they are idling as when they are working … we make them work!

The Virtual Green Cloud is Here

So look at cloud computing from an overall energy standpoint — we use about 80% less server power, thus need that much less cooling and that many fewer computers. Add virtualization and we use even fewer physical resources so the “embedded energy” of the hardware is even less.

The only losers in this whole new paradigm: the data centers (at least the ones who haven’t gone to cloud computing on their own) and the computer manufacturers.

Solar panels on the roof, more efficient cooling, and other “green data center” features are great improvements. But by enabling us to stop wasting power and resources our business, which is just one of many thousands like it, will save money and be green in a cloud computing environment.


  1. […] I come across many great studies on building Green Data Centers, I was reading this one today:  Cloud Computing: Truly Green Data Centers […]

    Pingback by Cloud Computing and Green Data Centers — April 12, 2009 @ 3:44 pm

  2. Don’t forget that you should have a backup to Amazon EC2. I live about 20 miles away from the datacenters that house the us-east1, 2 and 3 regions, and, though separate, they still could have a catastrophic issue and take your entire production, well, out of production.

    Because Amazon uses a modified version of Xen, consider finding a second cloud-based company that supports Xen (or any other cloud computing method) and make sure their datacenters are somewhere other than Amazon’s. This way, if there is some sort of problem with Amazon, you can at least get a few servers with the bare essentials started up, change your DNS, and be back up, even if at a somewhat lower capacity.

    And though YOU aren’t paying for it directly, all of your virtual machines are running on hardware that is, potentially, at 1/8 to 1/4 in capacity most of the time. It is likely that Amazon does everything in its power to, well, use less power per computing unit, and that their datacenters are more efficient than the one you are replacing, but there is still the cost of running the systems — A/C, power, filters, UPSes, multiple phases, etc.

    Comment by ooglek — April 20, 2009 @ 11:29 pm

  3. Hey ooglek —

    What you say is true, although I would say you’re talking about a pretty serious catastrophe. Our sites are eCommerce — I am looking for almost no downtime, but we’re not a bank, a flight control center, an online porn site, or some other absolutely critical resource that can’t go down ever.

    Our former data center is also in a single location in Boston so from a facilities standpoint we’re about at par. From a backup and redundancy standpoint we’re far better off, if only because it’s so simple to make a volume snapshot, and Amazon claims to store their S3 data in multiple physical locations. It’s also the case that Amazon has an entirely different data center now in Europe, and is working on getting a West Coast US center hooked into the EC2 fabric. It’s true that Amazon’s SLA for up-time is not quite at the level of the first-tier providers … however what matters to me is the ability to recover from an outage quickly; having no physical hardware removes a great deal of logistical issues.

    As for the utilization of hardware (which is what my post was really about :-), a friend who works at Amazon explained about their strategy. In effect, the larger their scale, the more they can use the excess capacity of a given machine to handle their own load and storage requirements. Of course their machines at not at 100%, but as scale increases, a smaller and smaller percentage of idle capacity is needed, and the cost per compute-unit (and storage unit) falls for Amazon, as well as for customers.

    While there are highly predictable daily and weekly trends for usage, the fluctuations that characterize our business are different than others; when we need more, someone else may need less — it’s the same model as insurance: spread risk over volume. Of course the same costs are required to run their servers — power, cooling, UPS, etc., but because it is done at scale, it can be done more efficiently.

    In our case, at least, there’s no doubt that the total cost of energy (both running and embedded) is far less (many times) in a cloud environment than in our traditional co-lo data center.


    Comment by Tom Harrison — April 21, 2009 @ 8:23 pm

  4. Thanks for the response Tom!

    I agree that Amazon can operate at a higher operating capacity than 25%, but there still needs to be available resources, as it could be bad if someone needed a lot of resources and Amazon couldn’t deliver.

    Having a backup plan is easier and cheaper when using cloud computing, since really you’re just running a few services on a virtual machine, plus your own codebase. I highly recommend looking at a few other companies who do cloud computing, and have a backup plan in place. EC2 is likely very complex, and a bad upgrade of server hardware could really hurt things. There was a several hour outage in the last 12 months where new EC2 instances could not be provisioned. This wasn’t a problem for most, but at a crucial time, it could be. While a full on outage is likely able to be mitigated, having a disaster plan is always a good idea. And with cloud computing, it’s easier to do, as you don’t need to deal with physical hardware.

    From your blog standpoint, I agree that that cloud computing will result in less energy usage for the same amount or more capacity to serve your customers, while simultaneously making you more nimble.

    Comment by ooglek — April 21, 2009 @ 8:53 pm

  5. Hey Tom,

    I came across this interesting piece tonight and thought I would share….this sounds like a fairly good idea, no?

    April 21, 2009
    Peer-to-peer power through microgrids
    Filed under: Complexity, Economics, Electricity — Tags: Complexity, Economics, Electricity — lkiesling @ 6:10 am
    Lynne Kiesling

    When we think of concepts like peer-to-peer networks and disintermediation, we usually think of industries that are very Internet-centric. But these concepts can, should, and will apply in electric power networks too: smart grid technology enables peer-to-peer power.

    The study referenced in that BBC article analyzes the potential for microgrids, and argues that the real potential from applying smart grid technology to create microgrids is in the ability to create a neighborhood peer-to-peer network in which neighboring customers can buy and sell from each other:

    “A microgrid is a collection of small generators for a collection of users in close proximity,” explained Dr Markvart, whose research appears in the Royal Academy of Engineering’s Ingenia magazine.

    “It supplies heat through the household, but you already have cables in the ground, so it is easy to construct an electricity network. Then you create some sort of control network.”

    That network could be made into a smart grid using more sophisticated software and grid computing technologies.

    As an analogy, the microgrids could work like peer-to-peer file-sharing technologies, such as BitTorrents, where demand is split up and shared around the network of “users”.

    As distributed generation and plug-in hybrid vehicles proliferate in the market, more numbers and types of electricity consumers will have the resources to be both buyers and sellers in such a peer-to-peer network. Look, for example, at the picture of a P2P network at the Wikipedia peer-to-peer entry.

    Now imagine that instead of computers, each of the entities depicted on this network is a home or small business in a microgrid network. Power, and commercial transactions, can flow in both directions between pairs on the network, and they can flow between any pairs of agents who have agreed to participate. Just think of what that can do to reliability, especially if you pair it up with transactive, price-responsive end-use technologies that have the type of behavior I described in this post on smart grid and complexity and this post on how intelligent end-use devices make a transactive smart grid valuable.

    If you are interested in learning more about a microgrid project, here’s a report on the Galvin Electricity Initiative prototype microgrid project at the Illinois Institute of Technology. It focuses on the technical details and capabilities of a microgrid to provide reliable, high-quality electric power service, not on the microgrid’s transactive capabilities, but it’s a good introduction.

    The technology exists for P2P power networks. The institutional structure, though, does not allow for such a decentralized, transactive network — the regulatory environment typically does not allow microgrids for a variety of reasons, including the monopoly granted to the local utility on the construction of distribution wires that cross public rights-of-way.

    Comment by Jean — April 21, 2009 @ 9:22 pm

  6. Jean —

    There is a clear and strong linkage between energy generation and computing. In the early days of computing, there were a small number of very large computers, made by a small number of manufacturers, all working independently. Interconnection was mostly as a second thought. Then came PCs, and for a long time, they were novelties, and then started working together a little, and even a little with the big computers, but only with a fight.

    The Internet changed everything — it forced a standard set of ways that computers could inter-operate; you could either accept the standard (which was, by the way, none of the ones by big companies like IBM, Microsoft, Novell and others), or be all alone. By the mid-1990s, finally everyone supported that standard, and things began to happen.

    The computing standard supported two way communication, distributed computing, instant and automatic routing, fault-tolerance, and all built on an open, non-proprietary standard.

    All of these lessons, and sadly some of the challenges, apply to energy. Local generation today is mostly a novelty — rooftop solar or wind seems quaint, and even if you do plug it in to the grid, the power company has no idea how much you generate, or what to do with it.

    Smart Meters are one part of the open standard that will allow your house to be part of a power network; they are smart because they can communicate, whether back to a large power plant, or to your next-door neighbor.

    The Smart Grid is the Internet for power — providing efficient routing, inter-operability (coal, gas, solar, wind, nuclear, whatever!), and an open platform that can instantly balance supply and demand.

    Getting from the extremely dumb and fragile and inefficient system we have today to one that can support a peer-to-peer or other decentralized energy distribution platform will take a lot of work, but mostly it’s about getting all the big companies who want to “win” to look back and realize that the real winners of the Internet are and will be the companies that seek open standards.

    We need to do everything we can to support the efforts of the Smart Grid and make sure that the open standards as proposed by the US and other governments are adopted quickly; with that out of the way, we may be able to move quickly to enable an energy revolution.

    And as a side note…

    The idea of local generation is far from new. I have on my bookshelf a copy of “Soft Energy Paths”, written by Amory Lovins, in 1977. He presents a very strong argument for local generation and distribution, and not once with any mention of global warming — one of his main arguments was that local generation provides greater security (no central plant to bomb), and reliability, and flexibility, and efficiency. If you look up the Smart Grid, you’ll see that these are exactly the same goals. 30 years later, we’re getting started.

    Lovins founded a great organization called the Rocky Mountain Institute (RMI) — they are still around and thinking the next great thoughts we’ll all come to see as wisdom in the future :-)

    Comment by Tom Harrison — April 21, 2009 @ 10:27 pm

  7. Here’s an interesting and topical link. CitiGroup has build a LEED Platinum data center that “consumes 30 percent of the power required for a conventional data center”.


    Comment by Tom Harrison — May 4, 2009 @ 7:01 pm

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.