The Operations Supplement

Sponsored by

Operations Supplement

INSIDE

Understanding data center M&E

Greasing the wheels

Concrete’s carbon curse

Not a drop to waste

> If you want your data center to be reliable, you may need to use lots of lubricant

> Can changing concrete’s composition curtail carbon in construction?

> Freshwater is running out. It’s time to take water management seriously

Secure your data center

You can never stop the rain, but you can harden your data center to avoid water ingress. Use Roxtec cable and pipe seals to prevent flooding, protect critical infrastructure and ensure business continuity. #nomoredowntime roxtec.com/datacenters

Operations Supplement

Sponsored by

Contents Getting real

4. Not a drop to waste As freshwater supplies dwindle, data centers need to look elsewhere

D

ata centers can become a very abstract - indeed, virtual - subject.

Their security, their efficiency, and even their finance can be little more than figures on a screen.

6. Don’t let hardware let you down If you want your data center to be reliable, you may need to use grease 8. Advertorial: Water incursion It's among the biggest causes of downtime - and getting more likely 10. Curing concrete's carbon curse The most basic building material has the biggest carbon footprint 13. M&E security: The killer in the chiller The cyber flaws of electrical and mechanical equipment are being fixed

But when your data goes to the cloud, it's still in on a physical drive somewhere. And when a high street shop closes, we can mourn the passing of a brick-and-mortar business - but the online retailer that replaces it is still housed in a physical building.

This Operations supplement is a reminder that our digital infrastructure has its roots deep in physicality.

Virtual operations may have no friction, but data centers have to take corrosion and water ingress into account.

A data center is a concrete structure, with physical needs.

The concrete curse

4

10

13

Before the data center is even built, its carbon footprint starts to rack up, when the first batch of cement is prepared and mixed into concrete (p10).

Our sector has the double virtues of high technological awareness, and strong financial resources. So it makes sense that data centers should be among the first industries to step up to the seemingly impossible task of counteracting the environmental impact of concrete.

Concrete's carbon emissions are a fundamental byproduct of chemical reactions that create it.

So the answer is to find ways to reverse that. We met the people who are fighting chemistry with chemistry.

The killer in the chiller We've known for some time that attackers can hit a data center through its mechanical and electrical equipment can open thanks to network connections which have not been secured (p13).

As maintenance becomes automated, more equipment is accessed remotely, and the chances of a serious incident are increasing.

But, finally, practical steps are being taken to close this loophole.

Keeping water in check Data centers need water. They drink too much of it. But in the wrong place, it can literally bring the facility down.

The world's largest data center operator is taking its responsibilities seriously: Digital Realty tells us how it is reducing its use of clean water (p4).

And Roxtec tells us how to keep water in its place, by preventing incursions (p8).

Greasing the contacts Hard drives, fans, chips, and contacts are the physical units that support our digital infrastructure.

They may be high-tech and solid state, but they need maintenance, especially when new ways to put data centers together increase wear and tear (p6).

Facing reality This year, the Covid-19 pandemic has locked us into virtual worlds, but it's also made us think again how to manage the physical business of keeping data centers working.

At DCD, we promise to keep grounded in the reality that supports our digital life.

Operations Supplement 3

Operations Supplement

waste

Not a drop to

As freshwater supplies dwindle, data centers need to look elsewhere

F

reshwater is increasingly in short supply.

The march of industrialization, extensive use of limited groundwater reserves, growing populations, and the impact of climate change mean that access to water is going to become one of the defining struggles of the 21st century.

"There's gonna be a 40 percent gap in freshwater supply and demand by 2030 according to the UN," Nalco Water VP and GM Heather DuBois told DCD. "It's just staggering, but yet we need to continue to grow."

Data center demand shows no sign of letting up, with gigawatts of capacity expected in the years ahead - much of which will be cooled with water.

That means a lot of water, at a time when the world needs it the most. That’s simply unsustainable.

Take Digital Realty - back in 2018, its data centers used around 1.4 billion gallons of water in a year, an astronomical sum, and that’s even with the majority of its facilities not using water for cooling. If all of that was potable water, it would be a wasteful travesty.

"Water is a key element of our environmental, social, and corporate governance journey," Digital's director of sustainability programs Aaron Binkley said.

“43 percent of our water use came from reclaimed sources last year - that's more than 660 million gallons of water across the portfolio.”

To get to that point took a significant amount of work, the company’s water sustainability lead Walter Leclerc told DCD. Getting to 100 percent water reclamation will take significantly more effort.

First, the company partnered with Nalco and parent company Ecolab to assess the

4

water use and water risk at all of its data centers. “We used the Water Risk Monetizer, which is a tool that Nalco, Microsoft and Trucost put together, to place each data center on a water maturity curve,” Leclerc said.

“We've done that at the local level, and now we've prioritized our hotspots. We know as of today which of our sites are high-risk water sites, medium-risk water sites, and low-risk water sites.”

This has meant collecting “a tremendous amount of data,” Leclerc said. “It can take three to four months per site to do assessments - Nalco went to every one of

Sebastian Moss Deputy Editor

company was able to take steps at each site to help reduce freshwater usage. “It can include everything from sulfuric acid dosing to reclaim water, to implementing pretreatment on systems, to looking at collecting rainwater.”

Harvesting precipitation “takes a tremendous amount of effort,” Leclerc said.

“We actually have two major projects that are about to come to a conclusion now, and they took two years to do.”

While the data center company likes to standardize designs where possible, the reality is that different locations have different needs and capacities. “Down

"The public is going to hold us to these as we move forward. I don't think anybody's going to have a choice." our water sites across the globe, they did a design assessment, they took samples.

“They first take water quality assessments, and then they do assessments of the watersheds, where those data centers were taking it off of. So it took a long time.”

State and local utility providers offer little in the way of data on water supplies, with limited regulatory data offered, along with a mandatory requirement to test for Legionella. “But that didn't help us from a water stewardship perspective,” Leclerc said. “We need to understand the quality of the water, we're talking about the pH, we're talking about the solids in it, and so on, because we need that information to develop the pipeline of projects at a site.”

Armed with up-to-date information on water use, availability, and quality, the

DCD Supplement • datacenterdynamics.com

in our Phoenix, Arizona, site we initially reclaimed water into that system, but then we outstripped the capacity of the city,” Leclerc said.

“So they actually told us to go back onto potable water. Now the city has caught up with us, and they said they can handle our reclaimed water volume, so we're trying to do another project next year to get it back.”

In places like Ashburn, where Digital has multiple huge campus developments, “those are opportunities to have a strategic dialogue with the water authority and say, ‘is reclaimed water available next to our campus?’” Digital’s Binkley said. “Or if it's not, we can give them some general guidance on what our water consumption would be and what our demand profile would look like.”

Due to the proximity of other companies’ data centers in some areas, it can make sense to collaborate with rivals - as well as non-data center companies - to make a case for reclaimed water availability. “We know Amazon is doing a huge building in Ashburn, so that's going to affect the reclaimed water supply,” Leclerc said.

Sometimes the particular location of a data center can open up unique cooling solutions, like Google found with its Hamina data center, which uses a cooling system it inherited from the paper plant which preceded it to draw on seawater.

With Digital’s Marseille facility (under the Interxion brand), the team realized that they were relatively near a decommissioned coal mine “that has waste cold water that's coming out of the ground that was traditionally pumped into the river and out to sea,” Binkley said.

“That's being piped about 14 kilometers to our Marseille data center campus, and it's being used as a chilled water source to limit cooling energy needs. It's helping to make the facility more efficient by getting free cooling out of what is a wastewater stream from an old coal mine.”

While the environmental benefits of such water saving moves is clear, what can make it easier for companies to sign off on is that it can also lead to cost savings. “We want to do the right thing when it comes to water stewardship,” Leclerc said, “but also the return on investment on some of these projects is like seven months,” with reclaimed water usually less expensive than freshwater.

“In the old days, the ROI was 10 years,” he added. “We're just finishing up a two year project at our Santa Clara location, where we're going to be taking 16 million gallons of potable water off that watershed and going to reclaimed water. This is great, but I'm also going to be collecting hundreds of thousands of dollars a year from the cost differential for the operations team. So it is a driver, absolutely.”

Still, the financial benefits aren’t as profound as one would expect given water’s precious nature. “Water is cheap, even in warm places where it is a scarce precious resource it's still relatively inexpensive,” Binkley added. Further savings can be made with evaporation credits, or in spending more on longer pipes that reach areas with lower water rates.

A number of customers are already demanding at least some action on freshwater usage. “Customers are becoming aware of water concerns and many of the big hyperscalers have made commitments,” Binkley said, although the level of demand for action is currently much less than that for renewable energy.

Here, Digital is trying to work out how to involve the customer in sustainable decision making. For renewable energy, the company has partnerships with solar and wind companies that mean its customers can decide whether to use renewable energy via power purchase agreements, or just run off of the grid. For water, it’s not immediately clear how much control the customer can have.

“We just started those conversations in the last month or so,” Leclerc said. “How can we bring Ecolab into our RFP process, and how can we leverage the partnership to do that with our customers? We're just not there yet.

“We're kind of hung up on it. It's easy when it's the customers paying for the source, but if they're just a customer in a site and they're not part of that water matrix perspective, it’s a lot harder.”

However that is achieved, it is expected that water sustainability will increasingly feature on customers’ priorities, just as we have seen the rapid expansion of demand for renewable energy sources for data center

“We want to do the right thing when it comes to water stewardship - but also the ROI on some of these projects is like seven months." workloads over the past few years.

Water scarcity is only going to get worse, and companies needlessly using limited freshwater resources will rightly face the ire of a desperate populace. “The public is going to hold us to these as we move forward,” Nalco’s DuBois said.

“I don't think anybody's going to have a choice as we look to the constraints that the environment is going to have in the coming years.”

Operations Supplement 5

Operations Supplement

Don’t let the hardware let you down If you want your data center to be reliable, you may need to use grease

A

data center is a complex system comprising technology and the humans that run it. Increasing reliability is a combination of many things. You need a good design, with no single points of failure. You also need to make sure your staff are well trained, to minimize the risk of human error.

And you need to maximize the fundamental reliability of your hardware.

Research suggests that there’s an upper limit to reliability, and no systems can be expected to work continuously for more than 200,000 hours (more than 22 years).

But the only way to get anywhere near this figure is to address the mechanical issues that will eventually bring down any hardware.

Preventing hardware failure is a combination of making sure you have reliable, high-quality equipment, and taking all steps to make sure it doesn’t wear out.

Most of the hard work is going to be done by vendors, to ensure kit is reliable before it arrives, but once inside the data center, it’s up to the owner of that equipment to take care of it.

“Maintenance is an insurance policy,” says Brian Kinkade, market development manager at Nye Lubricants. “It makes sense to do it.”

While most data center technicians already keep a close eye on temperature, humidity, and dust in the circulating air, Kinkade says the industry should also

6

be aware of physical wear and tear. His company specializes in preventing it, and his social media slogan is: “Solving reliability and performance challenges with grease.”

Data centers include plenty of moving parts, and also some non-moving parts which benefit from lubrication, says Kinkade, who has been developing business in the sector recently. To keep data centers up-and-running and minimize unplanned downtime, engineers have to know about potential failures and address the issues, he says.

Cooling fans and hard drives need specialized lubrication, and so does the large-scale mechanical and electrical plant, including cooling units, air handlers, and diesel generators.

Surprisingly, some of the more static parts of the data center also benefit from lubrication, Kinkade explains. The part of the data center which first got him involved in the sector is the busbar.

Busbars deliver power from the electrical room to the racks and servers. Although they’re stationary, they can still suffer from “fretting corrosion,” where a steady period of micromovements produces wear on the contacts.

The wear can remove plating on the

DCD Supplement • datacenterdynamics.com

contacts, exposing the underlying copper, which can then be oxidized. The oxide layer acts as an insulator, making the power connection less efficient, and eventually causing it to fail.

A lubricant film prevents that by minimizing contact between the metal surfaces, says Kinkade. It’s standard practice in high-vibration applications such as in motor vehicles, but he has found from experience that data center operators need to consider it on their busbars.

In normal operation, most data centers have little serious vibration - though this is something that might change when Edge roll-outs start placing some near are subway stations (see box) - but to push reliability high, lubricant on busbars can be good insurance.

But there’s another issue: to make the most of data center technicians, many organizations have moved to “rack and roll”

Peter Judge Global Editor

installation, where racks are populated with equipment in a central factory, then shipped to the data center for installation.

“When they plan on shipping racks fully loaded to the data center, operators and OEMs should do a transport, shock and vibration test,” says Kinkade. The vibration and shock during transport may be more than the equipment is designed to handle - and applying a lubricant during assembly may be a way to ensure the equipment isn’t harmed during transport.

At this point, the operator needs to look at the warranty, he says: “It’s a certain bunch of agreements as to where some equipment will work and for how long.” The connections between a server and a busbar will be designed to support a certain number of insertions, and a certain amount of vibration and shock.

Servers and switches are mostly designed to be inserted in a stationary rack, not inserted in a rack which is transported. It’s conceivable that shipping a rack with the servers inserted might exceed the lifetime expectation for the physical wear on those contacts.

As rack and roll has become standard, the OEMs have understood how to handle the equipment, and Kinkade believes hardware vendors have taken it into account, perhaps thickening the plating on contacts.

“There are still people that choose to ship stuff separately and install it at the data center. There is a mix,” he says. It will be down to the customer to make sure that the equipment they use is up to the use to which

it’s being put. Operators should keep careful track of the tolerances - to make sure that the equipment is sufficiently rugged, and apply any extra lubrication that is needed.

Another point about lubricating those contacts is that applying more lube can mitigate existing damage to connectors and prevent further fretting corrosion. Lubricants designed for the automotive industry are a good choice, as a car is an environment with plenty of vibration.

Nye also provides lubrication for cooling fans. Bearings are sealed inside the unit, and the best kind are sintered bearings - which consist of pressed metal powder with pores which are impregnated with a lubricant designed to keep the fan permanently lubricated for its lifetime. The lubricant should include antioxidants and be rated to work at high temperatures.

Because they are sealed, the lubrication is down to the fan manufacturer, and the same goes for hard drives, which have fluid dynamic bearings to allow the spindle motor to move freely without friction and wear.

As well as physical properties, that lubricant should have the right thermal range, so it doesn’t oxidize and cause debris.

Most of the time, apart from busbars and connectors, any lubrication is handled directly by the equipment vendor, and is a one-off intervention during manufacture.

But there is one exception to this - which may be on the increase.

As the industry becomes more focused on its environmental impact, moves to refurbish and re-use data center equipment are coming to the fore.

“If there’s a refurbish in the business model, where last-generation servers are repurposed, you could want some grease in that process,” says Kinkade. “As an insurance policy to minimize failures, it might make sense to do it anyway. You would also want to test and measure it to make sure it works.”

And of course, any reuse process is going to involve transportation again. If equipment is re-shipped loaded into a rack, then that will introduce further vibration and wear, and might need further grease.

It’s worth underlining this point.

IT people may focus on refurbishing the hardware, and clearing any data by scrubbing memory and drives.

The physical danger to IT systems might only be obvious to someone who lives and breathes lube.

Maintaining the Edge “People aren’t putting data centers next to subway stations,” says Kinkade, but he might be wrong there.

Edge data centers are emerging, which have to be positioned close to sources of data and users of that data, to support applications This means that small facilities are being proposed in sites such as the bases of cell towers, and in urban environments.

Some of these micro facilities could end up close to subway stations, or in other sites where there are environmental hazards.

Concerns have been growing that those installing such micro Edge data centers may not have taken the new environment into consideration.

Edge providers should be careful not to assume that the same maintenance routines applied in large data centers will apply in the Edge. In a recent technical bulletin ASHRAE, the group which specified much of the best practice used in large data centers, examined how small Edge facilities should be treated differently.

As well as the possibility of vibration, Edge data centers may get more exposure to outside atmospheric conditions. They are likely to only have one door between the equipment and the outside air, so any maintenance process could expose the electronics to condensation and dust, including gypsum and salts.

“If they get down inside a contact, like a DIMM or a processor, and you do a service on those, what you can do is actually smear the particles onto the contact. I liken it to smearing peanut butter or Nutella on toast,” said Jon Fitch, a data scientist from Dell and a member of ASHRAE’s TC 9.9 committee, in an interview with DCD (see DCD Magazine issue 38). “It's very thick and viscous and, by golly, if you want to get it off the toast, it's pretty hard to do!”

Blocking dust and condensation could be addressed by lubrication and other maintenance practices.

Servers are mostly designed to be inserted in a stationary rack, not in a rack which is transported - shipping a rack with servers might exceed the lifetime expectation for the physical wear Operations Supplement 7

Prevent The Threat Of Water Incursion Water incursion is among the biggest causes of data center downtime – and climate change increases the risk

I

n June this year, authorities in China discovered something extremely worrying. The flagship Three Gorges Dam in Hubei province had developed a number of cracks and, with the rainy season still with a month to run, speculation mounted that the two-kilometer long Dam could break.

That wouldn’t just cause power cuts. A break in the world's largest hydroelectric power station could cause a tidal wave that would flood thousands of villages, towns and cities downstream, including the provincial capital of Wuhan, putting the lives of tens of millions of people at risk.

But an equally worrying everyday concern for authorities in China is seepage – the sheer weight of water slowly forcing its way under, and maybe through, the mega-structure, possibly corroding its integrity in the process.

For infrastructure large and modest, water damage is one of the most insidious of everpresent threats. Indeed, Ponemon Institute’s

8

National Survey on Data Center Outages indicated that while UPS battery failures featured in 65 percent of data center outages, water incursion was cited in 35 percent.

Better by design It goes without saying that the best risk mitigation for a data center is good design from the outset, with the facility sited in a well chosen, low-risk location.

In terms of location, environmental risks can be identified from official sources, such as the Environment Agency’s Public Register for Environmental Information in the UK, the European Environmental Agency in the EU, and the US Geological Survey (USGS), National Oceanic and Atmospheric Administration (NOAA) and the Federal Emergency Management Agency (FEMA) in the US. A Roxtec white paper, Water Mitigation in Data Centers, points out that historical data will highlight the areas at greatest risk from natural disasters, including flooding. This knowledge

DCD Supplement • datacenterdynamics.com

Water Mitigation in Data Centers Click here Humidity Effects in Substations Click here

can ensure that a good location is selected.

At a more granular level, site selection guidelines can further mitigate some of the risks posed by flooding, which ought to be highlighted in any good survey. These include elevation, the natural incline of the site and the typical level of the local water table. The latter can provide an indicator of the risk of water intrusion under heavy or even moderate levels of rainfall.

The ANSI/BICSI 002-2019 Data Center Design Standard provides guidance for selecting a site with a low water table, highlighting the risks of water ingress into facilities in low-lying areas. This may not directly threaten a data center facility itself, but vaults, ducts, and electrical equipment could all be at risk, with knock-on effects for resilience, while increased moisture levels could also pose a longer term threat.

Moisture penetration may not directly cause a failure, but can contribute to the deterioration of the insulation system, leading

Roxtec | Advertorial

to equipment failure, as well as corrosion, and cable and equipment failure. Research also indicates that exposure to moisture is one of the biggest contributing causes of switchgear bus failures.

Long term damage One of the most common longer-term forms of insulation damage to cabling is ‘water treeing’, the result of water ingress in microcracks in stress points, which are further weakened by the presence of water, especially if subjected to a cycle of freezing and thawing.

Even cables with insulation optimized for challenging environments can be compromised by splices, and terminations and joints subjected to poor workmanship by installers. These shortcomings may not lead to an immediate failure but, longer term, water and moisture ingress in and around power distribution ducts can lead to premature cable failure and, therefore, downtime.

It is not just power cables, either. Fiber optic cables are also threatened by moisture, with well-documented effects including signal attenuation due to water molecules penetrating micro-cracks, connector corrosion, signal loss and even mechanical damage due to freezing.

Some elements of design, such as sloped ducts directing water away from buildings and equipment can help, as well as locating vaults above the local water table, but moisture damage is an ever-present threat.

Storm rising

uncertainty and new challenges. Higher temperatures will mean greater humidity – every one-degree celsius rise in temperature causing a seven percent increase in ambient moisture – and many places will be subjected to more rainfall, more frequently, as a result.

Rising sea levels will also increase the risk of catastrophic flooding in coastal and low-lying areas.

A report by the UK’s Department of Environment, Food and Rural Affairs, Adapting the ICT Sector to the Impacts of Climate Change, highlights the main risks, both above and below ground.

And, while there is much uncertainty surrounding climate change and what it will mean, the US Global Change Research Program report Climate Change Impacts in the US puts the risk of rising sea level into context. It calculates that over the last century, sea levels rose by approximately one foot (20 centimeters), with the rate of increase doubling since 1992.

However, the uncertainty caused by climate change is reflected in the wideranging estimate for sea level rise over the rest of this century: between one and four feet (30 centimeters to 92 centimeters).

As things currently stand, a rise of just one foot (20 centimeters) will affect 235 data centers, 771 points-of-presence (PoPs), 53 landing stations, and 42 Internet exchange points (IXPs), according to the academic paper, Lights Out: Climate Change Risk to Internet Infrastructure.

If climate change causes sea-level rises at the upper end of estimates, cities in the US such as Boston, New York, Miami, San Mateo, and Newark will all be directly affected by the end of the century, and more than 4,000 miles of fiber conduit will require engineers to be equipped with flippers and snorkels in order to access them.

Of course, the risks posed by building data centers in coastal cities – where capital cities are typically located – can be mitigated by building further inland, notwithstanding

Fiber optic cables are threatened by moisture, with well-documented effects causing serious damage the need for many operators to be located as close as possible to financial centers for the usual reason of latency. However, climate change is expected not just to bring increased precipitation, but more frequent heavy rainfall events and, therefore, flooding. Already, according to the US Global Change Research Program, the US Northeast has experienced a 71 percent increase in heavy downpours, while elsewhere across the US, increased rainfall has been more modest, but still significant. The potential impact on rivers and the local water table will need to be factored into site selection considerations.

That means that in addition to closely following the ANSI/Telecommunications Industry Association’s TIA-942 standard, more attention ought to be paid to devilish details such as utility ducts and distribution vaults as potential sources of water and moisture ingress. Obviously, specifying high levels of mechanical sealing, purpose-built for underground ducts, at the construction stage is ideal, and more cost-effective than applying remedial solutions after a problem of water ingress has been discovered.

Rooftop penetration should not be overlooked either, with cooling systems and other rooftop infrastructure also audited for a similar level of sealing against water ingress, as part of a wider evaluation of potential penetration throughout the whole building envelope.

As things currently stand, only a minority of data center operators have even started taking climate change into consideration. But the bottom line is that many of the steps that can mitigate the risk of water incursion today will be even more relevant in the future.

Roxtec develops, manufactures and delivers flexible sealing solutions for cable and pipe penetrations. Roxtec modular-based transits are used for cable sealing and pipe sealing in numerous demanding industries and projects worldwide as the seals provide certified protection against multiple risks, including fire, gas and water. www.roxtec.com/datacenters info@roxtec.com

Climate change, of course, will add both

Operations Supplement 9

Operations Supplement

Curing concrete’s carbon curse Data centers create emissions before they are built. It’s time to address the footprint of basic building materials

T

here’s a massive source of emissions in your data centers - and you may not be aware of it. Before you even install any equipment, the building itself has already created tonnes of carbon dioxide and added to global warming - thanks to the concrete in the walls and floors.

Concrete is the second-largest source of greenhouse gases on the planet, contributing eight percent of the world’s CO2 emissions, making it a worse polluter than the airline industry. Every building put up adds a burden to the atmosphere, and the structural material of a data center

contributes to its “embodied” environmental impact, before the equipment begins to use energy and create an “operational” environmental impact.

Far-sighted data center operators are starting to pay attention to this. The most environmentally friendly approach is to avoid pouring new concrete where possible.

Some companies, like France’s OVHcloud, make a point of always re-purposing buildings, to avoid the environmental impact of demolition and rebuilding.

Some hyperscale companies, that have to build new giant data centers, are paying attention to the impact of their concrete. In 2020, both Amazon and Microsoft invested

10 DCD Supplement • datacenterdynamics.com

Peter Judge Global Editor

in a company that reduces the emissions of the concrete industry, by injecting waste CO2 into the mix.

The company, CarbonCure, already has at least one customer in the data center industry: Compass Datacenters. It hopes to remove 500 megatonnes of carbon dioxide annually from the concrete industry by 2030.

For CarbonCure, the problem is much wider than data centers - because the world is in the throes of an orgy of building, which is constructing the equivalent of a new New York City every month.

“If concrete were a country, it would be the third-worst polluter after the US and China,” said Eric Dunford, director of

sustainability at CarbonCure, in a recent online lecture for the American Institute of Architects (AIA).

“Most of the growth is coming from nations that are urbanizing and catching up with where the US and Europe are now,” said Dunford. “China has poured more concrete in the last three years than the US has poured in its entire existence as a nation.”

Concrete is made of a combination of aggregate - material like sand or gravel - and a cement which binds it together.

Concrete is a fine powder which magically “cures” when water is added, forming insoluble crystals which bind the aggregate together. It’s wonderful stuff, but it’s an environmental villain: it makes up around 12 percent of the weight of the concrete, but produces 95 percent of its CO2 footprint.

“The emissions mostly come from a basic process that is impossible to do without if we want to have cement and concrete,” said Dunford. “We go out and dig up limestone deposits, also known as calcium carbonate.

That enters a cement kiln, which is heated at a high temperature [2600°F/1400°C] under pressure, in order to get calcium oxide which is the precursor of cement.”

Heating the limestone to that high temperature obviously requires energy, and the energy source will produce emissions.

But there’s a further cause of emissions: calcination.

Limestone is heated and loses water and carbon dioxide, becoming “clinker” which is largely calcium oxide. It is then ground into the fine cement powder used to make concrete. But when the calcium carbonate in limestone becomes calcium oxide, it gives off carbon dioxide.

“That CO2 is shot out of the stack and becomes part of the global CO2 in the atmosphere,” said Dunford.

Making a tonne of Portland cement creates 1.04 tonne of CO2, and other emissions are minimal by comparison, he says: “At the end of the day, for every tonne of cement that goes into your building, you’re looking at a tonne of CO2 being produced. Production and transport are minimal compared to the impact of the cement.”

There are a few ways to reduce the amount of emissions from concrete. The first is to reduce the emissions cement kilns cause, by shifting the fuel used to renewable sources.

The second, and most obvious, way to reduce emissions is to use less cement.

Some low- emission concrete adds “fly ash” or slag to the mix, substances which don’t give off CO2 when heated.

However, they don’t have the binding

“If concrete were a country, it would be the third-worst polluter after the US and China” power of lime (calcium oxide). Traditional concrete mixtures have a certain amount of cement tailored to produce the right strength and other properties.

There’s a third way to reduce the overall emissions of concrete, and that is to use the cement to capture and store carbon dioxide.

“Carbon capture is the idea that if I capture CO2 that’s been emitted and find a way to store it somewhere out of harm’s way, that would be a good thing,” said Dunford. Efforts at carbon capture have been expensive and challenging, focusing on things like extracting carbon dioxide from polluting stacks, pumping it underground in oil wells, and trying to ensure it stays there.

As with most waste-disposal problems, it’s much better if it’s possible to find a route that turns the waste into something useful: “What if we found a way to uptake that CO2 that we see as a waste emission - and do something with it. If we make CO2 valuable, then we could find a way to create a market,

take it from its sources and use it.”

What CarbonCure offers is “mineralization” taking CO2, and injecting it into the wet concrete before it sets. “The CO2 wants to find that other part of its original molecule and reform calcium carbonate,” said Dunford. “It reforms calcium carbonate inside concrete when it is being made.

In practice, CarbonCure sprays CO2 into the concrete as it is mixed. “That requires some math and science to figure out the right amount of CO2 the concrete can handle,” said Dunford.

The CO2 comes out of pressurized tanks as “dry ice,” said Dunford. “It looks a lot like snow - it’s like what comes out of a fire extinguisher.”

The fact that it’s solid is important: “Once it is injected, it rapidly reconverts into calcium carbonate as a solid material.

There’s no bubbles of CO2 inside the concrete.” The process of stirring dry ice into concrete also means less than ten percent of the gas escapes into the atmosphere.

Operations Supplement 11

Operations Supplement Somewhat surprisingly, there’s very little downside. While most of the strength of concrete comes from calcium silicates in the cured cement, the extra calcium carbonate increases the strength by ten percent.

That means that concrete makers can reduce the cement content, said Dunford.

“Most of the time we want stronger concrete, and if you cut cement out of concrete, you see the strength go down,” he explained.

“You can have concrete of same strength as you always had, but CO2 is part of the reason you achieve that strength.”

This magnifies the total savings: “You’re looking at about 25lb of CO2 reduction per cubic yard,” said Dunford. That’s coming from one pound of CO2 going into the mix, and 20lb to 40lb reduced by limiting the amount of cement going in. You can leverage this small amount of CO2 to get a bigger amount of reduction.”

Of course, it’s important to keep in context that this only offsets five percent of the total global warming potential of the context, because most of the CO2 emitted still goes up the smokestack, and it still takes a lot of energy to make the clinker in the first place.

Still, it can be combined with other efforts such as using fly ash and shifting fuels.

And it is a practical contribution. “CO2 mineralized concrete isn’t just a pipe dream,” Dunford said. “It’s not something that’s a lab experiment or sitting on some masters student’s desk. It’s something that is happening in real time.”

And in the context of a building, it

stacks up well. For instance, Dunford cites a 350,000 square foot office building on a one-acre site, which used CO2 mineralized concrete: “It saved 680 tonnes of CO2 equivalent. That’s the equivalent of 800 years of a forest growing on that site.”

So far, the technique is only used in a small proportion of the world’s concrete.

More than five million cubic yards of CO2mineralized concrete have been produced, which sounds like good progress, but it is less than one-thousandth of the concrete the world uses in a year. Each year, more than five billion cubic yards (or nearly 10 billion tonnes) are poured.

The technique does require CO2 to arrive in pressurized cylinders, but that is how the world’s CO2 industry operates - and it is already gearing up to collect and use waste CO2 from industries such as ammoniamaking and fossil fuel generation.

“CO2 comes from industrial suppliers,” said Dunford. “They scrub CO2 out of the stacks at oil and gas refineries or ethanol facilities or ammonia manufacturers, and they take that CO2 capture it and compress it

12 DCD Supplement • datacenterdynamics.com

"Most of the time we want stronger concrete, and if you cut cement out of concrete, you see the strength go down. You can have concrete of same strength as you always had, but CO2 is part of the reason you achieve that strength.” and distribute it around.”

Dunford points out that if CO2mineralized concrete provides a bigger market for that recovered CO2, it will stimulate a bigger effort to capture it. “As demand for CO2 rises in the concrete sector, you might see a number of things happen.

You might see more installations, and greater capture.”

And that’s where the tech industry comes in. Companies such as Infosys and LinkedIn have used CO2-mineralized concrete in their headquarters in Indianapolis and San Francisco. “Private owners who care about this are jumping the gun,” said Dunford.

And data centers could be a good way to seed a steady demand. They are built to a price, but there’s a high profit margin in the industry, and a general awareness of environmental issues.

That’s why some data center companies are happy to pay the premium for CO2 mineralized concrete. In May 2020, Compass Datacenters made the change: "We estimate using CarbonCure will reduce our CO2 footprint by an average of 1,800 tons per campus," Compass CIO Nancy Novak said at the time. "That's the equivalent CO2 sequestered by 2,100 acres of forest or driving a car four million miles" Amazon has promised to use CO2 mineralized concrete in its forthcoming HQ2 in Virginia - and gone further by investing in the company. A new investment round in September 2020 was led by Amazon’s Climate Pledge Fund and Breakthrough Energy Ventures, climate tech fund backed by billionaires including Bill Gates Jeff Bezos, Marc Benioff, Reid Hoffman, and Jack Ma. Microsoft has also invested.

In future, some governments may start to regulate to encourage the use of the material (there are already signs that New York might go that way, says Dunford). Those efforts will be much more likely to succeed if there’s already awareness of the possibilities.

Operations Supplement

M&E security: The killer in your chiller The conversation about cyber security of electrical and mechanical equipment is progressing slowly – but it is progressing

D

ata centers often flaunt their physical security measures, erecting tall fences, blocking entrances with man-traps, and running blanket CCTV coverage to impress the occasional visitor. Servers inside are protected with firewalls and an assortment of software tools, running under the watchful eye of the security operations center.

But there is one aspect of data center security that is rarely talked about, and yet presents a growing problem: the gaping security holes in the mechanical and electrical kit that powers and cools the IT equipment.

The fact that operational technology (OT) systems like chillers and diesel gensets are vulnerable to attacks over network connections is no secret - DCD has reported on the subject repeatedly throughout the years. It ties into the broader debate about the security of industrial control systems (ICS) the same types of attacks that could be used against data centers could also disable power grids, hospitals, or manufacturing facilities.

“ICS and SCADA infrastructures are increasingly becoming vulnerable to attack, which can have disastrous consequences for mission-critical infrastructures,” Massimiliano Falcinelli, head of Security Infrastructure at the International Atomic Energy Agency, warned us back in 2016.

At first glance, it should be simple enough to fix – so how come the problem is still there?

Anything you put on a network instantly becomes a target, whether that’s a corporate laptop, a baby monitor, or an air handler.

For a simple illustration of this principle, see Shodan – the much-beloved search engine that can track down virtually any Internet-connected device, from sex toys to

power plants. The ability to browse random CCTV footage has long been one of its main attractions.

The project was started in 2009 by bioinformatics graduate John Matherly as a way to track the popularity of software tools used on the Internet. The fact people are putting millions of unsecured devices online was discovered almost by accident.

“I wish I could say that I had the foresight that refrigerators would get connected to the Internet, but I didn’t,” Matherly admitted in a recent interview with The Daily Swig.

“And I got lucky that people in the security community discovered Shodan.”

Shodan is often used to prove just how many industrial facilities fail at the most basic aspects of security. ‘Industrial control systems’ is a perennially featured category, along with misconfigured databases that are exposed for the whole world to see.

Once a networked device is discovered, a potential attacker can start researching specific vulnerabilities and exploits to take control, looking to deploy something like the Triton malware, that can trick systems into initiating emergency shutdown.

Gaining access can sometimes be as easy as entering the default password set by the manufacturer. Some of the Internetaccessible ICS used in industrial settings don’t require any authentication at all having been installed before the Internet went mainstream.

Today, pretty much every piece of data center equipment is networked: this includes CRACs, PDUs, and UPS systems, and the technical space is only going to get more connected.

For example, much-hyped predictive maintenance, in which a machine learning system monitors the state of the equipment and notifies the owner when individual parts

Max Smolaks Contributor

appear to be at the end of their lifespan, relies on a tonne of data being exchanged between the piece of equipment and the monitoring system. At the same time, robots are entering the data center, allowing a potential attacker to wreak havoc on a much more granular scale.

Luckily, the list of reasons to physically disable a data center is really short, since conventional cyber criminals are likely to focus their attention on more profitable endeavors. Possible motives include corporate sabotage, hacktivism, and terrorism - and in the current political climate, it is the last option that gets the gears moving.

In the US, the efforts to close holes in ICS security were spearheaded by the Industrial Control Systems Cyber Emergency Response Team (ICS-CERT). In 2018, the organization became part of a brand new federal agency, the Cybersecurity & Infrastructure Security Agency (CISA), explicitly tasked with protecting the US from “terrorist threats” and armed with a sizeable annual budget of $3 billion in 2020.

Its responsibilities include developing “a comprehensive national plan for securing the key resources and critical infrastructure of the United States, including power production, generation, and distribution systems, information technology and telecommunications systems (including satellites), electronic financial and property record storage and transmission systems, emergency communications systems, and the physical and technological assets that support those systems.”

CISA continues to issue alerts and advisories on ICS attacks and vulnerabilities, but is temporarily in a state of disarray, following the dismissal of its first director, Christopher Krebs, via tweet - ostensibly over suggesting that there was no evidence of any

Operations Supplement 13

Operations Supplement issues with the voting systems used in the recent elections.

In the European Union, a similar role to CISA is fulfilled by the EU Agency for cybersecurity (ENISA) and a network of dedicated Cyber Security Incident Response Teams (CSIRTs) in every member state, established in line with the requirements of the ‘Directive on security of network and information systems’ (NIS Directive) adopted in 2016.

Among other things, the directive asked member states to draw up lists of ‘operators of essential services’ that would be required to pay special attention to cyber security measures, or face penalties – including organizations in energy, transport, banking and finance, healthcare, drinking water distribution, and of special interest to this publication – digital infrastructure. Even though data centers specifically did not fall under the scope of the directive, five EU countries labeled them ‘essential’ anyway.

While Western governments were busy creating new specialist agencies, the security industry responded to the warnings of an impending IoT-pocalypse with a new generation of software tools aimed squarely at ICS systems. Many familiar names went after the slice of the ICS pie, including Check Point, Kaspersky, Tripwire, and Tenable.

Schneider Electric partnered with Nozomi Networks for industrial security. Emerson developed a proprietary Power and Water Cybersecurity Suite.

As the final piece of the puzzle, there are now plenty of opportunities to update professional skills to face ICS threats in the data center, thanks to training courses and certifications that simply didn’t exist five years ago, from organizations like the SANS Institute and even DatacenterDynamics’ own DCPro training division.

After nearly a decade of talking about the problem of OT equipment security, it remains in the twilight zone. While there’s broad consensus that attacks against critical infrastructure, including data centers, are possible and would have dire consequences, there is a lack of notable examples of carnage brought by weaknesses in ICS – beyond instances like Stuxnet and alleged attacks against Ukrainian power grids.

This means it gets treated as a problem for another day - a bit like the subject of the safety of Soviet nuclear reactors before the Chernobyl disaster. After all, there are other, less complicated ways to disrupt data centers, like Distributed Denial of Service (DDoS) attacks that simply clog up the network with bogus traffic.

The good news is data center operators don’t have to wait for the equivalent of an open-air reactor core fire to bolster their defenses against these new threats.

The techniques developed to secure manufacturing equipment can be applied in data centers, and even the basic measures can go a long way.

The National Institute of Standards and Technology recommends using firewalls or unidirectional gateways to stop network traffic from passing directly between the corporate and ICS networks, applying security patches as soon as they are tested in field conditions, blocking all unused ports, assigning user privileges only to people who are authorized to use the systems, and having a robust incident response plan in place in case the equipment is actually disabled.

For a more thorough understanding of vulnerabilities, an audit of mechanical,

14 DCD Supplement • datacenterdynamics.com

electrical and plumbing systems might be in order – and several cyber security firms now offer a professional assessment as part of their ICS security services.

Cyber security of OT equipment in a data center is still a relatively recent field, and its many problems are not going to be solved overnight – but the pressure to secure more visible parts of national critical infrastructure in highly uncertain times will undoubtedly have a knock-on effect on how server farms handle their chillers and PDUs.

Thermal attacks in multi-tenant data centers Although not a direct attack on the mechanical and electrical systems of the data center, a thermal attack, described in a 2018 paper published in Performance Evaluation Review, is a fascinating tactic that seeks to overload the cooling system by injecting additional thermal loads into a rack of servers.

Colocation data center operators normally monitor each tenant’s server power usage on a per-rack or even per-server basis, thus ensuring that each tenant’s power stays below its subscribed power capacity.

However, they cannot measure power consumption behind-themeter – which could be bolstered by something like a distributed UPS system that conceals a small battery in every server.

Batteries hidden in servers could be used by a malicious actor as a supplemental power source to run compute-intensive workloads, considerably increasing the heat produced and eventually triggering a shutdown, with the attack remaining all but invisible to power monitoring software.

The authors suggest that this might be the first ever instance where data center batteries have been leveraged for an adversarial purpose.

The same team built a system that ‘listens’ to data center noise to detect periods of high server utilization in order to maximize the damage from any potential attack on the power supply.

“ICS and SCADA infrastructures are increasingly becoming vulnerable to attack, which can have disastrous consequences for mission-critical infrastructures.”

Operations Supplement 15

Save your cloud from drowning You can never stop the rain, but you can harden your data center to avoid water ingress. Use Roxtec cable and pipe seals to prevent flooding, protect critical infrastructure and ensure business continuity. #nomoredowntime roxtec.com/datacenters