Computer room static blowing server power supplies

[T]

A raised floor server room with 6 servers, fiber optic patch panels, large
control system cabinets and large UPS system, has been recently been plagued
by computer power supply failures. The supplies fail when apparently many
tens of thousands of volts jump from inside the PS around the switching
regulators to the chassis of the supply taking out all the components. This
has happened to 8 supplies so far. A visible blue flash has been witnessed
several times by pewrsons in the room.

The servers so far affected are all sitting on the raised computer center
floor. What ususally happens seems to be when a person enters the 12x20ft
room or gets near a server a large snap is heard and the server is on the
backup redundant PS, if it has not already failed.

We have been adding grounding bonding from all computer case to the bldg
steel, raised floor structure, etc to try to stop this. Its still happening.
The computers are various brands and varoious ages, from 1 week old to 5
years old. The probelm began occuring 2 months ago. The data center was
built 12 years ago.

We are at a loss to figure out how the charge is building up on the inside
of the power supplies. I am thinking about the common power source, via the
power cords Hot, Neutral and Ground conductor.

The green wire should be bonded to the PC case. Apparently the PS regulator
board floats above chassis potential. Apparently a large potential differnce
is building. It jumps a 1 inch gap to the chassis. Burn marks from repeated
arc overs are evindent (the arc that makes it fail is not the 1st time it
occurs!)

The Hot and Nuetral..how could a charge be coming in on these conductors and
getting past the MOVs to build a potential on the boards.


Any ideas would be appreciated.
TJS

 

[GregS]

A raised floor server room with 6 servers, fiber optic patch panels, large
control system cabinets and large UPS system, has been recently been plagued
by computer power supply failures. The supplies fail when apparently many
tens of thousands of volts jump from inside the PS around the switching
regulators to the chassis of the supply taking out all the components. This
has happened to 8 supplies so far. A visible blue flash has been witnessed
several times by pewrsons in the room.

The servers so far affected are all sitting on the raised computer center
floor. What ususally happens seems to be when a person enters the 12x20ft
room or gets near a server a large snap is heard and the server is on the
backup redundant PS, if it has not already failed.

We have been adding grounding bonding from all computer case to the bldg
steel, raised floor structure, etc to try to stop this. Its still happening.
The computers are various brands and varoious ages, from 1 week old to 5
years old. The probelm began occuring 2 months ago. The data center was
built 12 years ago.

We are at a loss to figure out how the charge is building up on the inside
of the power supplies. I am thinking about the common power source, via the
power cords Hot, Neutral and Ground conductor.

The green wire should be bonded to the PC case. Apparently the PS regulator
board floats above chassis potential. Apparently a large potential differnce
is building. It jumps a 1 inch gap to the chassis. Burn marks from repeated
arc overs are evindent (the arc that makes it fail is not the 1st time it
occurs!)

The Hot and Nuetral..how could a charge be coming in on these conductors and
getting past the MOVs to build a potential on the boards.

It sounds like your UPS's should take care of everything.
I woulds have the UPS's checked out. I would have building electricians
check things out. I would have installed whole circuit transient protection
at the breaker box feeding the room.

greg

 

[Ban]

A raised floor server room with 6 servers, fiber optic patch panels,
large control system cabinets and large UPS system, has been recently
been plagued by computer power supply failures. The supplies fail
when apparently many tens of thousands of volts jump from inside the
PS around the switching regulators to the chassis of the supply
taking out all the components. This has happened to 8 supplies so
far. A visible blue flash has been witnessed several times by
pewrsons in the room.
The servers so far affected are all sitting on the raised computer
center floor. What ususally happens seems to be when a person enters
the 12x20ft room or gets near a server a large snap is heard and the
server is on the backup redundant PS, if it has not already failed.

We have been adding grounding bonding from all computer case to the
bldg steel, raised floor structure, etc to try to stop this. Its
still happening. The computers are various brands and varoious ages,
from 1 week old to 5 years old. The probelm began occuring 2 months
ago. The data center was built 12 years ago.

We are at a loss to figure out how the charge is building up on the
inside of the power supplies. I am thinking about the common power
source, via the power cords Hot, Neutral and Ground conductor.

The green wire should be bonded to the PC case. Apparently the PS
regulator board floats above chassis potential. Apparently a large
potential differnce is building. It jumps a 1 inch gap to the
chassis. Burn marks from repeated arc overs are evindent (the arc
that makes it fail is not the 1st time it occurs!)

The Hot and Nuetral..how could a charge be coming in on these
conductors and getting past the MOVs to build a potential on the
boards.

Any ideas would be appreciated.
TJS

Put a humidifier inside the room, the air is too dry and static electricity
builds up.

 

[T]

This was th 1st thing we fixed. The RH was 15%, we raised it to 45% (its
winter). The adjacent control rrom is around 15% .
TJS

 

[petrus bitbyter]

A raised floor server room with 6 servers, fiber optic patch panels, large
control system cabinets and large UPS system, has been recently been
plagued by computer power supply failures. The supplies fail when
apparently many tens of thousands of volts jump from inside the PS around
the switching regulators to the chassis of the supply taking out all the
components. This has happened to 8 supplies so far. A visible blue flash
has been witnessed several times by pewrsons in the room.

The servers so far affected are all sitting on the raised computer center
floor. What ususally happens seems to be when a person enters the 12x20ft
room or gets near a server a large snap is heard and the server is on the
backup redundant PS, if it has not already failed.

We have been adding grounding bonding from all computer case to the bldg
steel, raised floor structure, etc to try to stop this. Its still
happening. The computers are various brands and varoious ages, from 1 week
old to 5 years old. The probelm began occuring 2 months ago. The data
center was built 12 years ago.

We are at a loss to figure out how the charge is building up on the inside
of the power supplies. I am thinking about the common power source, via
the power cords Hot, Neutral and Ground conductor.

The green wire should be bonded to the PC case. Apparently the PS
regulator board floats above chassis potential. Apparently a large
potential differnce is building. It jumps a 1 inch gap to the chassis.
Burn marks from repeated arc overs are evindent (the arc that makes it
fail is not the 1st time it occurs!)

The Hot and Nuetral..how could a charge be coming in on these conductors
and getting past the MOVs to build a potential on the boards.


Any ideas would be appreciated.
TJS

Is ground still connected to good old earth? You'll have to check to be
sure. Suppose the metal of the floor and installation has been grounded
carefully but you'll have to check this too. An open ground connection may
be the cause of you problems. Especially an open connection that is closed
again by someone entering the room.

On all PCs and servers I know, the secondary of the power supply has been
connected to the enclosure which in turn is grounded. So I assume the spark
jumps from the primary of the power supply the the chassis. Looks like
neutral and hot are floating which means that the neutral has not been
connected to ground, at least not properly. (Don't know where neutral should
be grounded in your place.)

There are other possibillities. Did someone install a badly insulated neon
sign near you? Some other renovation maybe? Something else I can't even
imagine but not very likely. I'm pretty sure something is not grounded
properly anymore, but what?

petrus bitbyter

 

[[email protected]]

T wrote:
[...]

The servers so far affected are all sitting on the raised computer center
floor. What ususally happens seems to be when a person enters the 12x20ft
room or gets near a server a large snap is heard and the server is on the
backup redundant PS, if it has not already failed.
[...]
Any ideas would be appreciated.
TJS

A near identical situation was cured simply by spraying the carpet
tiles in the computer room and access corridor, with anti static
solution.
john

 

[T]

We have ordered some conductive matting for walking on. Waiting for it while
supplies are failing!

T wrote:
[...]

The servers so far affected are all sitting on the raised computer center
floor. What ususally happens seems to be when a person enters the 12x20ft
room or gets near a server a large snap is heard and the server is on the
backup redundant PS, if it has not already failed.
[...]
Any ideas would be appreciated.
TJS

A near identical situation was cured simply by spraying the carpet
tiles in the computer room and access corridor, with anti static
solution.
john

 

[T]

My technicians have been testing the grounds to the steel and they are
intact. We have a new fiber pull coming into the room just today via and
existing conduit. The failures have been occuring for almost 2 months but
much more frequently these last 2 weeks.

 

[Michael A. Terrell]

My technicians have been testing the grounds to the steel and they are
intact. We have a new fiber pull coming into the room just today via and
existing conduit. The failures have been occuring for almost 2 months but
much more frequently these last 2 weeks.

I would have the electrical system inspected. The power transformer
for the building may not be properly grounded. The ground rod or wiring
may be damaged, leaving the transformer floating above ground.


--
Service to my country? Been there, Done that, and I've got my DD214 to
prove it.
Member of DAV #85.

Michael A. Terrell
Central Florida

 

[TJS]

An update; We are investigating "zinc whiskers" phenomina as a possible
cause. The zinc filaments are produced from the plywood core floor panels
supported by the zinc electrocoted support structure. When floor panels are
lifted or disturbed the conductive dust can get into the power supply and
lead to shorts between the modern closely spaced SMT devices.
See here for a white paper http://www.dataclean.com/pdf/zincwhiskers3.pdf

The failures/arcs in our supplies have all arced from the 120vac chopper
regulator collector to the PS case. If the zinc whisker problem is our case
then the blue flash is just a 120/170p-p vac arc to ground, and not a static
discharge.

We are looking at replacing our floor tiles with the modern ultra low static
floor tiles.

I'm a little skeptical that this is the problem, but it is true that 2
months ago we pulled floor tiles to look for fiber conduits and that about
the time the problems began

TJS

 

[Bennett Price]

Is the air conditioning overhead supply or raised floor supply?

 

[Franc Zabkar]

An update; We are investigating "zinc whiskers" phenomina as a possible
cause. The zinc filaments are produced from the plywood core floor panels
supported by the zinc electrocoted support structure. When floor panels are
lifted or disturbed the conductive dust can get into the power supply and
lead to shorts between the modern closely spaced SMT devices.
See here for a white paper http://www.dataclean.com/pdf/zincwhiskers3.pdf

The failures/arcs in our supplies have all arced from the 120vac chopper
regulator collector to the PS case. If the zinc whisker problem is our case
then the blue flash is just a 120/170p-p vac arc to ground, and not a static
discharge.

We are looking at replacing our floor tiles with the modern ultra low static
floor tiles.

I'm a little skeptical that this is the problem, but it is true that 2
months ago we pulled floor tiles to look for fiber conduits and that about
the time the problems began

TJS

What are the chances of zinc whiskers causing *exactly* the same fault
in *8* PSUs?

- Franc Zabkar

 

[T]

I'm skeptical about zinc whiskers and the white paper does come from a
cleaning service company, enough said. But still it is a consideration

 

[T]

AC is from overhead

 

[Franc Zabkar]

I'm skeptical about zinc whiskers and the white paper does come from a
cleaning service company, enough said. But still it is a consideration

I wonder why the arc jumps a 1" gap between the chopper and the
chassis. Why not the much smaller (?) gap on the underside of the PCB?

- Franc Zabkar

 

[GregS]

What are the chances of zinc whiskers causing *exactly* the same fault
in *8* PSUs?

Identical or different?

 

[James Sweet]

A raised floor server room with 6 servers, fiber optic patch panels, large
control system cabinets and large UPS system, has been recently been plagued
by computer power supply failures. The supplies fail when apparently many
tens of thousands of volts jump from inside the PS around the switching
regulators to the chassis of the supply taking out all the components. This
has happened to 8 supplies so far. A visible blue flash has been witnessed
several times by pewrsons in the room.

The servers so far affected are all sitting on the raised computer center
floor. What ususally happens seems to be when a person enters the 12x20ft
room or gets near a server a large snap is heard and the server is on the
backup redundant PS, if it has not already failed.

We have been adding grounding bonding from all computer case to the bldg
steel, raised floor structure, etc to try to stop this. Its still happening.
The computers are various brands and varoious ages, from 1 week old to 5
years old. The probelm began occuring 2 months ago. The data center was
built 12 years ago.

We are at a loss to figure out how the charge is building up on the inside
of the power supplies. I am thinking about the common power source, via the
power cords Hot, Neutral and Ground conductor.

The green wire should be bonded to the PC case. Apparently the PS regulator
board floats above chassis potential. Apparently a large potential differnce
is building. It jumps a 1 inch gap to the chassis. Burn marks from repeated
arc overs are evindent (the arc that makes it fail is not the 1st time it
occurs!)

The Hot and Nuetral..how could a charge be coming in on these conductors and
getting past the MOVs to build a potential on the boards.


Any ideas would be appreciated.
TJS

I think static is highly unlikely as the cause here. My guess would be
overvoltage on the line, transients, or inadequate cooling.

 

[Rich The Newsgroup Wacko]

.

potential differnce is building. It jumps a 1 inch gap to the chassis.
Burn marks from repeated arc overs are evindent (the arc that makes it
fail is not the 1st time it occurs!)

Any ideas would be appreciated.

Call an exorcist. ;-)
--
Cheers!
Rich
------
"If you find for your verse there's no call,
And you can't afford paper at all,
For the true poet born,
However forlorn,
There is always the lavat'ry wall."

 

[Rich Grise]

I'm a little skeptical that this is the problem, but it is true that 2
months ago we pulled floor tiles to look for fiber conduits and that about
the time the problems began

Well, skeptical is good, but usually when troubleshooting, the first
question is, "What changed?"

This is a no-brainer. Start looking through your contract with the floor
installer, to see if you can have him eat the cost of replacing the killer
tile.

Good Luck!
Rich

 

[Rich Grise]

What are the chances of zinc whiskers causing *exactly* the same fault
in *8* PSUs?

Pretty good, considering that the tiles are the only thing that changed.

I probably wouldn't say, "zinc whiskers", but I'm pretty sure that, except
for galvanizing garbage cans, zinc is AWFUL! It gets all over everything.
I once worked at a place where they had to scrap a $100,000.00 (or so)
Ultra-High-Vacuum bell jar because someone had installed brass connectors -
the zinc outgassed, and contaminated the whole thing.

Maybe you could leach the zinc out of your tile by washing it down with
muriatic acid. ;-) [1]

I've also seen an installation where a bunch of boxes running off 277V
lighting power started arcing, and the only conductive thing in the room
was the dust from the desert setting, which could very well have been
conductive - it was only about 100 miles from Great Salt Lake. We fixed
it by putting furnace filters on top of all of the boxes, which the idiot
designer had put all of the intake air vents on the top of. =:-O

It was a pretty sweet TDY[2], though - $1000.00/week plus per diem plus
expenses plus airplane ticket plus car rental. ;-)

Good Luck!
Rich
[1] That's a joke, by the way. If you do use muriatic acid on zinc, be
sure that you're in a very very well-ventilated place, because it makes
hydrogen gas. But then, you could use the zinc chloride for sunscreen;
just mix it with yogurt. ;-)
[2] Temporary DutY