Looking for a UPS Design That Doesn't Overheat Batteries

[[email protected]]

Our company has had a long-standing problem where UPS batteries will at
various points in their lifetime suddenly overheat, sometimes
catastrophically to the point where the battery casing starts to melt and
you can actually smell the gases from the battery leaking. So far we
have been lucky to catch such thermal events with temperature sensors but
it has always been a goal of mine to better understand why this happens,
and to
find some UPS system where it can be avoided entirely. To date, we have
seen these problems with APC Symmetra tower, Symmetra rackmount, and
SmartUPS.

After working with an electrician, I have a theory about why this is
happening, and if correct, the theory suggests a different design for UPS
systems that would avoid the problem. I am hoping some manufacturer has
already implemented this idea and someone can refer me to their products.

This is a very complex problem and the best solutions are EXPENSIVE!

Well your industrial electrician should have first told you (electricity
101) that you need a PM( Preventative Maintenance) schedule on UPSes.
This would involved thaking the UPS offline monthly/quarterly and testing
the batteries and other portions of the UPS. The industrial size UPS all
come with offers from 3rd party companies to perform PM on you UPS gear.
If you buy cheap equipment, a quality 3rd party company (which will usually
also sell you an extended warranty as part of the PM) will tell you to buy
something industrial grade, like a 3-phase UPS that is interfaces to a
Gen_Set.


Also I always design UPS systems, no matter how large, to have a
1:1 isolation transformer upstream of the all power feedeing the UPS.
This eliminates most power quality problems from the local utility
or the ass_hole down the street that is wreaking havoc on the electrical
grid because he's using a 3phase welder and the idiot that run the
utility commissions (here in the US) or elsewhere to not have the common
sense to test/monitor and require such folk to put in equipment
to mitigate the effects of certain types of industrial equipment
on the electrical grid. (expensive equipment will allow you to monitor
the power quality (or lack thereof) from you local electrical utility feed.

On all of the UPS systems we use generic "brick" batteries are joined
together in a series, then the leads from the ends of these battery chains
are connected to the UPS. The problem is that batteries rarely fail all
together. If a 12V battery should be considered discharged and not
useful at around 10V, and you have two 12V batteries joined in series,
what happens when one of the batteries maintains a full charge at 12V but
the other battery in the series starts to lose its ability to hold charge
and slips
below some critical level? From the point of view of the UPS, it
doesn't see anything about the state of individual batteries. The UPS
only sees that the total voltage of the two 12V batteries in series has
fallen from 24V to 22V.

Maybe an electrical engineer can step in here and explain what is
happening,

UPS batteries and battery charging circuits and semiconductors leave much
to be desired. If you spend some time researching 'gel-cel' batteries and
other types of batteries used in UPSes, you find there are subtle
difference in the batteries. When these differences are combined with
with poorly designed charging circuits are poorly chosen semiconductors
in the charging circuitry, trade offs are made. These are compounded
by subtle differences in the electrical characteristics of different
batteries supplied from a variety of low_cost suppliers.

If you look at battery charging semi conductors, such as those
provide my Maxim, you can read the data sheets for some really
good/detailed discussions on a myriad of issues related to this
complex problem.

but my pure guess is that to maintain the same power output, an
increased
amount of current probably has to flow through the batteries. That
creates problems with heating for the "good" battery, which is still
measuring 12V. Now that 12V is receiving too much current, overcharges,
overheats, and at some point the casing of the battery starts to melt.
I haven't done enough experimentation to determine if it is the good
battery
or bad battery that is overheating. To be honest, in such situations I
have often seen evidence that both batteries start to melt. Perhaps this
is nothing more than one battery being in physical proximity to the
overheating battery and therefore gaining heat from its physical contact.
The only thing that is common to all cases is that one of the two
batteries has discharged and should have been replaced before the
overheating event took place.

Regardless of the actual mechanism for the overheating we are observing,
it seems to me that the obvious solution is to design UPS systems to
physically
connect to each 12V battery individually. Forget connecting multiple
batteries in series, at least don't do that at the battery itself. By
connecting to and monitoring individual batteries, now the UPS can see
when an individual battery falls below some critical voltage threshold and
put it
into a special recharge state (not put any load on it). If the battery
fails to recharge, the UPS can declare the battery defective and can
signal
the condition by an LED on the battery's compartment. If there is a
network attached monitoring system, the UPS can send an e-mail.

Aside from increasing safety and utility of the monitoring system, such a
design would allow much easier re-use of off-the-shelf batteries,
improving
ease-of-use in making battery changes and lowering cost. While I realize
that APC in particular has no desire to make anything regarding batteries
non proprietary, maybe some other vendor has a UPS design that puts a
direct monitoring circuit on each individual 12V brick battery, thus
avoiding the overheating problem I have described?

Any information on why this overheating takes place, how to avoid it, and
any referrals to third party UPS products that employ a more robust design
are appreciated.

Overheating is unfortunately a vestige of a variety of problems with this
complex issue. Search and read as much of this information is available via
googling the net.......

HTH,

James

 

[[email protected]]

Will wrote:

Does any UPS manufacturer make a unit that will do about 3000VA -
rackmount or tower configuration - that will treat the batteries better
and use less
aggressive charging in favor of long battery life? And of those units,
does anyone make a unit where the batteries are given enough space between
them that overheating in one battery stays localized to that battery?

On smaller UPS systems ( less that 10,000 VA) I'd suggest you make sure
the UPS vendor uses a 'ferro-resonant' transformer in the UPS. It is a big
heavy iron core that will filter most power quality feed problems
(particularly transients) out from the utility power feed, thus making the
UPS more durable and less transient electrical effects that deteriorate
the solid state components and the batteries in the UPS.

BEST use to make a 'Ferrups' but they sold out and you have to do
your research as this sort of UPS will be larger, much heavier
and definitely more expensive. Alternatively, you can get use cheap UPSes
and get your electrician to install a big 1:1 (one to one) isolation
transformer upstream of all of the upses in your facility.

If you get an industrial grade UPS as the vendor about 3rd party maintenance
PM costs and extended warranties which include PM or do the PM yourself
(if you are a qualified electrician)....

HTH,

James

 

[budgie]

Could you reformulate the above statements as a set of requirements I can
put into a requirements document for a new UPS? I'm guessing that the
requirements would be something similar to the following, but I am hoping
you will make these statements more precise:

- UPS should use a recharge / recovery scheme that maximizes battery
lifetime rather than minimizing recharge times, or should as a worst case
allow this to be a user configurable option.

- UPS should use an appropriate float voltage for the batteries it uses.

I avoid the word "should" in specifications. Unless you intend to scour
responses with a fine tooth comb and solicit clause-by-clause statements of
compliance, you are wasting your time. Even then, if the supplier states
"comply" to a "should", it doesn't mean he actually DOES what you preferred. I
"should" eat less fatty food, but I don't.

Focus on the recovery time, and maximum recharge current being less than (say)
60% of manufacturer's recommended max charge rate - the derating being for
thermal reasons given that the MR figure will usually be free air rated.

Focus on float voltage being whatever gives about 85-90% S.O.C. Any higher and
you WILL progressively cook the batteries.

- UPS should allow the use of deep cycle batteries.

?? Deep cycle vs SLA/VRLA. You need to decide on one battery type.

I'm not sure whether you are looking at the 3kVA size or not, so battery choice
and ventilation are unknowns here. SLA's are widely used simply because they
avoid ventilation issues.

I'm also not sure whether you are buying one or a hundred. Writing a spec for
one is a bit of overkill unless it is BIG (50kVA or more), otherwise just keep
the spec as purchasing guidelines to enable interrogation of wannabe suppliers.

- UPS should provide a way to use an appropriate topping charge every six
months with the battery.

Not only pointless, but probably deleterious in gelled electrolyte types. See
above re battery type.

If I take an older UPS gel-cell battery which is in series as 2 x 12V, and I
test each 12V and find that one is giving a good 12V reading when charged
and the other never goes above 10V, why can't I just replace the dead 10V
battery with a fresher 12V? How is replacing the dying battery promoting
overheating?

By the time one battery has died - unless we are talking of infant mortality -
it's mates have been subject to the same regime of use/abuse, and should be
replaced before they follow. You are creating a series string of unequals, which
is a recipe for failure. It's almost like changing tyres one at a time, except
with tyres you can at least see their condition. False economy.

 

[budgie]

Years ago, when wet cells were the only game, we ran a synchronous
generator as a motor driving a flywheel. When power dropped, the
contactor to the line did too, and a clutch engaged an internal
combustion engine that ran on illuminating gas. The system never failed,
but short dropouts -- they're more frequent than we had imagined -- were
painful. Switching back to line power required that the generator be
phase locked by hand before the contactor was re-energized. We couldn't
let the flywheel carry the load while the contactor remained closed for
the obvious reason.

Geez, Jerry, autosync has been around for ages.

In our configuration, the three gensets were (obviously) all initiated after the
outage had lasted 15 secs. They were 2*300kVA and the added one was 900kVA. We
used a PLC to set the rules: if the 300's were up and the 900 wasn't, they
would attempt to parallel (autosync) off-load and then connect to the gen bus.
If the 900 came up and the others weren't already on the bus (the usual case) it
would connect, and the others would then abandon their off-load parallelling
attempts and join individually.

It took a bit of fine tuning to handle the dynamics of different sized machines,
but it worked a treat. Worst case when all three fired was about 50 secs to
have them all parallelled on load.

I have seen the in-line prime-mover, alternator, motor combos. My main issue
with them was the frequency drop when the flywheel accelerated the prime mover
up from 0 to 1500 (1800 your side?) rpm.

The rotary that the big bank HO used was by "HH". It ran a controlled fluid
coupling, with the flywheel at ~3000rpm. Once motor drive failed, as the
flywheel decelerated the coupling was controlled so the alternator remained at
line frequency. But you only had a finite (read *short*) time to get the prime
mover(s) up. And really only one shot.

Our PM's were Detroit two-strokes. The 900kVA would be at speed within seven
secs of the start of cranking - presuming it fired of course. But a failure to
fire withing 30 secs would be the end of the bank (who also used a DD prime
mover on their genset).

 

[Will]

Focus on float voltage being whatever gives about 85-90% S.O.C. Any higher and
you WILL progressively cook the batteries

What is "S.O.C."?

?? Deep cycle vs SLA/VRLA. You need to decide on one battery type.

I guess deep cycle would be a better choice for battery life, but probably
requires a lot of additional cost and no one will support them in a smaller
UPS?

 

[Jasen Betts]

Our company has had a long-standing problem where UPS batteries will at
various points in their lifetime suddenly overheat, sometimes
catastrophically to the point where the battery casing starts to melt and
you can actually smell the gases from the battery leaking. So far we have
been lucky to catch such thermal events with temperature sensors but it has
always been a goal of mine to better understand why this happens, and to
find some UPS system where it can be avoided entirely. To date, we have
seen these problems with APC Symmetra tower, Symmetra rackmount, and
SmartUPS.

a shorted cell could do that if you've batteries in parallel.

Maybe an electrical engineer can step in here and explain what is happening,
but my pure guess is that to maintain the same power output, an increased
amount of current probably has to flow through the batteries. That
creates problems with heating for the "good" battery, which is still
measuring 12V. Now that 12V is receiving too much current, overcharges,
overheats, and at some point the casing of the battery starts to melt.

It'd lots of power for that to happen. many UPSs only charge at a low rate...

Regardless of the actual mechanism for the overheating we are observing, it
seems to me that the obvious solution is to design UPS systems to physically
connect to each 12V battery individually. Forget connecting multiple
batteries in series, at least don't do that at the battery itself. By
connecting to and monitoring individual batteries, now the UPS can see when
an individual battery falls below some critical voltage threshold and put it
into a special recharge state (not put any load on it).

If the battery
fails to recharge, the UPS can declare the battery defective and can signal
the condition by an LED on the battery's compartment. If there is a
network attached monitoring system, the UPS can send an e-mail.

If a battery fails it's time to replace all of them
either they failed early and it's possiblty a bad batch,
or you've missed the due date to replace them.

Bye.
Jasen

 

[budgie]

What is "S.O.C."?

Sorry, State of Charge

I guess deep cycle would be a better choice for battery life, but probably
requires a lot of additional cost and no one will support them in a smaller
UPS?

I'd venture you're correct.

 

[William P.N. Smith]

If I take an older UPS gel-cell battery which is in series as 2 x 12V, and I
test each 12V and find that one is giving a good 12V reading when charged
and the other never goes above 10V, why can't I just replace the dead 10V
battery with a fresher 12V? How is replacing the dying battery promoting
overheating?

Ah, we've discovered the problem with your existing UPSen.

First, replace the batteries on a regular schedule, before they fail.

Second, battery strings must be matched, so replace all the batteries
in a string at the same time.

Please note: You haven't specified your requirements very well. The
answers for hundreds of kilowatts of UPS for a stock exchange and a
few kilowatts for a local business are significantly different, and
you are getting answers all over the map.

 

[[email protected]]

... replace the batteries on a regular schedule, before they fail.

Sounds expensive, vs periodically testing and replacing any that don't
pass the test.

Second, battery strings must be matched, so replace all the batteries
in a string at the same time.

Sounds expensive. How well-matched must they be?

Nick

 

[William P.N. Smith]

Our company has had a long-standing problem where UPS batteries will at
various points in their lifetime suddenly overheat, sometimes
catastrophically to the point where the battery casing starts to melt and
you can actually smell the gases from the battery leaking.

I wonder if you don't have a bit of confusion between cause and effect
here. Old batteries can short a cell when the plates age (and swell)
sufficiently, generating quite a bit of heat when the stored energy in
that cell is released. This _can_ cause an avalanche effect in nearby
cells and batteries, but the primary cause is ignoring the PM schedule
on battery replacement...

 

[Jerry Avins]

budgie wrote:

...

Geez, Jerry, autosync has been around for ages.

So have I.

In our configuration, the three gensets were (obviously) all initiated after the
outage had lasted 15 secs. They were 2*300kVA and the added one was 900kVA. We
used a PLC to set the rules: if the 300's were up and the 900 wasn't, they
would attempt to parallel (autosync) off-load and then connect to the gen bus.
If the 900 came up and the others weren't already on the bus (the usual case) it
would connect, and the others would then abandon their off-load parallelling
attempts and join individually.

We were a lot smaller than that, about 12 KVA, kludged together from war
surplus equipment.

It took a bit of fine tuning to handle the dynamics of different sized machines,
but it worked a treat. Worst case when all three fired was about 50 secs to
have them all parallelled on load.

Great fun when you get it all working.

I have seen the in-line prime-mover, alternator, motor combos. My main issue
with them was the frequency drop when the flywheel accelerated the prime mover
up from 0 to 1500 (1800 your side?) rpm.

We have turbines at the sewage plant that run on Diesel fuel, with
gravity feed from 500-gallon tanks indoors and the rest of the storage
underground. They are set to wait 45 seconds before starting, and go on
line 15 seconds later. It takes another minute or so for frequency to
settle to spec, but not much cares. The pumps and aerators can wait a
bit. The local power plant uses similar turbines to run their peaking
generators, but they are on turning gear when idle. Ours can wait still.

The rotary that the big bank HO used was by "HH". It ran a controlled fluid
coupling, with the flywheel at ~3000rpm. Once motor drive failed, as the
flywheel decelerated the coupling was controlled so the alternator remained at
line frequency. But you only had a finite (read *short*) time to get the prime
mover(s) up. And really only one shot.

Our PM's were Detroit two-strokes. The 900kVA would be at speed within seven
secs of the start of cranking - presuming it fired of course. But a failure to
fire withing 30 secs would be the end of the bank (who also used a DD prime
mover on their genset).

Jerry

 

[repatch]

Sounds expensive, vs periodically testing and replacing any that don't
pass the test.

It is expensive, but you have to do it, since by replacing battery per
battery you will end up with a string of unmatched batteries in various
states of degredation. That's a perfect recipe for the symptoms you've
described.

The NUMBER ONE rule of "battery packs" is strings of batteries must be
replaced at the same time, no mixing and matching is allowed.

Sounds expensive. How well-matched must they be?

Same date code is the only thing I'd be happy with.

TTYL

 

[Jerry Avins]

Same date code is the only thing I'd be happy with.

And, of course, same maker and model number.

It's simple, really. Once one cell in a team dies of old age, it's time
to treat its buddies as honored veterans and put them out to pasture.
Used individually instead of hitched as a team, they may have a useful
contribution in a high-school lab or science club.

Jerry

 

[[email protected]]

It is expensive, but you have to do it, since by replacing battery per
battery you will end up with a string of unmatched batteries in various
states of degredation. That's a perfect recipe for the symptoms you've
described.

I haven't described any symptoms.

The NUMBER ONE rule of "battery packs" is strings of batteries must be
replaced at the same time, no mixing and matching is allowed.

According to "solar consultant" George Ghio?

Same date code is the only thing I'd be happy with.

Spoken like a hide-bound bureaucrat How about matching voltages within
some range or equivalent series resistances? Given a max charging current,
we could use these imbalances to predict the max temp rise.

Nick

 

[jk]

Our company has had a long-standing problem where UPS batteries will at
various points in their lifetime suddenly overheat, sometimes
catastrophically to the point where the battery casing starts to melt and
you can actually smell the gases from the battery leaking. So far we have
been lucky to catch such thermal events with temperature sensors but it has
always been a goal of mine to better understand why this happens, and to
find some UPS system where it can be avoided entirely. To date, we have
seen these problems with APC Symmetra tower, Symmetra rackmount, and
SmartUPS.

Maybe an electrical engineer can step in here and explain what is happening,
but my pure guess is that to maintain the same power output, an increased
amount of current probably has to flow through the batteries.

IN both charge and discharge, this is true

That
creates problems with heating for the "good" battery, which is still
measuring 12V.

THe same current flows through both.

Regardless of the actual mechanism for the overheating we are observing, it
seems to me that the obvious solution is to design UPS systems to physically
connect to each 12V battery individually. Forget connecting multiple
batteries in series, at least don't do that at the battery itself. By
connecting to and monitoring individual batteries, now the UPS can see when
an individual battery falls below some critical voltage threshold and put it
into a special recharge state (not put any load on it). If the battery
fails to recharge, the UPS can declare the battery defective and can signal
the condition by an LED on the battery's compartment. If there is a
network attached monitoring system, the UPS can send an e-mail.

What you want here is not that (Separate chargers for each battery
gets expensive) but a cell/ battery monitor system. Such as Cell
watch, or Alber.

"Real" ups systems use them all the time.
jk

 

[jk]

One idea that has been missed so far: If you are really serious about
reliability (most UPS users aren't) and don't like dealing with batteries,
consider installing a Rotary UPS. You'll find plenty of good info via a
Google search.

BZZZT, won't work. Rotary UPS systems still use batteries, and still
have all of the attendant problems. You are perhaps thinking of
Flywheel UPS systems. THose, while they work for the VERY short term,
give support in time periods measured in SECONDS, and require (on the
ones I have worked on) several minutes to HOURS to recharge the fly
wheel. They also have a low KW/KHW support per sq ft of consumed
space in my opinion.

I know it's "old school" and more expensive initial outlay, but Rotary UPS's
are used by the majority of the world's Stock Exchanges and major data
centres for all of the reasons you mentioned in your post - and on power
failure, they just work.

On all of the data centers I have worked on, only two had rotary UPS
systems, and only one had a flywheel system. Certainly not the
majority.

HTH,
Cameron

jk

 

[jk]

One idea that has been missed so far: If you are really serious about
reliability (most UPS users aren't)

Truer words were never typed. Amazing isn't it, how much money people
will spend to ALMOST have reliable power to "Mission critical"
systems.

jk

 

[Peter Bennett]

What is "S.O.C."?



I guess deep cycle would be a better choice for battery life, but probably
requires a lot of additional cost and no one will support them in a smaller
UPS?

Deep cycle batteries can be had as flooded cell, gel or AGM (gel and
AGM are also known as SLA/VRLA.

I'm not sure that a deep cycle battery is the best type for a UPS,
since a UPS will demand fairly high currents from the batteries when
supplying power - as I understand it, deep cycle batteries are
optimized for delivering low to moderate currents over a long period,
and to survive many charge/discharge cycles. I would expect most UPS
applications to have very few charge/discharge cycles (unless the
power service is _very_ poor).

 

[Peter Bennett]

Sounds expensive. How well-matched must they be?

Nick

Quite well - since two batteries permanently connected in series are
acting as a single battery, all cells will have the same "life
experience". Chances are that when one cell fails, most of others are
nearing end-of-life.

 

[Will]

I wonder if you don't have a bit of confusion between cause and effect
here. Old batteries can short a cell when the plates age (and swell)
sufficiently, generating quite a bit of heat when the stored energy in
that cell is released. This _can_ cause an avalanche effect in nearby
cells and batteries, but the primary cause is ignoring the PM schedule
on battery replacement...

When testing batteries, is there some key sign that might disclose imminent
failure of one of the 12V bricks? Would we for example see the voltage
decline in a non-linear way as it declines from 12V to 10V?