I’m currently in San Diego to attend my cousin’s wedding. (It’s going to be held at the Wild Animal Park, so yeah, she’s pretty cool.)
On the plane ride here I was scribbling down some ideas on how to explain Higgs physics (more on that later) when the guy next to me asks, “Are you from Fermi?” Unfortunately, I didn’t get his name, but he was a software engineer who wrote some of the user interface for the Tevatron control center. Specifically, I think he wrote the GUI for the anti-proton injector, but since I’m a theorist and they don’t let us near that stuff, I’m not entirely clear which part that is.
Anyway, after some talk about the LHC, and the consternation about the recent magnet failure and helium leak, he told me some entertaining stories about troubles at the Tevatron.
For example, in 1997 they installed a brand new low-conductivity-water cooling system. To test it, however, they piped in well water, and once the test was complete, they let it stand. Unfortunately, this let those pipes become ideal breeding grounds for metal-eating bacteria. So, even though the tests were successful, when they tried to turn the coolant system back on six months later more that 200 leaks suddenly appeared. The only information on this I could find was on page 11 of this PDF, which was written at the time, and doesn’t say how long the damage took to repair. Obviously it was repaired and the machine has functioned beautifully since then — aside, that is, from the other unexpected problems that cropped up and were repaired.
What this means is not that we should laugh at the Fermilab engineers, although a titanic fight between a high-energy particle accelerator and metal-eating bacteria is pretty funny, and as I say in the title would make a successful, though by no means good, movie.
JULIA
They’re almost to the control center, we’re humanity’s last hope!
ROBERT
Quick! We can hit them with the particle beam!
JULIA
No, we can’t achieve superconductivity without the cooling system, that’s why they started there. They’re far more clever than we ever expected.
ROBERT
Damn you, metal-eating bacteria!!
$50 million on opening day, easy.
Uh, I got a little sidetracked, where was I? Oh yeah, making fun and pointing blame isn’t the point. The point is that these are gigantic projects where multitudes of things can go wrong. You don’t succeed in building them by avoiding all problems, you do it by putting into place mechanisms to avoid as many problems as possible, and then dealing efficiently with the rest problems as they arise.
Ken Bloom makes a similar point at the US LHC blog, in reference to the issues with the start up of the Tevatron Run II, when the beam intensity was far below the design goal. I’ve seen the occasional anguished post in the blogosphere wondering how the LHC designer could let this happen, so remembering some of the previous glitches at highly successful machines like the Tevatron might help to put it in persepctive. I’m sure there are other good stories about accelerator hiccups, post your favorite in the comments.
LEP had its share too – beer bottle in the beampipe (urban lmyth?), a shutdown to retrieve a lost robot in the ring (good movie material there too), a beam dump triggered by someone trying to unplug a printer to change an ink cartridge, and grabbing the wrong power cord, not to mention the famous Energy resolution – TGV train effect and the influence of terrestrial tides and Lac Leman water levels. None of these stopped the successes then either, but they do happen.
Having to spend 3.5 weeks warming up, fixing something and then spending another 3.5 weeks cooling down again seems to defeat efficiency. Should have paid the extra billion to have that really fancy a/c unit installed!
Are you making fun of “Blasphemy”? I read that a few months back and found it quite a hoot. No Higg’s was found in that novel, but the rabbit’s foot sure was a fundamental particle, wasn’t it?
Nick: This is the fancy a/c unit! It was a pretty severe failure, and I don’t want to say it’s inconsequential. The point us just that, when you’re building new complicated machines they will find creative ways to fail. Someone, I forget who, recently phrased it as: “each of these machines is it’s own prototype”; that sums it up pretty well.
David: I’ve never read “Blasphemy”, but I’ll have to check it out. I was just aiming for overblown disaster movies in general.