I’m currently in San Diego to attend my cousin’s wedding. (It’s going to be held at the Wild Animal Park, so yeah, she’s pretty cool.)
On the plane ride here I was scribbling down some ideas on how to explain Higgs physics (more on that later) when the guy next to me asks, “Are you from Fermi?” Unfortunately, I didn’t get his name, but he was a software engineer who wrote some of the user interface for the Tevatron control center. Specifically, I think he wrote the GUI for the anti-proton injector, but since I’m a theorist and they don’t let us near that stuff, I’m not entirely clear which part that is.
For example, in 1997 they installed a brand new low-conductivity-water cooling system. To test it, however, they piped in well water, and once the test was complete, they let it stand. Unfortunately, this let those pipes become ideal breeding grounds for metal-eating bacteria. So, even though the tests were successful, when they tried to turn the coolant system back on six months later more that 200 leaks suddenly appeared. The only information on this I could find was on page 11 of this PDF, which was written at the time, and doesn’t say how long the damage took to repair. Obviously it was repaired and the machine has functioned beautifully since then — aside, that is, from the other unexpected problems that cropped up and were repaired.
What this means is not that we should laugh at the Fermilab engineers, although a titanic fight between a high-energy particle accelerator and metal-eating bacteria is pretty funny, and as I say in the title would make a successful, though by no means good, movie.
They’re almost to the control center, we’re humanity’s last hope!
Quick! We can hit them with the particle beam!
No, we can’t achieve superconductivity without the cooling system, that’s why they started there. They’re far more clever than we ever expected.
Damn you, metal-eating bacteria!!
$50 million on opening day, easy.
Uh, I got a little sidetracked, where was I? Oh yeah, making fun and pointing blame isn’t the point. The point is that these are gigantic projects where multitudes of things can go wrong. You don’t succeed in building them by avoiding all problems, you do it by putting into place mechanisms to avoid as many problems as possible, and then dealing efficiently with the rest problems as they arise.
Ken Bloom makes a similar point at the US LHC blog, in reference to the issues with the start up of the Tevatron Run II, when the beam intensity was far below the design goal. I’ve seen the occasional anguished post in the blogosphere wondering how the LHC designer could let this happen, so remembering some of the previous glitches at highly successful machines like the Tevatron might help to put it in persepctive. I’m sure there are other good stories about accelerator hiccups, post your favorite in the comments.