Internet giants Google and Facebook have discovered they are experiencing computer chip failures that can corrupt data or make it difficult to unlock encrypted files. Facebook says hardware manufacturers must take notice of the problem, which has emerged due to the vast scale of computing resources the firms use.
The issue surfaced at Google when multiple teams of engineers reported problems with their computations, but the company’s usual diagnostic tools showed no problem. An investigation revealed that individual chips were responsible for repeated faults. In certain cases, researchers could prompt problems by changing a chip’s temperature.
These “silent errors” are caused by bits on the chips flipping from 0 to 1 or vice versa. Cosmic radiation can cause bits to flip, so computers destined for space have to be specially designed to prevent this. The errors spotted by Google and Facebook manifest in a similarly sudden way, but are instead due to ever-shrinking chips exhibiting unpredictable behaviours.
Advertisement
Google says the issue is down to manufacturers squeezing an ever-higher density of components onto chips in a quest for higher performance, which leaves smaller margins for error. It says that problems can manifest months or years after installation as chips essentially wear out and develop issues.
An industry rule-of-thumb known as Moore’s law states that the number of transistors in a circuit will double roughly every two years, but the rate of miniaturisation has slowed as engineers reach circuit densities beyond which electrons cannot be reliably controlled.
These faults can lead to corruption of small amounts of data, but if that data is itself key to unlocking or finding other data, then the effects can quickly magnify. In one case, Google found that files encrypted by one machine could only be decrypted by that machine, even when the correct key was used on other computers.
While the flaws may be exceedingly rare for individual computer chips, companies like Google and Facebook rely on hundreds of thousands of servers, each containing numerous chips and devices that can individually contain a billion transistors. Google declined to speak to 51¶ŻÂţ about the issue, but in a it pointed to finding hundreds of affected chips.
The company now tests chips throughout their lifespan to identify suspect chips as they age, and says it has devoted decades of cumulative engineer time to solving the problem.
Sriram Sankar at Facebook, , says that, in his experience, the issues could be found across all chip manufacturers. “This problem of silent errors is going to happen across silicon devices. Whether CPUs or custom chips, you’re gonna have silent errors.”
Sankar says the problem can occur during design, manufacturing or after months or years of operation. “The reason we call them silent errors is because there is no error correction or error detection of these errors,” he says. “That’s why this is an important area that the industry should focus on. I think all of the industry should take notice.”
One solution is redundancy – critical systems such as aircraft computers might have three machines all running the same code and constantly comparing the output to check accuracy – but the cost of doing this across Google or Facebook’s vast operations would be prohibitive.
Sankar and his colleagues propose several cheaper fixes, including running automated tests on chips during maintenance or using less intensive testing applications that can run alongside a chip’s everyday tasks. Facebook has started doing this on the company’s servers over recent months, says Sankar.
Once these error-detection systems have found a problematic chip, engineers can simply turn it off and leave it in place, which is the most efficient solution, or replace it with a new chip. Sometimes, a faulty chip is left running in a server, but is virtually handed over to a team that investigates the nature of silent errors. This zoo of troublesome chips informs Facebook’s research and helps to develop even better tools for detecting them, says Sankar.
at the University of Sheffield, UK, says chips are now so complex that he would be surprised if any of them are designed and manufactured perfectly.
“Processors do very strange things,” he says. “They’re coming up with cleverer and cleverer ways of doing things, having many, many cores, and you get some really strange behaviours.”
