The saga began earlier this year, when Freund was flying back from a visit to his parents in Germany. While reviewing a log of automated tests, he noticed a few error messages he didn’t recognise. He was jet-lagged, and the messages didn’t seem urgent, so he filed them away in his memory.
But a few weeks later, while running some more tests at home, he noticed that an application called SSH, which is used to log into computers remotely, was using more processing power than normal. He traced the issue to a set of data compression tools called xz Utils, and wondered if it was related to the earlier errors he’d seen.
“This could have been the most widespread and effective backdoor ever planted in any software product,”
Alex Stamos, the chief trust officer at SentinelOne, a cybersecurity research firm
(Don’t worry if these names are Greek to you. All you really need to know is that these are all small pieces of the Linux operating system, which is probably the most important piece of open-source software in the world. The vast majority of the world’s servers — including those used by banks, hospitals, governments and Fortune 500 companies — run on Linux, which makes its security a matter of global importance.)
Like other popular open-source software, Linux gets updated all the time, and most bugs are the result of innocent mistakes. But when Freund looked closely at the source code for xz Utils, he saw clues that it had been intentionally tampered with.
In particular, he found that someone had planted malicious code in the latest versions of xz Utils. The code, known as a backdoor, would allow its creator to hijack a user’s SSH connection and secretly run their own code on that user’s machine.
Loading
At first, Freund doubted his own findings. Had he really discovered a backdoor in one of the world’s most heavily scrutinised open-source programs?
“It felt surreal,” he said. “There were moments where I was like, I must have just had a bad night of sleep and had some fever dreams.”
But his digging kept turning up new evidence, and last week, Freund sent his findings to a group of open-source software developers. The news set the tech world on fire. Within hours, a fix was developed and some researchers were crediting him with preventing a potentially historic cyberattack.
“This could have been the most widespread and effective backdoor ever planted in any software product,” said Alex Stamos, the chief trust officer at SentinelOne, a cybersecurity research firm.
If it had gone undetected, Stamos said, the backdoor would have “given its creators a master key to any of the hundreds of millions of computers around the world that run SSH.” That key could have allowed them to steal private information, plant crippling malware, or cause major disruptions to infrastructure — all without being caught.
Microsoft CEO Satya Nadella praised Freund’s “curiosity and craftsmanship.”Credit: AP
Nobody knows who planted the backdoor. But the plot appears to have been so elaborate that some researchers believe only a nation with formidable hacking chops, such as Russia or China, could have attempted it.
According to some researchers who have gone back and looked at the evidence, the attacker appears to have used a pseudonym, “Jia Tan,” to suggest changes to xz Utils as far back as 2022. (Many open-source software projects are governed via hierarchy; developers suggest changes to a program’s code, then more experienced developers known as “maintainers” have to review and approve the changes.)
The attacker, using the Jia Tan name, appears to have spent several years slowly gaining the trust of other xz Utils developers and getting more control over the project, eventually becoming a maintainer, and finally inserting the code with the hidden backdoor earlier this year. (The new, compromised version of the code had been released, but was not yet in widespread use.)
Loading
Freund declined to guess who might have been behind the attack. But he said that whoever it was had been sophisticated enough to try to cover their tracks, including by adding code that made the backdoor harder to spot.
“It was very mysterious,” he said. “They clearly spent a lot of effort trying to hide what they were doing.”
Since his findings became public, Freund said, he had been helping the teams who are trying to reverse-engineer the attack and identify the culprit. But he’s been too busy to rest on his laurels. The next version of PostgreSQL, the database software he works on, is coming out later this year, and he’s trying to get some last-minute changes in before the deadline.