The Linux xz fiasco

[igor_kavinski]

What we know about the xz Utils backdoor that almost infected the world

Malicious updates made to a ubiquitous tool were a few weeks away from going mainstream.

arstechnica.com

Everything I know about the XZ backdoor

Please note: This is being updated in real-time. The intent is to make sense of lots of simultaneous discoveries

boehs.org

Just frickin' unfathomable. How do serious developers get into a situation like that? It's like there is no proper protocol for verifying what a new version of a software does and if it doesn't introduce any undesirable side effects or worse, deliberate security holes to deliver future payloads! The entire opensource thing seems to be based on implicit trust. Everyone saw those malicious individuals (Jia and Jigar) as nothing out of the ordinary. Jigar bitching about some stable software not getting updates and the maintainer just agreeing with him? Who the hell is a stranger to tell a hobbyist maintainer of a widely used piece of software that there aren't enough updates? Collins should've told Jigar to F the hell off! The whole thing just makes me furious beyond words.

 

[igor_kavinski]

This: https://infosec.exchange/@fr0gger/112189232773640259

https://media.infosec.exchange/infosec.exchange/media_attachments/files/112/189/232/249/630/121/original/b5157017fbcebb8d.png

Just...diabolical

 

[igor_kavinski]

Always remember, people:

The social engineering aspect of this was mentioned on a mastodon thread, which I think every project contributor should take to heart:

RT Carol (Nichols || Goulding) ꙮ @carol@crabby.fyi
the lesson I'm choosing to take from xz, as an oss maintainer, is that anyone trying to pressure or guilt me into doing something should immediately be told no, for security reasons
RT mybarkingdogs @mybarkingdogs@freeradical.zone
@ carol This is literally a good lesson for EVERYONE in anything, not even just software.
Giving into pressure/guilt is DANGEROUS
In personal relationships, it's one of the worst mistakes: it tells an abuser/manipulator you're a target.
In anything financial, it's often a baited hook for a scam
In politics it gets you pulled into anything from outright far-right fascist bullshit like qanon to "left" (but not really left, obviously!) groups that are state-sponsored ops or personality cults

 

[igor_kavinski]

This person/group had the patience of a monk: https://research.swtch.com/xz-timeline

 

[manly]

I wouldn't say open source is built on implicit trust. This was clearly a long con, and by appearances it was weeks away from succeeding until a weekend warrior saved us. I don't know if it was dumb luck, but the attackers somehow figured out the project maintainer didn't have enough time to be maintainer. I suspect most OSS projects vet their contributors a little better than this, or at least they will now have to.

Although not the same thing, this is reminiscent of the OpenSSL Heartbleed vulnerability a decade ago, when just one full-time developer was in charge of complex software that underpins the entire Internet. Trillion dollar companies depend on a lot of FOSS code, and often times they reap much greater benefits than they are contributing monetarily.

Yearly donations to the OpenSSL project were about US$2,000.

Heartbleed - Wikipedia

en.wikipedia.org

 

[WelshBloke]

Just frickin' unfathomable. How do serious developers get into a situation like that?

Because it's one guy who has been responsible for maintaining liblzma and has been the subject of a combination social engineering and technical attack by probable nation state actors?

Also this isn't a Linux/open source thing. Open SSH depends on this and that's in everything!

 

[WelshBloke]

The comments on JiaT75s git hub contributions are fairly entertaining now if anyone is interested!

 

[igor_kavinski]

The scariest thing is how Linux tools have such cryptic, undefined behaviors that even experienced programmers couldn't see from a cursory glance at the code that something suspicious was being done. They need to develop tools to "decipher" C/C++ code and present it graphically so anyone is able to understand what the code is actually doing. Without some tool like this to aid developers in their code reviews, malicious code will keep getting overlooked. The nuances of C are insane and it really is a dangerous language to help you shoot your foot off!

 

[mikeymikec]

"reviewing the code is difficult with open source, it's so complicated"

... so closed source ftw? Disclosure of security breaches if / when the targeted company feels like it?

Another example of closed source thinking is Apple processors' 'GoFetch' vulnerability which seems to be regarded in multiple quarters as unfixable (or at least unfixable without a significant performance hit).

How do you think this (OP) scenario would have played out in a closed source commercial environment? Plenty of oversight and code review of security sensitive code with committed teams of non-overworked, well-paid and competent employees, given all the time they need because security and brand trust is more important than revenue?

Personally I'd bet that for every story like this involving OSS, there's a hundred closed-source similar stories that have never been told.

 

[igor_kavinski]

Thoughts from a former googler/facebooker: https://rachelbythebay.com/w/2024/04/02/autoconf/

 

[Red Squirrel]

Open source is great in general, the issue is the mentality of writing programs/libraries that are overly complicated, having a manual that is not really all that great, then telling everyone to RTFM, instead of writing the program in such a way that it is self explaining and user friendly. I never have to RTFM when using Windows programs. Everything is just self explanatory. There is also a mentality against "reinventing the wheel" and instead creating layers and layers of dependencies.

This mentality creates tons of layers of complexity when things rely on each other, and you end up with messes like this where nobody really even bothers to review code anymore.

 

[WelshBloke]

Open source is great in general, the issue is the mentality of writing programs/libraries that are overly complicated, having a manual that is not really all that great, then telling everyone to RTFM, instead of writing the program in such a way that it is self explaining and user friendly. I never have to RTFM when using Windows programs. Everything is just self explanatory. There is also a mentality against "reinventing the wheel" and instead creating layers and layers of dependencies.

This mentality creates tons of layers of complexity when things rely on each other, and you end up with messes like this where nobody really even bothers to review code anymore.

Sooooo..... Are you saying that Microsoft programs are not over complicated, have good documention and doesnt have layers of dependancy!!???

 

[Indus]

Because it's one guy who has been responsible for maintaining liblzma and has been the subject of a combination social engineering and technical attack by probable nation state actors?

Also this isn't a Linux/open source thing. Open SSH depends on this and that's in everything!

Russian ?

 

[lantis3]

Sooooo..... Are you saying that Microsoft programs are not over complicated, have good documention and doesnt have layers of dependancy!!???

At least Windows source code is controlled by one company,

yet Linux code is not. https://en.wikipedia.org/wiki/List_of_Linux_distributions

 

[Red Squirrel]

Closed source proprietary code does have some advantages. As much as I don't like Apple, I think they have things dialed in the best, because they control the hardware too. I can't fault them for how they run things. As a user, I still don't like it though. MS has the challenge of trying to support thousands of different pieces of hardware. Then open source community has that challenge plus managing code from thousands of different people. I think all 3 ways of doing things have their advantages and disadvantages.

 

[manly]

Sooooo..... Are you saying that Microsoft programs are not over complicated, have good documention and doesnt have layers of dependancy!!???

Clearly he doesn't know WTF he's talking about. He's written some Linux scripts and fancies himself an expert.
Not to say it's always the case, but the essential Unix philosophy is KISS/small software components that can easily interact on the command-line.

At least Windows source code is controlled by one company,

yet Linux code is not. https://en.wikipedia.org/wiki/List_of_Linux_distributions

Is being controlled by one company a positive or negative? Perhaps neither, quite frankly. MS spends a lot of money on code quality and security. They also have certain programs where they share source code with certain outside partners. One could argue this creates a situation where outsiders could exploit vulnerabilities, but you don't have the transparency of OSS to discover these problems out in the open. All modern non-trivial software is incredibly complex. Microsoft and Apple have to patch bugs on an ongoing basis and nobody knows exactly how many zero-days are in the wild before they are fixed.

W.r.t. to software supply chain attacks, the world has been hit by a few in the past number of years. IIRC the rest exploited commercial software vendors and unlike this xz malware, they were not caught in time.

(IIRC the Stuxnet package that the U.S./Israel used to attack Iran's Natanz exploited some 7 zero-days in Windows.)

 

[WelshBloke]

Russian ?

It's a bit murky. It looks like a Chinese injection but it also looks like it's meant to look like a Chinese injection.
There's a few anomalies with IP addresses and names that suggests it's someone in Eastern Europe (ie Russia) that's trying to make it look like it's Chinese.
There was also VERY good opsec on this and that points at Russia as well, they are very good at this.

 

[Red Squirrel]

Pretty much sums it up lol.

 

[WelshBloke]

Pretty much sums it up lol.

View attachment 96487

If this had gotten through it would have been exactly that!

 

[mv2devnull]

https://www.theverge.com/2024/4/2/24119342/xz-utils-linux-backdoor-attempt contains a quote:
“The xz fiasco has shown how a dependence on unpaid volunteers can cause major problems. Trillion dollar corporations expect free and urgent support from volunteers.”

That is, the "closed source proprietary code" can and does depend on open source snippets like xz/liblzma, so MS products can depend on same bits as some versions of openssh do. We, users, have no way to see that directly, and the "benefits of proprietary" do not ensure any better vetted result.