Skip to main content
Topic solved
This topic has been marked as solved and requires no further attention.
Topic: New package: linux-firmware-xz (Read 1103 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

New package: linux-firmware-xz

Recently, archlinux compressed firmwares via zstd. However, old kernels such as 5.15-lts do not support the compression. Someone posted xz-compressed firmware at https://aur.archlinux.org/packages/linux-firmware-xz

As the universe repo ships linux-lts515, could the repo ships linux-firmware-xz altogether?

Some discussion at the linux-lts515 page:
https://aur.archlinux.org/packages/linux-lts515

Re: New package: linux-firmware-xz

Reply #1
xz has great compression but it's only single threaded in it's standard incarnation, and the multi-threaded xz alternative versions have never been widely adopted, which is equally true of most of the older compression methods such as bz2. zstd with the right options can offer both decent compression and speed on modern multi-core processors and it's being actively developed and improved, although I was dubious about it initially, in practice it does seem quite good - mainly due to the multithreading. That's why they switched - great to know there is an AUR package for this though, so you can still have an old kernel and the latest firmware, rather than using old firmware for that forever more.

Re: New package: linux-firmware-xz

Reply #2
I do not want to talk about which compression method is better. I just suggested to include linux-firmware-xz in the universe repo, and explained the reason. And personally I hope to have a prebuilt linux-firmware-xz, because my network environment is hostile to git and I use a socks5 proxy to surf on the internet, but it is not easy to use the proxy with makepkg. If the request was reject, I would find some other workarounds.

Re: New package: linux-firmware-xz

Reply #3
you might have a chance if you ask chaotic-aur guys!

Re: New package: linux-firmware-xz

Reply #4
I need the 5.15-lts for some old PCs.

The Question is: If the PC ist so old, that it needs the 5.15, is an up to date Firmeware a good Idea? - Mabe some old (but needed) firmeware will be removed someday, you have the next Problem.

Thus I thing old kernel and firmeware belongs together and keep the last official xz-firmware.

Re: New package: linux-firmware-xz

Reply #5
We need to care about obsoleting firmware, but performance improvements and bug fixes are also important. For now, I decide to wait and see if the AUR package will be updated when the next tag of linux-firmware is released.

Re: New package: linux-firmware-xz

Reply #6
5.15 is only the second LTS in a line of 7 (9 if you include those civil infrastructure maintained ones), I wouldn't call that too old :)

Re: New package: linux-firmware-xz

Reply #7
xz has great compression but it's only single threaded in it's standard incarnation, and the multi-threaded xz alternative versions have never been widely adopted, which is equally true of most of the older compression methods such as bz2. zstd with the right options can offer both decent compression and speed on modern multi-core processors
Already xz #1 is good as zstd #19 and more than 4x faster (both with 4 Threads):

-----
$ time xz -1kT4 fractalus-1.0.0-3-x86_64.pkg.tar;ls -AlF *.xz;rm *.xz
real   0m10,959s
user   0m43,600s
sys   0m0,256s
-rw-r--r-- 1 user user 142939028 28. Mai 2021  fractalus-1.0.0-3-x86_64.pkg.tar.xz

$ time zstd -19T4 fractalus-1.0.0-3-x86_64.pkg.tar;ls -AlF *.zst;rm *.zst
real   0m44,705s
user   2m26,647s
sys   0m0,541s
-rw-r--r-- 1 user user 142596520 28. Mai 2021  fractalus-1.0.0-3-x86_64.pkg.tar.zst
-----

Even xz #7 (#6 is standard) is still faster than zstd #19:

-----
$ time xz -7kT4 fractalus-1.0.0-3-x86_64.pkg.tar;ls -AlF *.xz;rm *.xz
real   0m43,467s
user   2m17,472s
sys   0m0,588s
-rw-r--r-- 1 user user 129776096 28. Mai 2021  fractalus-1.0.0-3-x86_64.pkg.tar.xz
-----

So zstd is a waste of time, space and power, if you compare it to xz with same compression or speed.

Re: New package: linux-firmware-xz

Reply #8
So zstd is a waste of time, space and power, if you compare it to xz with same compression or speed.

Yeah if you only tell half the story
Spoiler (click to show/hide)
So xz is faster compressing (with your settings) but slightly bigger file size (2.6GB vs 2.4GB)

Let's reduce the maxed out compression setting for zstd.
Spoiler (click to show/hide)
Now zst is faster than xz but still has a smaller archive size.

Again
Spoiler (click to show/hide)
Blazingly fast and still a smaller file size than xz on xz's highest level of compression.


Lets extract the archive
Spoiler (click to show/hide)
So even setting  xz to multi threaded decompression it's slower by quite a bit.

And you have to know or trust that whatever is doing the decompression is using the multi threaded setting.

Using default settings to untar the archive
Spoiler (click to show/hide)
I can understand why the switch to zst for firmware and for those on older kernels you can use the AUR package or make your own
In general I think firmware should be kept up to date. By it's nature you don't know normally what security holes and bugs the updated firmware is fixing.




Re: New package: linux-firmware-xz

Reply #10
So zstd is a waste of time, space and power, if you compare it to xz with same compression or speed.
Yeah if you only tell half the story
I don't know comprehensive your example is.

Usually it is like I wrote: No matter if compressing a tar, backing up Linux with FSArchiver (s. Attachment - some years old, but zstd becomes not very much better since then) or Windows with ntfsclone (i. e.):

ntfsclone /dev/sda1 -so - | zstd -169T4 > Windows10.zst
   vs.
ntfsclone /dev/sda1 -so - | xz -1T4 > Windows10.xz

I can't bench it, because my only Windows is this:
Code: [Select]
image: WinXPhome.qcow2
file format: qcow2
virtual size: 1 GiB (1073348608 bytes)
disk size: 303 MiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zstd
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false
I need it sometimes for DVB-S Videocut with VideReDo 4.

Of cause zstd has a much faster decompression (as fast as lzo) and on the faster levels for the speed a very good compression. But if I want the best compression, I use xz or 7-zip with this settings:
Code: [Select]
7zr a -mx -mqs -m1=lzma:d=4G

Btw:

For btrfs I use "compress=zstd:2" (fstab and /etc/udisks2/mount_options.conf), because #3 and higher are nearly not better, but uses more and more CPU-Time.

Re: New package: linux-firmware-xz

Reply #11
I don't know comprehensive your example is.
I'm not sure exactly what you mean ?
I wouldn't say it's in anyway comprehensive. But it appears to show that on my system if I don't crank up the compression to the max with zstd then it can beat xz for compression and decompression times yet still have smaller archive sizes with the texlive-docs package files.

Decompression it seems is always much faster with zstd. Much much faster if xz uses single threaded decompression. Which I learned yesterday is the only option if the xz archive was compressed single threaded.

So in the use case of archives which will be frequently extracted but compressed rarely (if ever by a typical user) zstd seems a reasonable choice. Not "a waste of time, space and power," imho.

I'm glad to read though that Artix may well be providing a xz firmware package for those using older kernels.


Re: New package: linux-firmware-xz

Reply #13
Can somebody please explain what is the use case an the benifit of this?

Re: New package: linux-firmware-xz

Reply #14
Kernels older than 5.19 have no support for the recently introduced zstd compressed firmware. This means if you need to downgrade to an older kernel than this or install an older kernel from the AUR for whatever reason you may experience serious hardware problems and things like wifi stop working unless you first install an xz compressed firmware package.