Recently, archlinux compressed firmwares via zstd. However, old kernels such as 5.15-lts do not support the compression. Someone posted xz-compressed firmware at https://aur.archlinux.org/packages/linux-firmware-xz
As the universe repo ships linux-lts515, could the repo ships linux-firmware-xz altogether?
Some discussion at the linux-lts515 page:
https://aur.archlinux.org/packages/linux-lts515
xz has great compression but it's only single threaded in it's standard incarnation, and the multi-threaded xz alternative versions have never been widely adopted, which is equally true of most of the older compression methods such as bz2. zstd with the right options can offer both decent compression and speed on modern multi-core processors and it's being actively developed and improved, although I was dubious about it initially, in practice it does seem quite good - mainly due to the multithreading. That's why they switched - great to know there is an AUR package for this though, so you can still have an old kernel and the latest firmware, rather than using old firmware for that forever more.
I do not want to talk about which compression method is better. I just suggested to include linux-firmware-xz in the universe repo, and explained the reason. And personally I hope to have a prebuilt linux-firmware-xz, because my network environment is hostile to git and I use a socks5 proxy to surf on the internet, but it is not easy to use the proxy with makepkg. If the request was reject, I would find some other workarounds.
you might have a chance if you ask chaotic-aur guys!
I need the 5.15-lts for some old PCs.
The Question is: If the PC ist so old, that it needs the 5.15, is an up to date Firmeware a good Idea? - Mabe some old (but needed) firmeware will be removed someday, you have the next Problem.
Thus I thing old kernel and firmeware belongs together and keep the last official xz-firmware.
We need to care about obsoleting firmware, but performance improvements and bug fixes are also important. For now, I decide to wait and see if the AUR package will be updated when the next tag of linux-firmware is released.
5.15 is only the second LTS in a line of 7 (9 if you include those civil infrastructure maintained ones), I wouldn't call that too old :)
Already xz #1 is good as zstd #19
and more than 4x faster (both with 4 Threads):
-----
$ time xz -1kT4 fractalus-1.0.0-3-x86_64.pkg.tar;ls -AlF *.xz;rm *.xz
real 0m
10,959suser 0m43,600s
sys 0m0,256s
-rw-r--r-- 1 user user
142939028 28. Mai 2021 fractalus-1.0.0-3-x86_64.pkg.tar.xz
$ time zstd -19T4 fractalus-1.0.0-3-x86_64.pkg.tar;ls -AlF *.zst;rm *.zst
real 0m
44,705suser 2m26,647s
sys 0m0,541s
-rw-r--r-- 1 user user
142596520 28. Mai 2021 fractalus-1.0.0-3-x86_64.pkg.tar.zst
-----
Even xz #7 (#6 is standard) is still faster than zstd #19:
-----
$ time xz -7kT4 fractalus-1.0.0-3-x86_64.pkg.tar;ls -AlF *.xz;rm *.xz
real 0m
43,467suser 2m17,472s
sys 0m0,588s
-rw-r--r-- 1 user user
129776096 28. Mai 2021 fractalus-1.0.0-3-x86_64.pkg.tar.xz
-----
So zstd is a waste of time, space and power, if you compare it to xz with same compression or speed.
Yeah if you only tell half the story
❯ time xz -1kT4 texlive-doc-2023.66594-19-any.pkg.tar
________________________________________________________
Executed in 107.39 secs fish external
usr time 427.92 secs 322.00 micros 427.92 secs
sys time 1.80 secs 23.00 micros 1.80 secs
❯ time zstd -19T4 texlive-doc-2023.66594-19-any.pkg.tar
texlive-doc-2023.66594-19-any.pkg.tar : 70.43% ( 3.38 GiB => 2.38 GiB, texlive-doc-2023.66594-19-any.pkg.tar.zst)
________________________________________________________
Executed in 191.90 secs fish external
usr time 755.59 secs 415.00 micros 755.59 secs
sys time 3.40 secs 31.00 micros 3.40 secs
So xz is faster compressing (with your settings) but slightly bigger file size (2.6GB vs 2.4GB)
Let's reduce the maxed out compression setting for zstd.
root@Artix /t/test# sync; echo 3 > /proc/sys/vm/drop_caches
root@Artix /t/test# time zstd -10T4 texlive-doc-2023.66594-19-any.pkg.tar
texlive-doc-2023.66594-19-any.pkg.tar : 72.54% ( 3.38 GiB => 2.45 GiB, texlive-doc-2023.66594-19-any.pkg.tar.zst)
________________________________________________________
Executed in 14.79 secs fish external
usr time 49.06 secs 5.00 micros 49.06 secs
sys time 2.81 secs 317.00 micros 2.81 secs
Now zst is faster than xz but still has a smaller archive size.
Again
root@Artix /t/test# time zstd -4T4 texlive-doc-2023.66594-19-any.pkg.tar
texlive-doc-2023.66594-19-any.pkg.tar : 74.92% ( 3.38 GiB => 2.53 GiB, texlive-doc-2023.66594-19-any.pkg.tar.zst)
________________________________________________________
Executed in 4.44 secs fish external
usr time 10.00 secs 9.00 micros 10.00 secs
sys time 2.35 secs 326.00 micros 2.35 secs
Blazingly fast and still a smaller file size than xz on xz's highest level of compression.
Lets extract the archive
root@Artix /t/test# sync; echo 3 > /proc/sys/vm/drop_caches
root@Artix /t/test# time xz -dT16 texlive-doc-2023.66594-19-any.pkg.tar.xz
________________________________________________________
Executed in 7.42 secs fish external
usr time 108.48 secs 348.00 micros 108.48 secs
sys time 2.30 secs 0.00 micros 2.30 secs
root@Artix /t/test# sync; echo 3 > /proc/sys/vm/drop_caches
root@Artix /t/test# time xz -d texlive-doc-2023.66594-19-any.pkg.tar.xz
________________________________________________________
Executed in 94.42 secs fish external
usr time 92.45 secs 0.00 micros 92.45 secs
sys time 1.89 secs 351.00 micros 1.89 secs
root@Artix /t/test# sync; echo 3 > /proc/sys/vm/drop_caches
root@Artix /t/test# time zstd -d texlive-doc-2023.66594-19-any.pkg.tar.zst
texlive-doc-2023.66594-19-any.pkg.tar.zst: 3625216000 bytes
________________________________________________________
Executed in 1.83 secs fish external
usr time 1.86 secs 13.00 micros 1.86 secs
sys time 2.01 secs 337.00 micros 2.01 secs
So even setting xz to multi threaded decompression it's slower by quite a bit.
And you have to know or trust that whatever is doing the decompression is using the multi threaded setting.
Using default settings to untar the archive
root@Artix /t/t/extract# time tar -xf ../texlive-doc-2023.66594-19-any.pkg.tar.xz
________________________________________________________
Executed in 93.53 secs fish external
usr time 92.49 secs 320.00 micros 92.49 secs
sys time 2.95 secs 0.00 micros 2.95 secs
root@Artix /t/t/extract# time tar -xf ../texlive-doc-2023.66594-19-any.pkg.tar.zst
________________________________________________________
Executed in 1.96 secs fish external
usr time 1.93 secs 15.00 micros 1.93 secs
sys time 2.83 secs 362.00 micros 2.83 secs
I can understand why the switch to zst for firmware and for those on older kernels you can use the AUR package or make your own
In general I think firmware should be kept up to date. By it's nature you don't know normally what security holes and bugs the updated firmware is fixing.
It's a reasonable request, we'll see what we can do. Don't expect very frequent updates though, especially as it's targeted towards older systems anyway.
I don't know comprehensive your example is.
Usually it is like I wrote: No matter if compressing a tar, backing up Linux with FSArchiver (s. Attachment - some years old, but zstd becomes not very much better since then) or Windows with ntfsclone (i. e.):
ntfsclone /dev/sda1 -so - | zstd -169T4 > Windows10.zst
vs.
ntfsclone /dev/sda1 -so - | xz -1T4 > Windows10.xz
I can't bench it, because my only Windows is this:
image: WinXPhome.qcow2
file format: qcow2
virtual size: 1 GiB (1073348608 bytes)
disk size: 303 MiB
cluster_size: 65536
Format specific information:
compat: 1.1
compression type: zstd
lazy refcounts: false
refcount bits: 16
corrupt: false
extended l2: false
I need it sometimes for DVB-S Videocut with VideReDo 4.
Of cause zstd has a much faster decompression (as fast as lzo) and on the faster levels for the speed a very good compression. But if I want the best compression, I use xz or 7-zip with this settings:
7zr a -mx -mqs -m1=lzma:d=4G
Btw:
For btrfs I use "compress=zstd:2" (fstab and /etc/udisks2/mount_options.conf), because #3 and higher are nearly not better, but uses more and more CPU-Time.
I'm not sure exactly what you mean ?
I wouldn't say it's in anyway comprehensive. But it appears to show that on my system if I don't crank up the compression to the max with zstd then it can beat xz for compression and decompression times yet still have smaller archive sizes with the texlive-docs package files.
Decompression it seems is always much faster with zstd. Much much faster if xz uses single threaded decompression. Which I learned yesterday is the only option if the xz archive was compressed single threaded.
So in the use case of archives which will be frequently extracted but compressed rarely (if ever by a typical user) zstd seems a reasonable choice. Not "a waste of time, space and power," imho.
I'm glad to read though that Artix may well be providing a xz firmware package for those using older kernels.
The linux-firmware-xz packages have been added to [universe], wait a few hours until the mirrors (https://wiki.artixlinux.org/Main/Repositories#Universe) pick them up.
Can somebody please explain what is the use case an the benifit of this?
Kernels older than 5.19 have no support for the recently introduced zstd compressed firmware. This means if you need to downgrade to an older kernel than this or install an older kernel from the AUR for whatever reason you may experience serious hardware problems and things like wifi stop working unless you first install an xz compressed firmware package.