New package: linux-firmware-xz

Topic: New package: linux-firmware-xz (Read 1153 times) previous topic - next topic

0 Members and 2 Guests are viewing this topic.

New package: linux-firmware-xz

29 July 2023, 15:32:52

Recently, archlinux compressed firmwares via zstd. However, old kernels such as 5.15-lts do not support the compression. Someone posted xz-compressed firmware at https://aur.archlinux.org/packages/linux-firmware-xz

As the universe repo ships linux-lts515, could the repo ships linux-firmware-xz altogether?

Some discussion at the linux-lts515 page:
https://aur.archlinux.org/packages/linux-lts515

Re: New package: linux-firmware-xz

Reply #1 – 29 July 2023, 19:13:50

xz has great compression but it's only single threaded in it's standard incarnation, and the multi-threaded xz alternative versions have never been widely adopted, which is equally true of most of the older compression methods such as bz2. zstd with the right options can offer both decent compression and speed on modern multi-core processors and it's being actively developed and improved, although I was dubious about it initially, in practice it does seem quite good - mainly due to the multithreading. That's why they switched - great to know there is an AUR package for this though, so you can still have an old kernel and the latest firmware, rather than using old firmware for that forever more.

Re: New package: linux-firmware-xz

Reply #2 – 30 July 2023, 06:01:32

I do not want to talk about which compression method is better. I just suggested to include linux-firmware-xz in the universe repo, and explained the reason. And personally I hope to have a prebuilt linux-firmware-xz, because my network environment is hostile to git and I use a socks5 proxy to surf on the internet, but it is not easy to use the proxy with makepkg. If the request was reject, I would find some other workarounds.

Re: New package: linux-firmware-xz

Reply #3 – 30 July 2023, 08:35:33

you might have a chance if you ask chaotic-aur guys!

Re: New package: linux-firmware-xz

Reply #4 – 31 July 2023, 08:14:03

I need the 5.15-lts for some old PCs.

The Question is: If the PC ist so old, that it needs the 5.15, is an up to date Firmeware a good Idea? - Mabe some old (but needed) firmeware will be removed someday, you have the next Problem.

Thus I thing old kernel and firmeware belongs together and keep the last official xz-firmware.

Re: New package: linux-firmware-xz

Reply #5 – 31 July 2023, 09:53:26

We need to care about obsoleting firmware, but performance improvements and bug fixes are also important. For now, I decide to wait and see if the AUR package will be updated when the next tag of linux-firmware is released.

Re: New package: linux-firmware-xz

Reply #6 – 31 July 2023, 14:27:18

5.15 is only the second LTS in a line of 7 (9 if you include those civil infrastructure maintained ones), I wouldn't call that too old

Re: New package: linux-firmware-xz

Reply #7 – 31 July 2023, 15:22:58

Quote from: ####### – on 29 July 2023, 19:13:50

xz has great compression but it's only single threaded in it's standard incarnation, and the multi-threaded xz alternative versions have never been widely adopted, which is equally true of most of the older compression methods such as bz2. zstd with the right options can offer both decent compression and speed on modern multi-core processors

Already xz #1 is good as zstd #19 and more than 4x faster (both with 4 Threads):

-----
$ time xz -1kT4 fractalus-1.0.0-3-x86_64.pkg.tar;ls -AlF *.xz;rm *.xz
real   0m10,959s
user   0m43,600s
sys   0m0,256s
-rw-r--r-- 1 user user 142939028 28. Mai 2021 fractalus-1.0.0-3-x86_64.pkg.tar.xz

$ time zstd -19T4 fractalus-1.0.0-3-x86_64.pkg.tar;ls -AlF *.zst;rm *.zst
real   0m44,705s
user   2m26,647s
sys   0m0,541s
-rw-r--r-- 1 user user 142596520 28. Mai 2021 fractalus-1.0.0-3-x86_64.pkg.tar.zst
-----

Even xz #7 (#6 is standard) is still faster than zstd #19:

-----
$ time xz -7kT4 fractalus-1.0.0-3-x86_64.pkg.tar;ls -AlF *.xz;rm *.xz
real   0m43,467s
user   2m17,472s
sys   0m0,588s
-rw-r--r-- 1 user user 129776096 28. Mai 2021 fractalus-1.0.0-3-x86_64.pkg.tar.xz
-----

So zstd is a waste of time, space and power, if you compare it to xz with same compression or speed.

Re: New package: linux-firmware-xz

Reply #8 – 31 July 2023, 17:17:15

Quote from: Caramon2 – on 31 July 2023, 15:22:58

So zstd is a waste of time, space and power, if you compare it to xz with same compression or speed.

Yeah if you only tell half the story
Spoiler (click to show/hide)

Code: [Select]

❯ time xz -1kT4 texlive-doc-2023.66594-19-any.pkg.tar

________________________________________________________
Executed in  107.39 secs    fish           external
   usr time  427.92 secs  322.00 micros  427.92 secs
   sys time    1.80 secs   23.00 micros    1.80 secs


❯ time zstd -19T4  texlive-doc-2023.66594-19-any.pkg.tar
texlive-doc-2023.66594-19-any.pkg.tar : 70.43%   (  3.38 GiB =>   2.38 GiB, texlive-doc-2023.66594-19-any.pkg.tar.zst) 

________________________________________________________
Executed in  191.90 secs    fish           external
   usr time  755.59 secs  415.00 micros  755.59 secs
   sys time    3.40 secs   31.00 micros    3.40 secs

So xz is faster compressing (with your settings) but slightly bigger file size (2.6GB vs 2.4GB)

Let's reduce the maxed out compression setting for zstd.
Spoiler (click to show/hide)

Code: [Select]

root@Artix /t/test# sync; echo 3 > /proc/sys/vm/drop_caches
root@Artix /t/test# time zstd -10T4 texlive-doc-2023.66594-19-any.pkg.tar
texlive-doc-2023.66594-19-any.pkg.tar : 72.54%   (  3.38 GiB =>   2.45 GiB, texlive-doc-2023.66594-19-any.pkg.tar.zst) 

________________________________________________________
Executed in   14.79 secs    fish           external
   usr time   49.06 secs    5.00 micros   49.06 secs
   sys time    2.81 secs  317.00 micros    2.81 secs

Now zst is faster than xz but still has a smaller archive size.

Again
Spoiler (click to show/hide)

Code: [Select]

root@Artix /t/test# time zstd -4T4 texlive-doc-2023.66594-19-any.pkg.tar
texlive-doc-2023.66594-19-any.pkg.tar : 74.92%   (  3.38 GiB =>   2.53 GiB, texlive-doc-2023.66594-19-any.pkg.tar.zst) 

________________________________________________________
Executed in    4.44 secs    fish           external
   usr time   10.00 secs    9.00 micros   10.00 secs
   sys time    2.35 secs  326.00 micros    2.35 secs

Blazingly fast and still a smaller file size than xz on xz's highest level of compression.

Lets extract the archive
Spoiler (click to show/hide)

Code: [Select]

root@Artix /t/test# sync; echo 3 > /proc/sys/vm/drop_caches
root@Artix /t/test# time xz -dT16  texlive-doc-2023.66594-19-any.pkg.tar.xz

________________________________________________________
Executed in    7.42 secs    fish           external
   usr time  108.48 secs  348.00 micros  108.48 secs
   sys time    2.30 secs    0.00 micros    2.30 secs

root@Artix /t/test# sync; echo 3 > /proc/sys/vm/drop_caches
root@Artix /t/test# time xz -d  texlive-doc-2023.66594-19-any.pkg.tar.xz

________________________________________________________
Executed in   94.42 secs    fish           external
   usr time   92.45 secs    0.00 micros   92.45 secs
   sys time    1.89 secs  351.00 micros    1.89 secs

root@Artix /t/test# sync; echo 3 > /proc/sys/vm/drop_caches
root@Artix /t/test# time zstd -d texlive-doc-2023.66594-19-any.pkg.tar.zst
texlive-doc-2023.66594-19-any.pkg.tar.zst: 3625216000 bytes                    

________________________________________________________
Executed in    1.83 secs    fish           external
   usr time    1.86 secs   13.00 micros    1.86 secs
   sys time    2.01 secs  337.00 micros    2.01 secs

So even setting xz to multi threaded decompression it's slower by quite a bit.

And you have to know or trust that whatever is doing the decompression is using the multi threaded setting.

Using default settings to untar the archive
Spoiler (click to show/hide)

Code: [Select]

root@Artix /t/t/extract# time tar -xf ../texlive-doc-2023.66594-19-any.pkg.tar.xz

________________________________________________________
Executed in   93.53 secs    fish           external
   usr time   92.49 secs  320.00 micros   92.49 secs
   sys time    2.95 secs    0.00 micros    2.95 secs

root@Artix /t/t/extract# time tar -xf ../texlive-doc-2023.66594-19-any.pkg.tar.zst 

________________________________________________________
Executed in    1.96 secs    fish           external
   usr time    1.93 secs   15.00 micros    1.93 secs
   sys time    2.83 secs  362.00 micros    2.83 secs

I can understand why the switch to zst for firmware and for those on older kernels you can use the AUR package or make your own
In general I think firmware should be kept up to date. By it's nature you don't know normally what security holes and bugs the updated firmware is fixing.

Re: New package: linux-firmware-xz

Reply #9 – 31 July 2023, 22:53:06

Quote from: glgl-schemer – on 29 July 2023, 15:32:52

As the universe repo ships linux-lts515, could the repo ships linux-firmware-xz altogether?

It's a reasonable request, we'll see what we can do. Don't expect very frequent updates though, especially as it's targeted towards older systems anyway.

Re: New package: linux-firmware-xz

Reply #10 – 01 August 2023, 12:00:59

Quote from: gripped – on 31 July 2023, 17:17:15

Quote from: Caramon2 – on 31 July 2023, 15:22:58
So zstd is a waste of time, space and power, if you compare it to xz with same compression or speed.
Yeah if you only tell half the story

I don't know comprehensive your example is.

Usually it is like I wrote: No matter if compressing a tar, backing up Linux with FSArchiver (s. Attachment - some years old, but zstd becomes not very much better since then) or Windows with ntfsclone (i. e.):

ntfsclone /dev/sda1 -so - | zstd -169T4 > Windows10.zst
vs.
ntfsclone /dev/sda1 -so - | xz -1T4 > Windows10.xz

I can't bench it, because my only Windows is this:

Code: [Select]

image: WinXPhome.qcow2
file format: qcow2
virtual size: 1 GiB (1073348608 bytes)
disk size: 303 MiB
cluster_size: 65536
Format specific information:
    compat: 1.1
    compression type: zstd
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
    extended l2: false

I need it sometimes for DVB-S Videocut with VideReDo 4.

Of cause zstd has a much faster decompression (as fast as lzo) and on the faster levels for the speed a very good compression. But if I want the best compression, I use xz or 7-zip with this settings:

Code: [Select]

7zr a -mx -mqs -m1=lzma:d=4G

Btw:

For btrfs I use "compress=zstd:2" (fstab and /etc/udisks2/mount_options.conf), because #3 and higher are nearly not better, but uses more and more CPU-Time.

Re: New package: linux-firmware-xz

Reply #11 – 01 August 2023, 13:02:55

Quote from: Caramon2 – on 01 August 2023, 12:00:59

I don't know comprehensive your example is.

I'm not sure exactly what you mean ?
I wouldn't say it's in anyway comprehensive. But it appears to show that on my system if I don't crank up the compression to the max with zstd then it can beat xz for compression and decompression times yet still have smaller archive sizes with the texlive-docs package files.

Decompression it seems is always much faster with zstd. Much much faster if xz uses single threaded decompression. Which I learned yesterday is the only option if the xz archive was compressed single threaded.

So in the use case of archives which will be frequently extracted but compressed rarely (if ever by a typical user) zstd seems a reasonable choice. Not "a waste of time, space and power," imho.

I'm glad to read though that Artix may well be providing a xz firmware package for those using older kernels.

Re: New package: linux-firmware-xz

Reply #12 – 01 August 2023, 21:39:49

The linux-firmware-xz packages have been added to [universe], wait a few hours until the mirrors pick them up.

Re: New package: linux-firmware-xz

Reply #13 – 05 September 2023, 17:52:13

Can somebody please explain what is the use case an the benifit of this?

Re: New package: linux-firmware-xz

Reply #14 – 05 September 2023, 18:22:57

Kernels older than 5.19 have no support for the recently introduced zstd compressed firmware. This means if you need to downgrade to an older kernel than this or install an older kernel from the AUR for whatever reason you may experience serious hardware problems and things like wifi stop working unless you first install an xz compressed firmware package.