Skip to main content ↑ ↓
Topic: Benchmarking Artix & Comparing it to other distributions (Read 750 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Benchmarking Artix & Comparing it to other distributions

So, here's a project that I've been working on for the past week or so: benchmarking Artix (all of the available init systems) and other distributions.
I tested different systems in separate virtual machines and wrote the results down. The data is available on a website (in fact, it has all of the info in its entirety), in a JSON file, and, since this forum's software allows tables, here's a truncated one:
ValueArchArtix OpenRCArtix runitArtix s6Artix dinitDebian
First boot5.2 seconds9.6 seconds6.6 seconds4.7 seconds5 seconds8 seconds
Second boot5.2 seconds8.1 seconds6.5 seconds4.7 seconds5 seconds6.8 seconds
Optimized Boot-5.7 seconds6 seconds---
Shutdown2.3 seconds6.3 seconds4.6 seconds5.9 seconds2.8 seconds2.8 seconds
RAM usage270 MiB250 MiB257 MiB258 MiB251 MiB280 MiB
CPU usage0.2%0.2%0.2%0.8%0.2%0.2%
Packages installed129143141145140220
First boot + services10.5 seconds13.6 seconds6.4 seconds6.3 seconds5.4 seconds9.2 seconds
Second boot + services7.2 seconds13.3 seconds5.9 seconds6.5 seconds5.4 seconds9.5 seconds
Optimized boot + services-9.1 seconds5.8 seconds---
Shutdown + services2.3 seconds8.6 seconds5.3 seconds4.7 seconds2.6 seconds3 seconds
For more values & notes, please see the website. Also, all of the sources can be found on Codeberg.

So, what's the moral of the story? Personally, I think that an extra 10 MiB of RAM and a difference of a few seconds in boot/shutdown times don't really matter. What matters is if you get hit by "a stop job is running" or bugs. Which means that rather than choosing your init by comparing perfomance, you should choose it by analyzing the:
- Core design
- Security
- Community support
- Stability
and so on. Unfortunately, it's not really possible to benchmark those in an objective way, so that task is up to the user.

PS: I plan on doing more measurements (e.g. testing with a desktop environment) and maybe repeating all of this in the future. So, any contributions/suggestions/corrections are welcome! ;)

Re: Benchmarking Artix & Comparing it to other distributions

Reply #1 –
Interesting.
It's nice to see the differences between the Artix inits.

I've read on here that dinit was fast and this confirms it but I agree "a difference of a few seconds in boot/shutdown times don't really matter". Not enough for me to consider switching from openrc. Not just for better up / down times anyway.

I can't really see how how the init should make much, or any, difference to 'real world' benchmarks once the system is fully up. Assuming all the non init parts of the OS remain the same across the tested versions ?
But regardless I look forward to seeing the results of your further testing.

Thanks for the effort.

Re: Benchmarking Artix & Comparing it to other distributions

Reply #2 –
Quote
I can't really see how how the init should make much, or any, difference to 'real world' benchmarks once the system is fully up.
Mostly I want to see how quickly the login manager appears, and try to reproduce the "a stop job is running" thing (if anyone knows which services usually cause this please let me know)
By the way, here are some more ideas:
- Testing ISO boot times
- Replicating a server setup (mostly installing services like nginx)

Re: Benchmarking Artix & Comparing it to other distributions

Reply #3 –
Quote from: MaxLPM – on
and try to reproduce the "a stop job is running" thing.
Code: [Select]
#!/bin/bash
SYSTEMD="insane"
counter=1
while [ "$SYSTEMD" != "sane" ]; do
    echo -ne "A stop job is running for a vague reason (${counter}s \ 99999999999999s)\r"
    counter=$((counter + 1))
    sleep 1
done
That's pretty much exactly how they do it  ;)

Re: Benchmarking Artix & Comparing it to other distributions

Reply #4 –
Quote from: MaxLPM – on
PS: I plan on doing more measurements (e.g. testing with a desktop environment) and maybe repeating all of this in the future. So, any contributions/suggestions/corrections are welcome! ;)
Great job, thanks. Perhaps you'll find this little script may save you some time creating automated installations, I use it to test our base ISOs functionality.

Re: Benchmarking Artix & Comparing it to other distributions

Reply #5 –
Slow and steady can be safer, it allows hardware to power up and services to run in n orderly manner, and at shutdown it allows jobs to complete and data to be saved. Whether that's important depends on what hardware and software is in use.
 If you want to create a hang at shutdown you need to create a service that keeps running and blocks SIGTERM and perhaps other fatal signals so it won't quit until SIGKILL is sent, you might be able to deduce the delay time until this happens from just looking at the init code though.
You can do this in a simple BASH script, and most other languages, e.g.:
https://stackoverflow.com/questions/24848843/how-do-i-stop-a-signal-from-killing-my-bash-script
 

Re: Benchmarking Artix & Comparing it to other distributions

Reply #6 –
Quote from: MaxLPM – on
I plan on doing more measurements (e.g. testing with a desktop environment) and maybe repeating all of this in the future. So, any contributions/suggestions/corrections are welcome! ;)

RTFM:

https://joborun.neocities.org/joborun
https://joborun.neocities.org/goals
https://wiki.gentoo.org/wiki/GCC_optimization
"Wer alles kann, macht nichts richtig"

Artix USE="runit openrc slim openbox lxde gtk2 qt4 qt5 qt6 conky
-gtk3 -gtk4 -adwaita{cursors,themes,icons} -gnome3 -kde -plasma -wayland "