sudo dinitctl list
...
[{+} ] fsck-root
[ {X}] early-fs-fstab.target
[ {-}] fsck
[{+} ] early-console.target
...
dinit 0.18.0-5
dinit-base 0.18.0-5
dinit-rc 0.4.7-1
sudo dinitctl status early-fs-fstab.target
Service: early-fs-fstab.target
State: STOPPED (failed to start; exited - status 64)
All other services run just fine. Haven't notice any issues with it but wanted to report just in case.
TLDR: probably nothing to worry about as your important devices are still being mounted,
else you would be having further problems. Just "solved" this on my machine so I will walk you through what I did and what might be the problem for you as well.
Looking inside
early-fs-fstab.target, we have:
type = scripted
command = /usr/bin/mount -a -t nonfs,nonfs4,nosmbfs,nocifs,nocodafs,noncpfs,noshfs,nofuse,nofuseblk,noglusterfs,nodavfs,nofuse.glusterfs -O no_netdev
restart = false
depends-on = early-fs-pre.target
depends-ms = fsck
waits-for = early-root-rw.target
Interpretation: when
early-fs-fstab.target is called by dinit, it runs the following command once before finishing the service:
mount -a -t nonfs,nonfs4,nosmbfs,nocifs,nocodafs,noncpfs,noshfs,nofuse,nofuseblk,noglusterfs,nodavfs,nofuse.glusterfs -O no_netdev
This command tries to mount all the devices listed at
/etc/fstab, except for the drives with the
types that are explicitly excluded in the command with the -t flag and the 'no' prefix (e.g.
mount -t nonfs: ignore any drive that is of the nfs type. Same logic for the other drive types).
Knowing this, it is easy to infer that the problem is likely some badly declared entry at
/etc/fstab.
In my case, the entries on the file that were causing errors were these two:
# /dev/sdc1 LABEL=ARTIX_202308
UUID=2023-08-13-15-59-29-00 /run/artix/bootmnt iso9660 ro,relatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8 0 0
# /dev/loop0
/dev/loop0 /run/artix/sfs/rootfs squashfs ro,relatime,errors=continue,threads=single 0 0
As you can see the specified mount-points were
/run/artix/bootmnt and
/run/artix/sfs/rootfs respectively, but checking these locations i noticed
/run/artix does not exist on my computer.
I'm not really sure what these mount points are supposed to be (maybe they are used for the first install of the OS?), and I don't remember setting them up myself either, but since they were causing the error (and weren't getting mounted anyways), I just commented them out:
# /dev/sdc1 LABEL=ARTIX_202308
# UUID=2023-08-13-15-59-29-00 /run/artix/bootmnt iso9660 ro,relatime,nojoliet,check=s,map=n,blocksize=2048,iocharset=utf8 0 0
# /dev/loop0
# /dev/loop0 /run/artix/sfs/rootfs squashfs ro,relatime,errors=continue,threads=single 0 0
And just like that, the error is gone! Well, at least it was gone for me. Note that this doesn't actually fix anything as the entries are still
not being mounted, though the system itself works fine without them. I chose to do this because as I said I'm not sure as to what those entries are for and thus would rather leave them commented out.
NOTE: I got all of this knowledge from just reading the relevant files and the man entry for dinit/mount, so if anyone has any corrections and/or an explanation as to what these two fstab entries are (I also have a commented out 'overlay' entry that I don't understand), please respond!
@bizio thanks for such a detailed reply! I inspected my fstab can't spot anything. Got couple ntfs drives and btrfs subvolumes gonna be pain to debug. Last time I edited fstab on my debian server I wasn't able to ssh to it on next reboot and ended up reinstalling debian because I couldn't recover (even after chroot'ing and editing fstab back), so I'm little hesitant about touching it on my daily driver :D
Forgot to add this to original post:
sudo dinitctl status early-fs-fstab.target
Service: early-fs-fstab.target
State: STOPPED (failed to start; exited - status 64)
Not sure where to see logs or something more then just status code
self-note:
/usr/lib/dinit.d/early-fs-fstab.target
There isn't any logging for the service configured via the service description file, and mount doesn't do any logging of its own, so unfortunately, you can't.
However, the service description file settings are documented (https://davmac.org/projects/dinit/man-pages-html/dinit-service.5.html) so you can easily add logging by editing the file. See the "log-type", "logfile", "log-buffer-size" settings in particular.
/usr/lib/dinit.d/easrly-fs-fstab.target
type = scripted
command = /usr/bin/mount -a -t nonfs,nonfs4,nosmbfs,nocifs,nocodafs,noncpfs,noshfs,nofuse,nofuseblk,noglusterfs,nodavfs,nofuse.glusterfs -O no_netdev
restart = false
depends-on = early-fs-pre.target
depends-ms = fsck
waits-for = early-root-rw.target
log-type = file
logfile = /var/log/early-fs-fstab.log
sudo touch /var/log/early-fs-fstab.log
After reboot log file is empty, did I do something wrong?
I have even on a normal install (some tablet with buggy gpu which hangs once in a while), failure to boot with that service even if the volume repaired successfully from the unclean unmount.
It really shouldn't, i mean if fsck succeeded in repairing it doesn't exit with >1 is that right? Or it's 0 just if there's a clean filesystem with nothing to check?
via "man fsck":
The exit status returned by fsck is the sum of the following
conditions:
0
No errors
1
Filesystem errors corrected
2
System should be rebooted
4
Filesystem errors left uncorrected
The sample rootfscheck script (https://github.com/davmac314/dinit/blob/master/doc/linux/services/rootfscheck.sh) included in the Dinit source distribution shows how 2/4 can be handled.
Thanks, so that is what's
missing not working just right here, the /usr/lib/dinit/fsck script looks similar.
I think the case argument for code 1 works correctly on another pc i just tested an unclean unmount with, but i don't see the echo...
... and the case argument for 2 it seems it gets stuck, also saw no echod text and probably that command "reboot --use-passed-cfd -r" doesn't pass and exits the script itself with non 0 then?
@konimex Sorry but i don't have the energy to test further.
# it's a bitwise-or, but we are only checking one filesystem
case $? in
0) ;; # nothing
1) # fixed errors
echo "WARNING: The root filesystem was repaired, continuing boot..."
sleep 2
;;
2) # system should be rebooted
echo "WARNING: The root filesystem was repaired, rebooting..."
sleep 5
reboot --use-passed-cfd -r
;;
4) # uncorrected errors
echo "WARNING: The root filesystem has unrecoverable errors."
echo " A recovery shell will now be started for you."
echo " The system will be rebooted when you are done."
sulogin
reboot --use-passed-cfd -r
;;
*) ;;
esac
Right, looks like it's incorrectly assuming an exact return value of 1, 2, or 4, rather than a bitmask. I don't think you'll ever get 2 by itself, it would always be combined with 1 (at least).
I suspect
--use-pass-cfd won't work with bash (I think, from memory, bash closes "extra" file descriptors), so that's one potential cause of problems, but note that option is not needed if the control socket was already created (eg if /run has already been mounted with a tmpfs via the initial ram disk, which I think should normally be the case for Artix?)
same issue here. same exit code. Nothing unusual in fstab as well.
And I still don't see the way to fix it.
On my system (with dinit 0.19.0-1) I get
unknown reason from
dinitctl:
❯ sudo dinitctl status early-fs-fstab.target
Service: early-fs-fstab.target
State: STOPPED (failed to start; unknown reason)
Although in
/var/log/daemon.log I find:
2024-10-18T07:35:32+02:00 phoenix dinit[1]: Service early-fs-fstab.target command failed with exit code 64
2024-10-18T07:35:32+02:00 phoenix dinit[1]: service early-fs-fstab.target failed to start.
Maybe we should update to
dinit 0.19.1, since there have been important fixes ?
with
dinit-0.19.1-1
dinitctl list
[ {X}] early-fs-fstab.target
[ {-}] fsckdinitctl status early-fs-fstab.target
Service: early-fs-fstab.target
State: STOPPED (failed to start; exited - status 64)
Well, first of all, thanks for updating
dinit to
0.19.1.
There is a slight improvement -
dinitctl now reports the exit code (64):
❯ sudo dinitctl status early-fs-fstab.target
Service: early-fs-fstab.target
State: STOPPED (failed to start; exited - status 64)
But the pesky
early-fs-fstab.target is still failing:
2024-10-19T09:42:00+02:00 phoenix dinit[1]: Service early-fs-fstab.target command failed with exit code 64
2024-10-19T09:42:00+02:00 phoenix dinit[1]: service early-fs-fstab.target failed to start.
* edited do correct a typo
I noticed that the
random-seed service is also failed; I started it (and it remains active).
After that I manually started the
early-fs-fstab.target service; it did not bail out, but it did not finish starting either :
The command
dinitctl start early-fs-fstab.target
hangs indefinitely.
When interrogating
dinit about the status of
early-fs-fstab.target , I get this:
dinitctl status early-fs-fstab.target
Service: early-fs-fstab.target
State: STARTING
Activation: explicitly started
I have just updated to 0.19.1 and after issuing the command:
dinitctl status early.fs.fstab.target
I get the following output:
Service: early-fs-fstab.target
State: STOPPED (dependency failed/terminated)
(from a root terminal btw)
Edited for typos.
It is waiting on the fsck service, which is configured to run on the console (and the console is busy as the login service is running on it). In general you cannot stop and start critical system services like this; the Artix configuration (perhaps incorrectly) allows you to stop early-fs-fstab.target without bringing the whole system down, but you won't be able to start it again. Reboot the system to do so. (That doesn't help if the service is failing to start, of course, but I can't help with that issue I'm afraid).
Thanks for clearing that up.
The curious thing is that if I run the command corresponding to the
early-fs-fstab.target service, from the root console, it executes just fine.
Could it be that some rights and/or environment variables are not yet as they are expected when the service is started, which leads to the failure ?
It would be of tremendous help to be able to "dump" all the contents of the context in which the service is executed (that includes the environment variables, the rights, and the available services).
I suppose there is no way to make
dinit to tell us all the detail, right ? Something like a special flag that lists in a trace file everything happening in
dinit ...
Well you can change the executable ("command = ") to something that dumps whatever information you want.
It seems to me like a better place to start would be just to configure some kind of logging for the "early-fs-fstab.target" service. The service description doesn't currently specify a "log-type" or "logfile" at all.
Well, it did crap out immediately - because the file-system is not yet ready and the log file cannot be written.
type = scripted
command = /usr/bin/mount -a -t nonfs,nonfs4,nosmbfs,nocifs,nocodafs,noncpfs,noshfs,nofuse,nofuseblk,noglusterfs,nodavfs,nofuse.glusterfs -O no_netdev
restart = false
depends-on = early-fs-pre.target
depends-ms = fsck
waits-for = early-root-rw.target
# added log file specification
logfile = /var/log/dinit/early-fs-fstab.target.log
My
/var is mounted on another partition - which was not yet mounted when
dinit tried to open the log file, because
early-fs-fstab.target was not yet processed.
[ OK ] early-root-rw.target
dinit: early-fs-fstab.target: execution failed - opening log file: No such file or directory
[FAILED] early-fs-fstab.target
I will try with
logfile = /root/early-fs-fstab.target.log
to see what it gives ...
Well, the plot thickens: now that the log has been created (
/root/early-fs-fstab.target.log), it says:
mount: /boot/efi: mount point does not exist.
dmesg(1) may have more information after failed mount system call.
The path
/boot/efi exists but is is not mounted - even once the boot process is finished;
/boot/efi is mounted from a
FAT32 partition (512Mb in size). It can be mounted manually (as
root #>
mount /boot/efi )
/boot itself is mounted on a
ext4 partition (768Mb in size), and it appears mounted once the boot process is finished.
Since I am not the only one encountering this issue, there are some questions:
- Is there some wrong assumption in the way the targets have been defined and dinit cannot resolve all of them correctly during boot time ?
- Is there a limitation in the way dinit uses system resources when booting ? I'm thinking "it misses some information or tries too early to obtain some information" kind of thing ...
So far, the file
early-fs-fstab.target was pristine;
Artix was installed not too long ago from the latest ISO available.
Have we reached a limitation / bug in :
a. the way
Artix was configured
or
b. the way
dinit handles the targets at boot time
What can I do to provide additional useful information / help ?
At this point I'd guess that the attempt to mount /boot/efi is made before /boot itself is mounted.
It's just the "mount" command that is failing, so I'm guessing that's the culprit. With a quick experiment I found that listing multiple mount points in /etc/fstab where an earlier mount point requires a later one to be mounted first, does indeed cause "mount -a" to fail. If I reverse the order then it's fine.
Check your /etc/fstab and see if you have /boot/efi listed before /boot itself.
Dinit's just a service manager; it just does what it is told. It's really not trying to have any higher level control of how the system works or what happens where. That's by design; it's supposed to be comprehensible. You configure it by writing service descriptions and it runs the commands in those according to the order required by the dependencies; there's no additional magic.
I'm inclined to say it's a bug in the "mount" utility which should be doing some basic analysis to check whether one mount point requires another to be processed first. But that's arguable, I guess. However, I don't think dinit is doing anything wrong here.
You are right !
The
fstab /boot/efi entry was indeed before the
/boot entry (as they were put by the install procedure).
Having swapped the two lines, on reboot there is no longer an error for the
early-fs-fstab.target service, which now is being displayed as it should:
...
[{+} ] early-root-rw.target
[{+} ] fsck-root
[{+} ] early-fs-fstab.target
[{+} ] fsck
[{+} ] early-console.target
[{+} ] early-keyboard.target
...
Phew, there were some head scratches with this one !
So far,
dinit does indeed do its job; if only it could talk a little bit more ... ;)
Thank you very much for the analysis and the help provided !
Related to the problem discovered: this article (https://superuser.com/questions/768042/linux-sort-etc-fstab-by-the-mountpoint-field) gives us a potential solution: sorting fstab by mountpoint.
Perhaps it is worth considering to include a similar command in the target command line ?
The following command will alphabetically sort all fstab entries:
grep "[[:space:]]/" /etc/fstab | sort -k2,2