Skip to main content ↑ ↓
Topic solved
This topic has been marked as solved and requires no further attention.
Topic: shutdown/reboot hangs when nfs has been mounted (Read 4140 times) previous topic - next topic
0 Members and 4 Guests are viewing this topic.

Re: shutdown/reboot hangs when nfs has been mounted

Reply #15 –
Quote from: mattz7 – on
Thank you. Now I'm thinking maybe the problem is that the network is down before the nfs is unmounted. But this is just my hypothesis...
That could be a possibility. I also made it to where mount-fstab does a soft fatal instead of hard, so if it doesn't do as it should, then it shouldn't crash the rest of the processes.

Edit: to try new version, you will need to enable gremlins in /etc/pacman.conf

Re: shutdown/reboot hangs when nfs has been mounted

Reply #16 –
I edited the file by hand so it looks like this:
Code: [Select]
[main]
@type = oneshot
@version = @vers@
@description = "Mount partition from /etc/fstab file"
@user = ( root )
@depends =( system-fsck )
@options = ( !log )

[start]
@execute =
(
    if { 66-yeller -cdp mount-fstab -1 /dev/console starts... }
    if -nt {

        mount -a

    }
    66-yeller -Fcdp mount-fstab -1 /dev/console crashed!
)

[stop]
@execute =
(
    if { 66-yeller -cdp mount-fstab -1 /dev/console stops... }
    if -nt {

        umount -aR

    }
    66-yeller -Fcdp mount-fstab -1 /dev/console failed!
)

After editing the file, I ran
Code: [Select]
sudo 66-enable -t boot -F boot@system

Unfortunately, this did not resolve the issue :(

I'm beginning to suspect my hypothesis might be true. Do you happen to know how to make sure the network is up while the nfs mount is still being unmounted?

EDIT: BTW, I use iwd for networking.

EDIT2: I'll try to move iwd from default to boot and see what happens.
 

Re: shutdown/reboot hangs when nfs has been mounted

Reply #17 –
Trying to order while shutting down is not really possible. At least not with the way s6-rc is designed. Assuming you know what service exactly hangs, a better approach would be to use the finish script to clean it up and make sure it's gone.

Re: shutdown/reboot hangs when nfs has been mounted

Reply #18 –
Quote from: Dudemanguy – on
Trying to order while shutting down is not really possible. At least not with the way s6-rc is designed. Assuming you know what service exactly hangs, a better approach would be to use the finish script to clean it up and make sure it's gone.
So what are you suggesting I should do?

BTW, I'm getting an fstab fatal error now when shutting down (see the picture attached).

Re: shutdown/reboot hangs when nfs has been mounted

Reply #19 –
Quote from: Dudemanguy – on
Trying to order while shutting down is not really possible. At least not with the way s6-rc is designed. Assuming you know what service exactly hangs, a better approach would be to use the finish script to clean it up and make sure it's gone.
With 66, there is no finish script.

@mattz7 The only thing else I know to try is to edit /usr/bin/reboot and add -f flag.
You can edit /usr/bin/shutdown to do:
66-hpr -pf
I would make a backup of the shutdown script before doing so. The -f flag forces shutdown and only syncs instead of going through normal reboot/shutdown process, so it should be near instant action

Edit: Before doing this, let me try something else...

Re: shutdown/reboot hangs when nfs has been mounted

Reply #20 –
Quote from: ndowens – on
With 66, there is no finish script.

@mattz7 The only thing else I know to try is to edit /usr/bin/reboot and add -f flag.
You can edit /usr/bin/shutdown to do:
66-hpr -pf
I would make a backup of the shutdown script before doing so. The -f flag forces shutdown and only syncs instead of going through normal reboot/shutdown process, so it should be near instant action

Edit: First try Edit 2
Edit 2 : Try changing your rc.shutdown script in /etc/66 to: https://termbin.com/fpmt

Re: shutdown/reboot hangs when nfs has been mounted

Reply #21 –
Quote from: mattz7 – on
So what are you suggesting I should do?

BTW, I'm getting an fstab fatal error now when shutting down (see the picture attached).
My idea wouldn't work actually. The finish script waits on the run to finish so if run hangs forever, then you will never execute finish. Hopefully ndowens idea to use -f works.

Re: shutdown/reboot hangs when nfs has been mounted

Reply #22 –
Ok, Before trying the above ideas. I am about to push a change that adds a timeout-kill to mount-fstab, so if it hangs past 3 secs, then it kills it.

Edit: I updated suite66 with a patch that excludes unmounting nfs, so this may avoid hanging. ( Currently in gremlins repo)

Re: shutdown/reboot hangs when nfs has been mounted

Reply #23 –
Just to recap some IRC discussion, suite66 works like s6-linux-init (the code is almost the same) in that its shutdown command umounts all the partitions for you. Presumably, you are a victim of the classic "nfs umounting hangs forever" bug in which the workaround is to use the -f flag. Neither s6-linux-init nor suite66 ever used the force option when umounting... and they probably shouldn't. That's hacky. Instead, the thought was to just skip umounting any filesystems if they are an NFS type. It's a one line code change. I think that should solve your issue and there's not any harm in not umounting NFS (it's not physically there). If that works, we'll hopefully get this upstreamed soon.

Re: shutdown/reboot hangs when nfs has been mounted

Reply #24 –
Thank you guys for all of your work. I do appreciate it. ;)  I might try the gremlins package later but if it doesn't work I'm going to go back to good-old-OpenRC.


EDIT: So I enabled the gremlins repo and updated the system.

Now, when I boot the machine, it hangs at "local-dmesg: info: starts..."
Also, there is a "mount-fstab: fatal: failed!" error. But this time around I can at least ctrl-alt-del the system.

Code: [Select]
# 66-intree -z
Name         : boot
Initialized  : no
Enabled      : no
Starts after : None
Current      : no
Allowed      : root
Symlinks     : svc->source db->source
Contents     : tty-earlier@tty12 system-hostname mount-run populate-run mount-tmp populate-tmp mount-proc mount-sys populate-sys mount-dev mount-pts mount-shm
               populate-dev mount-cgroups 00 system-hwclock modules-kernel system-random modules-system system-sysctl udevd-log udevd udevadm system-fontnkey
               all-Mount system-fsck mount-fstab all-System mount-rw local-authfiles local-loop local-sethostname local-time local-tmpfiles local-dmesg
               local-locale mount-netfs all-Local all-Runtime All tty-rc@tty1 tty-rc@tty2 tty-rc@tty3 tty-rc@tty4 tty-rc@tty5 boot@system

Name         : default
Initialized  : no
Enabled      : yes
Starts after : None
Current      : yes
Allowed      : root
Symlinks     : svc->backup db->backup
Contents     : cronie-log cronie dbus-log dbus bluetoothd-log bluetoothd syslog-ng-log syslog-ng iwd-log iwd ntpd-log ntpd

Re: shutdown/reboot hangs when nfs has been mounted

Reply #25 –
I have no further idea on how to solve this :\

Re: shutdown/reboot hangs when nfs has been mounted

Reply #26 –
Quote from: ndowens – on
I have no further idea on how to solve this :\
OK, I understand, but do you know how to troubleshoot why there is a "mount-netfs: fatal:crashed!" error message?

Let me just remind you that I have a noauto,nofail option in fstab so there shouldn't be a problem with boottime.

Re: shutdown/reboot hangs when nfs has been mounted

Reply #27 –
Let me update the mount script later when I have time, I have a kid to get ready for school and other stuff to do; and do what s6 does. Which will remove mount-netfs

Re: shutdown/reboot hangs when nfs has been mounted

Reply #28 –
Quote from: ndowens – on
Let me update the mount script later when I have time, I have a kid to get ready for school and other stuff to do; and do what s6 does. Which will remove mount-netfs
Sure, thank you for your time so far :)

EDIT:
Quote from: ndowens – on
66-env -t boot -e $EDITOR boot@system and change MNT_NETFS=!yes to MNT_NETFS=!no (where $EDITOR is something like vim ) as root and also run as root: 66-enable -t boot -F boot@system
This renders the system bootable again. Yey! :D

EDIT2: Now the system hangs at a different message when shutting it down while having the nfs share mounted (see the picture attached)
Crucially, now mount-fstab does not report any problems, yet the system hangs.

EDIT3:
Quote from: ndowens – on
@mattz7 The only thing else I know to try is to edit /usr/bin/reboot and add -f flag.
You can edit /usr/bin/shutdown to do:
66-hpr -pf
This indeed works (the screen goes blank instantly), but I think it's a hack rather than a solution.

IMPORTANT INFORMATION
I checked if I would have the same problem with an ethernet connection as opposed to wifi, and the problem disappeared when I used
Code: [Select]
dhcpcd eth0

So I think, iwd might be at fault here or wifi in general.

Apparently, wifi is at fault since switching to NetworkManager was to no avail.

Re: shutdown/reboot hangs when nfs has been mounted

Reply #29 –
Ok I've removed mount-netfs and changed mount-fstab to only mount all that doesn't have noauto or no_netdev set in mount options. HOPEFULLY this will not mount nfs on boot and no cause issue

Edit: new scripts version is in gremlins