Artix Linux Forum

Init systems => suite66-archived => Topic started by: mattz7 on 19 September 2021, 21:30:39

Title: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 19 September 2021, 21:30:39
Hi everyone!  :)

I just switched from OpenRC to suite66 and I encountered a cumbersome problem. When I mount an NFS fstab entry and reboot/shutdown the computer, it hangs (see the picture attached) and I have to force shutdown the system. Conversely, when I don't mount the NFS directory, the shutdown/reboot process runs without difficulties.

Let me just mention that I had the same problem with OpenRC. It could, however, be simply resolved by adding
Code: [Select]
rc_need="!netmount"
to /etc/rc.conf 

My fstab includes:
Code: [Select]
srv:/                   /nfs/srv               nfs4    rw,nofail,noauto,user       0       0

I would appreciate any suggestions on how to solve this problem.

Matt
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 19 September 2021, 21:51:02
For testing, edit /etc/66/module/boot@/service/mount/mount-netfs and change the last 66-yeller -f to -F and run 66-enable -t boot -F boot@system as root and see if this helps or not. The -f is for hard fatal and will stop the system, -F is fatal but doesn't crash boot.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 19 September 2021, 21:58:00
I did what you instructed me to but unfortunately it didn't help. It still hangs and I can't even ctrl-alt-del. I hope this can be somehow resolved as I wanted to try out something different than OpenRC which I've been accustomed to. 
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 19 September 2021, 22:08:35
Are you using nfs-suite66 package? Also try 66-env -t boot -e $EDITOR boot@system and change MNT_NETFS=!yes to MNT_NETFS=!no (where $EDITOR is something like vim ) as root and also run as root: 66-enable -t boot -F boot@system ; and see if that helps, though it shouldn't cause an effect as it only looks for things like nonfs4 and others to mount
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 19 September 2021, 22:11:59
Quote
Are you using nfs-suite66 package?

No I don't. What should I do to configure this init script besides installing nfs-utils-suite66?
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 19 September 2021, 22:14:30
No I don't. What should I do to configure this init script besides installing nfs-utils-suite66?
Did you do the 66-env part and 66-enable part of message above?
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 19 September 2021, 22:15:38
I will in a minute. I will edit my reply after I run the commands and reboot the system.

EDIT: Unfortunately, running the above commands did not help.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 19 September 2021, 22:21:29
I will in a minute. I will edit my reply after I run the commands and reboot the system.

EDIT: Unfortunately, running the above commands did not help.
Do you have nfs-utils installed?
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 19 September 2021, 22:23:05
Yes, I have nfs-utils installed and I just installed nfs-utils-suite66 but I'm a complete newbie to suite66 so I don't know how to set it all up.

EDIT: I can see that nfs-utils-suite66 is for server, not client so I guess I don't need it, right?
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 19 September 2021, 22:27:21
Yes, I have nfs-utils installed and I just installed nfs-utils-suite66 but I'm a complete newbie to suite66 so I don't know how to set it all up.

EDIT: I can see that nfs-utils-suite66 is for server, not client so I guess I don't need it, right?
Correct, I don't use NFS. In a vm, I used what you had in fstab and had no issue with startup.  Try with the NFS line commented out and see
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 19 September 2021, 22:31:25
I commented out the nfs line in fstab and unmounted the nfs drive. I could reboot without a hang.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 19 September 2021, 22:32:04
That usually means the fstab isn't setup correctly. If it is wrong, then it will hang
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 19 September 2021, 22:33:35
Quote
In a vm, I used what you had in fstab and had no issue with startup

You said "startup" but I have a problem with shutdown, not boot. I don't know if we've been on the same page this whole thread :P

EDIT: TBH I don't see anything specific about my nfs setup. I believe other users might experience the same problem. As for the fstab part, the entry seems to be correct because I can mount the nfs share even as a user. 
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 19 September 2021, 23:17:01
I added a [stop] part to the mount-fstab script in suite66-scripts-1.7.1.1-1 , currently it in que to build. Once it is built and mirrors syncs, you will get new version. I am unsure this will help or not, but I am hoping it is hanging on fstab part and it will help you.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 19 September 2021, 23:20:00
I added a [stop] part to the mount-fstab script in suite66-scripts-1.7.1.1-1 , currently it in que to build. Once it is built and mirrors syncs, you will get new version. I am unsure this will help or not, but I am hoping it is hanging on fstab part and it will help you.
Thank you. Now I'm thinking maybe the problem is that the network is down before the nfs is unmounted. But this is just my hypothesis...
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 19 September 2021, 23:24:17
Thank you. Now I'm thinking maybe the problem is that the network is down before the nfs is unmounted. But this is just my hypothesis...
That could be a possibility. I also made it to where mount-fstab does a soft fatal instead of hard, so if it doesn't do as it should, then it shouldn't crash the rest of the processes.

Edit: to try new version, you will need to enable gremlins in /etc/pacman.conf
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 19 September 2021, 23:31:29
I edited the file by hand so it looks like this:
Code: [Select]
[main]
@type = oneshot
@version = @vers@
@description = "Mount partition from /etc/fstab file"
@user = ( root )
@depends =( system-fsck )
@options = ( !log )

[start]
@execute =
(
    if { 66-yeller -cdp mount-fstab -1 /dev/console starts... }
    if -nt {

        mount -a

    }
    66-yeller -Fcdp mount-fstab -1 /dev/console crashed!
)

[stop]
@execute =
(
    if { 66-yeller -cdp mount-fstab -1 /dev/console stops... }
    if -nt {

        umount -aR

    }
    66-yeller -Fcdp mount-fstab -1 /dev/console failed!
)

After editing the file, I ran
Code: [Select]
sudo 66-enable -t boot -F boot@system

Unfortunately, this did not resolve the issue :(

I'm beginning to suspect my hypothesis might be true. Do you happen to know how to make sure the network is up while the nfs mount is still being unmounted?

EDIT: BTW, I use iwd for networking.

EDIT2: I'll try to move iwd from default to boot and see what happens.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: Dudemanguy on 19 September 2021, 23:56:21
Trying to order while shutting down is not really possible. At least not with the way s6-rc is designed. Assuming you know what service exactly hangs, a better approach would be to use the finish script to clean it up and make sure it's gone.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 20 September 2021, 00:01:11
Trying to order while shutting down is not really possible. At least not with the way s6-rc is designed. Assuming you know what service exactly hangs, a better approach would be to use the finish script to clean it up and make sure it's gone.
So what are you suggesting I should do?

BTW, I'm getting an fstab fatal error now when shutting down (see the picture attached).
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 20 September 2021, 00:20:56
Trying to order while shutting down is not really possible. At least not with the way s6-rc is designed. Assuming you know what service exactly hangs, a better approach would be to use the finish script to clean it up and make sure it's gone.
With 66, there is no finish script.

@mattz7 The only thing else I know to try is to edit /usr/bin/reboot and add -f flag.
You can edit /usr/bin/shutdown to do:
66-hpr -pf
I would make a backup of the shutdown script before doing so. The -f flag forces shutdown and only syncs instead of going through normal reboot/shutdown process, so it should be near instant action

Edit: Before doing this, let me try something else...
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 20 September 2021, 00:33:23
With 66, there is no finish script.

@mattz7 The only thing else I know to try is to edit /usr/bin/reboot and add -f flag.
You can edit /usr/bin/shutdown to do:
66-hpr -pf
I would make a backup of the shutdown script before doing so. The -f flag forces shutdown and only syncs instead of going through normal reboot/shutdown process, so it should be near instant action

Edit: First try Edit 2
Edit 2 : Try changing your rc.shutdown script in /etc/66 to: https://termbin.com/fpmt
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: Dudemanguy on 20 September 2021, 01:15:46
So what are you suggesting I should do?

BTW, I'm getting an fstab fatal error now when shutting down (see the picture attached).
My idea wouldn't work actually. The finish script waits on the run to finish so if run hangs forever, then you will never execute finish. Hopefully ndowens idea to use -f works.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 20 September 2021, 01:24:20
Ok, Before trying the above ideas. I am about to push a change that adds a timeout-kill to mount-fstab, so if it hangs past 3 secs, then it kills it.

Edit: I updated suite66 with a patch that excludes unmounting nfs, so this may avoid hanging. ( Currently in gremlins repo)
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: Dudemanguy on 20 September 2021, 03:06:56
Just to recap some IRC discussion, suite66 works like s6-linux-init (the code is almost the same) in that its shutdown command umounts all the partitions for you. Presumably, you are a victim of the classic "nfs umounting hangs forever" bug in which the workaround is to use the -f flag. Neither s6-linux-init nor suite66 ever used the force option when umounting... and they probably shouldn't. That's hacky. Instead, the thought was to just skip umounting any filesystems if they are an NFS type. It's a one line code change. I think that should solve your issue and there's not any harm in not umounting NFS (it's not physically there). If that works, we'll hopefully get this upstreamed soon.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 20 September 2021, 12:44:33
Thank you guys for all of your work. I do appreciate it. ;)  I might try the gremlins package later but if it doesn't work I'm going to go back to good-old-OpenRC.


EDIT: So I enabled the gremlins repo and updated the system.

Now, when I boot the machine, it hangs at "local-dmesg: info: starts..."
Also, there is a "mount-fstab: fatal: failed!" error. But this time around I can at least ctrl-alt-del the system.

Code: [Select]
# 66-intree -z
Name         : boot
Initialized  : no
Enabled      : no
Starts after : None
Current      : no
Allowed      : root
Symlinks     : svc->source db->source
Contents     : tty-earlier@tty12 system-hostname mount-run populate-run mount-tmp populate-tmp mount-proc mount-sys populate-sys mount-dev mount-pts mount-shm
               populate-dev mount-cgroups 00 system-hwclock modules-kernel system-random modules-system system-sysctl udevd-log udevd udevadm system-fontnkey
               all-Mount system-fsck mount-fstab all-System mount-rw local-authfiles local-loop local-sethostname local-time local-tmpfiles local-dmesg
               local-locale mount-netfs all-Local all-Runtime All tty-rc@tty1 tty-rc@tty2 tty-rc@tty3 tty-rc@tty4 tty-rc@tty5 boot@system

Name         : default
Initialized  : no
Enabled      : yes
Starts after : None
Current      : yes
Allowed      : root
Symlinks     : svc->backup db->backup
Contents     : cronie-log cronie dbus-log dbus bluetoothd-log bluetoothd syslog-ng-log syslog-ng iwd-log iwd ntpd-log ntpd
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 20 September 2021, 13:18:02
I have no further idea on how to solve this :\
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 20 September 2021, 13:19:51
I have no further idea on how to solve this :\
OK, I understand, but do you know how to troubleshoot why there is a "mount-netfs: fatal:crashed!" error message?

Let me just remind you that I have a noauto,nofail option in fstab so there shouldn't be a problem with boottime.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 20 September 2021, 13:25:19
Let me update the mount script later when I have time, I have a kid to get ready for school and other stuff to do; and do what s6 does. Which will remove mount-netfs
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 20 September 2021, 13:29:08
Let me update the mount script later when I have time, I have a kid to get ready for school and other stuff to do; and do what s6 does. Which will remove mount-netfs
Sure, thank you for your time so far :)

EDIT:
66-env -t boot -e $EDITOR boot@system and change MNT_NETFS=!yes to MNT_NETFS=!no (where $EDITOR is something like vim ) as root and also run as root: 66-enable -t boot -F boot@system
This renders the system bootable again. Yey! :D

EDIT2: Now the system hangs at a different message when shutting it down while having the nfs share mounted (see the picture attached)
Crucially, now mount-fstab does not report any problems, yet the system hangs.

EDIT3:
@mattz7 The only thing else I know to try is to edit /usr/bin/reboot and add -f flag.
You can edit /usr/bin/shutdown to do:
66-hpr -pf
This indeed works (the screen goes blank instantly), but I think it's a hack rather than a solution.

IMPORTANT INFORMATION
I checked if I would have the same problem with an ethernet connection as opposed to wifi, and the problem disappeared when I used
Code: [Select]
dhcpcd eth0

So I think, iwd might be at fault here or wifi in general.

Apparently, wifi is at fault since switching to NetworkManager was to no avail.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 20 September 2021, 14:56:20
Ok I've removed mount-netfs and changed mount-fstab to only mount all that doesn't have noauto or no_netdev set in mount options. HOPEFULLY this will not mount nfs on boot and no cause issue

Edit: new scripts version is in gremlins
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 20 September 2021, 15:01:12
Ok I've removed mount-netfs and changed mount-fstab to only mount all that doesn't have noauto or no_netdev set in mount options. HOPEFULLY this will not mount nfs on boot and no cause issue

Edit: new scripts version is in gremlins
But the NFS share was not mounted at boot. I had to mount it manually.

Did you manage to read IMPORTANT INFORMATION above? ;) ;) Apparently there is something wrong with the way wifi is handled at shutdown.

Anyhow, I'm waiting for the mirrors to sync and I'm going to try out your new revision from gremlins. Thank you! :)
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: Dudemanguy on 20 September 2021, 15:02:13
To be clear, if you manually umount the nfs partition before shutting down, does the system still hang?
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 20 September 2021, 15:02:58
To be clear, if you manually umount the nfs partition before shutting down, does the system still hang?
No. It doesn't hang then. I can even umount it without the -f option. 
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 20 September 2021, 15:04:40
But the NFS share was not mounted at boot. I had to mount it manually.

Did you manage to read IMPORTANT INFORMATION above? ;) ;) Apparently there is something wrong with the way wifi is handled at shutdown.

Anyhow, I'm waiting for the mirrors to sync and I'm going to try out your new revision from gremlins. Thank you! :)
Hmm for testing, what happens if you stop wifi then shutdown?
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 20 September 2021, 15:08:41
The system hangs when I poweroff after I do
Code: [Select]
66-stop iwd
  and don't umount the nfs share beforehand.  Only now, I receive one message at shutdown, namely
Code: [Select]
*** WARNING ** The system is going down NOW
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: Dudemanguy on 20 September 2021, 15:10:59
Thanks for the all testing! So that does confirm it is the nfs umounting in suite66 causing the hanging. That must mean our patch was not correct. I personally believe this should be handled in the umount step during shutdown in suite66, so I will go look at the code again.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 20 September 2021, 15:11:56
The system hangs when I poweroff after I do
Code: [Select]
66-stop iwd
  and don't umount the nfs share beforehand.  Only now, I receive one message at shutdown, namely
Code: [Select]
*** WARNING ** The system is going down NOW
Well darn, I usually only get that message myself anyway. Going to look and think further and see if there is something else that can be done..
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 20 September 2021, 15:13:14
Thanks for the all testing! So that does confirm it is the nfs umounting in suite66 causing the hanging. That must mean our patch was not correct. I personally believe this should be handled in the umount step during shutdown in suite66, so I will go look at the code again.
Thank you Sir! I'm looking forward to you reviewing the code! :)

IMPORTANT INFORMATION
So, apparently modifying /etc/66/service/iwd and running 66-enable -t default -F iwd solves the issue:
Code: [Select]
[main]
@type = longrun
@description = "iNet Wireless Daemon"
@version = 0.0.1
@user = ( root )

[start]
@execute = ( /usr/lib/iwd/iwd )

[stop]
@execute = ( umount /nfs/srv )
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: Dudemanguy on 20 September 2021, 15:51:01
@ndowens: discovered a trivial error (darn C) in the patch so the nfs partition was not being skipped. It's been corrected. The 'suite66-0.6.1.3-2.3' package is in gremlins now so if you could give that a try @mattz7.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 20 September 2021, 15:53:50
@ndowens: discovered a trivial error (darn C) in the patch so the nfs partition was not being skipped. It's been corrected. The 'suite66-0.6.1.3-2.3' package is in gremlins now so if you could give that a try @mattz7.

Sure I'll give it a try once the mirrors sync, but I also discovered that forcing iwd to umount the nfs share before it stops also solves the issue.

Thank you!
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: Dudemanguy on 20 September 2021, 15:59:28
Sure I'll give it a try once the mirrors sync, but I also discovered that forcing iwd to umount the nfs share before it stops also solves the issue.
Yeah that would also work. It'd be nice if umount would never hang however. If you could temporarily get rid of that [stop] part when testing, that would be great. Thanks for the testing so far!
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 20 September 2021, 16:02:45
Sure I'll give it a try once the mirrors sync, but I also discovered that forcing iwd to umount the nfs share before it stops also solves the issue.

Thank you!
Of course, I did
Code: [Select]
pacman -S iwd-suite66
66-enable -t default -F iwd

BTW, do you know which mirror has the 2.3 version you pushed?

EDIT: OK, got it. The last mirror from /etc/pacman.d/mirrorlist.

I will test that in a minute! :)

EDIT2: I had something that read "fstab failed" message, but the system shut down! THANK YOU! You da man! :D

I will mark the thread as solved, but can you further help me troubleshoot the message about fstab?  I will power on the computer and capture a picture of the message.


EDIT3: DAMN! It does not work. Maybe the last time around the iwd service kicked in with the stop procedure.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 20 September 2021, 16:12:33
Of course, I did
Code: [Select]
pacman -S iwd-suite66
66-enable -t default -F iwd

BTW, do you know which mirror has the 2.3 version you pushed?

EDIT: OK, got it. The last mirror from /etc/pacman.d/mirrorlist.

I will test that in a minute! :)

EDIT2: I had something that read "fstab failed" message, but the system shut down! THANK YOU! You da man! :D

I will mark the thread as solved, but can you further help me troubleshoot the message about fstab?  I will power on the computer and capture a picture of the message.


EDIT3: DAMN! It does not work. Maybe the last time around the iwd service kicked in with the stop procedure.
Well darn, OK. Came up with a solution to the idea you did to get it to work. SO this means, I will be editing network-related services such as iwd and others to umount in [stop]

Edit: for the mount-fstab thing, only happens during shutdown? If so it could be ignored as it shouldn't cause an issue. The command it runs shouldn't cause an issue
Code: [Select]
mount -a -O noauto,no_netdev
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 20 September 2021, 16:14:23
OK. I want to thank you @ndowens and @Dudemanguy for your time.

I can't wait to test out iwd-suite66 once it's there!

Hope y'all enjoying your day wherever you are. Greetings from Poland! :)

EDIT:

Edit: for the mount-fstab thing, only happens during shutdown? If so it could be ignored as it shouldn't cause an issue. The command it runs shouldn't cause an issue
Code: [Select]
mount -a -O noauto,no_netdev
I had this message the first time around when shutdown went well. I'll test it once again with the modified iwd script.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: Dudemanguy on 20 September 2021, 16:18:37
Dang that's a bummer that did not work. Maybe there's more to nfs mounts than I expected. In the meantime, we'll try to adjust the stop/finish scripts which is worth it anyway. Could you post the output of cat /proc/mounts with your nfs mounted? I'm not sure what nfs mounts look like in that file (so we know what to grep for exactly).
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 20 September 2021, 16:25:01
OK. I want to thank you @ndowens and @Dudemanguy for your time.

I can't wait to test out iwd-suite66 once it's there!

Hope y'all enjoying your day wherever you are. Greetings from Poland! :)

EDIT:
I had this message the first time around when shutdown went well. I'll test it once again with the modified iwd script.

I just pushed new iwd-suite66, hopefully this will work. May need to do 66-enable -F iwd once you get it
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 20 September 2021, 16:26:07
Just to recap, this is the SOLUTION.

/etc/66/service/iwd
Code: [Select]
[main]
@type = longrun
@description = "iNet Wireless Daemon"
@version = 0.0.1
@user = ( root )

[start]
@execute = ( /usr/lib/iwd/iwd )

[stop]
@execute = ( umount -a -t nfs  )

after that you run
Code: [Select]
# 66-enable -t default -F iwd

As for the fstab error, now I'm not getting it, so it must've been a one-time-thing.

Dang that's a bummer that did not work. Maybe there's more to nfs mounts than I expected. In the meantime, we'll try to adjust the stop/finish scripts which is worth it anyway. Could you post the output of cat /proc/mounts with your nfs mounted? I'm not sure what nfs mounts look like in that file (so we know what to grep for exactly).

/proc/mounts
Code: [Select]
srv:/ /nfs/srv nfs4 rw,nosuid,nodev,noexec,relatime,vers=4.2,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.1.52,local_lock=none,addr=192.168.1.115 0 0

Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 20 September 2021, 16:29:54
Rev 3 of iwd-suite66 uses the [stop] how you had it, so should work
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 20 September 2021, 16:33:14
I'm waiting for the mirrors to sync and once everything works, I'm going to mark this thread as solved :)

EDIT: DAMN IT. Now the system hangs. I don't know what happened since the last time. Let me do some more testing.

EDIT2: @ndowens so apparently, I was trying to
Code: [Select]
umount -t nfs -a
while I should be doing
Code: [Select]
umount -t nfs4,nfs -a

The latter option appears to work. Please account for it in the iwd-suite66 package.

BTW, I downgraded suite66 from gremlins to system, and did  MNT_NETFS=!no in 66-env -t boot -e $EDITOR boot@system

Thank you! :)
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 20 September 2021, 17:00:53
I'm waiting for the mirrors to sync and once everything works, I'm going to mark this thread as solved :)

EDIT: DAMN IT. Now the system hangs. I don't know what happened since the last time. Let me do some more testing.

EDIT2: @ndowens so apparently, I was trying to
Code: [Select]
umount -t nfs -a
while I should be doing
Code: [Select]
umount -t nfs4,nfs -a

The latter option appears to work. Please account for it in the iwd-suite66 package.

BTW, I downgraded suite66 from gremlins to system, and did 
Code: [Select]
MNT_NETFS=!yes
in 66-env -t boot -e $EDITOR boot@system

Thank you! :)
Thanks for clarifying, I removed suite66 from testing and will re-adjust the stop scripts
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 21 September 2021, 14:31:38
Hi guys!

I'm sorry to say this but the above solution does not work in my case, at least. It works when you manually run
Code: [Select]
66-stop iwd
but it doesn't work when you shutdown the system when an nfs share is still mounted.

I'm sorry to have misled you on this, but apparently I did stop iwd when testing it for the first time  :(  

I'll do more troubleshooting and I'll play around with the iwd script more, but still I would appreciate some more opinions on this from the more experienced suite66 users.

Best,
Matt
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 21 September 2021, 16:10:12
@mattz7 can you edit /etc/66/service/iwd and change umount -a -t .. to umount -f -a -t ... and 66-enable -F iwd and see if that helps
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 21 September 2021, 16:15:57
@mattz7 can you edit /etc/66/service/iwd and change umount -a -t .. to umount -f -a -t ... and 66-enable -F iwd and see if that helps

Thanks for the reply :)

Funny thing is it worked now, but let me double check, boot up the system once again, mount the nfs and shutdown again :P

EDIT:
OK, the second time it worked, too.

I'll edit /etc/66/services/iwd and delete the -f option, do 66-enable -F iwd and see if it doesn't work. If it doesn't work, it means you nailed it! :)

EDIT2:

So I reverted the changes in iwd, re-enabled the service, mounted nfs and shutdown the computer. Only this time, it worked too.

Sorry for all this misleading information, but seriously I have such a mindf*** right now. I don't know what has changed since the last time, but apparently now it works.


EDIT3: OK, not it hangs, so I guess I know what happened. The change needs a reboot for it to work. So my hypothesis on the spot is, the -f option really makes a difference.

What I'll do is I'll add the -f option again, re-enable iwd, power off the computer, turn it back on, mount the nfs share and power it off. This'll be the ultimate test.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 21 September 2021, 16:24:59
Thanks for the reply :)

Funny thing is it worked now, but let me double check, boot up the system once again, mount the nfs and shutdown again :P

EDIT:
OK, the second time it worked, too.

I'll edit /etc/66/services/iwd and delete the -f option, do 66-enable -F iwd and see if it doesn't work. If it doesn't work, it means you nailed it! :)

EDIT2:

So I reverted the changes in iwd, re-enabled the service, mounted nfs and shutdown the computer. Only this time, it worked too.

Sorry for all this misleading information, but seriously I have such a mindf*** right now. I don't know what has changed since the last time, but apparently now it works.
As long as it works :)
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 21 September 2021, 16:28:23
BTW, do you know why the -f option would make a difference here? Because I don't get it.

I'll check the -f option a couple more times.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 21 September 2021, 16:30:13
BTW, do you know why the -f option would make a difference here? Because I don't get it.

I'll check the -f option a couple more times.
It is suppose to help a bit with hangs with nfs unmounts and such, but not guaranteed to do so. But I am adding the change to the scripts, to maybe help avoid this even more in the future.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 21 September 2021, 16:35:07
Great news. I'm looking forward to you pushing the change to the repos.

Now I'm thinking, maybe the [stop] part is executed after the IWD process is killed. But then again, running 66-stop iwd manually unmounted the share even though it didn't have the -f option. I'm confused...

As for the testing, I checked the -f option after a reboot, and now shutting down works like a charm after mounting the nfs share.

A big thank you to y'all! :) 

EDIT:

@ndowens @Dudemanguy Alas, there is one more hitch. When shutting down from my KDE plasma desktop, so I believe through elogind, the shutdown process hangs.

When shutting down from TTY, the shutdown process is smooth like butter.

EDIT2:

Running
Code: [Select]
dbus-send --system --print-reply --dest=org.freedesktop.login1 /org/freedesktop/login1 org.freedesktop.login1.Manager.PowerOff boolean:false
also works. I need to do more research and testing and I'll get back to you with my findings if I get them.

EDIT3: So running
Code: [Select]
loginctl poweroff
also causes the system to hang while shutting down.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: Dudemanguy on 21 September 2021, 19:05:47
That's very odd. I always presumed that loginctl poweroff would simply just call the poweroff binary on your machine. So there shouldn't be any difference. Somehow something must be different with that call.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 21 September 2021, 20:23:03
I always just use terminal to reboot or shutdown. Where did you use the dbus-send cmd?

Side note, I just pushed update elogind-suite66 package that cleans up a bit of it, though I doubt it helps in this situation. I updated mount-cgroups to do partly what elogind script was doing.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: strajder on 21 September 2021, 21:53:46
EDIT3: So running
Code: [Select]
loginctl poweroff
also causes the system to hang while shutting down.
It seems that loginctl currently calls shutdown -h, not physically powering down the computer. You can use shutdown -p now to do that. I changed my sdmenu script (https://git.sr.ht/~strahinja/dwm/tree/master/item/sdmenu) (which I use as a shutdown menu) in my fork of dwm to do that.

I think the goal should be to eventually move away from loginctl as part of systemd anyway.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: Dudemanguy on 21 September 2021, 22:13:42
That's extremely unintuitive to say the least. Poweroff should be... well poweroff. Nice find.
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 21 September 2021, 22:46:03
I always just use terminal to reboot or shutdown. Where did you use the dbus-send cmd?

Side note, I just pushed update elogind-suite66 package that cleans up a bit of it, though I doubt it helps in this situation. I updated mount-cgroups to do partly what elogind script was doing.
I used it in yakuake in KDE plasma ::)
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 21 September 2021, 22:49:57
I used it in yakuake in KDE plasma ::)
As mentioned above, seems to be an elogind issue. I have never tried shutdown from plasma's menu :) . Usually I am in terminal doing w/e so I just do it there
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: mattz7 on 21 September 2021, 22:59:35
As mentioned above, seems to be an elogind issue. I have never tried shutdown from plasma's menu :) . Usually I am in terminal doing w/e so I just do it there
Sure, I've removed myself from the power group on the suite66 system so that I don't accidentally shutdown from within a plasma session.

But going back to the subject matter of this thread, I had a single time when I powered off my computer from TTY and it wouldn't shut down but hung, so I reinstalled Artix and now powering off worked without hassle. To be sure whether it works all the time I'll devote much more time to testing it.

BTW this time around I had to switch off mount-cgroups as the system wouldn't boot up. I'm suspecting either this option used to be turned off by default or something changed in the way the script was handled by suite66. 
Title: Re: shutdown/reboot hangs when nfs has been mounted
Post by: ndowens on 22 September 2021, 01:11:31
I changed mount-cgroups, though my system doesn't hang nor does it give me an error, odd... I did have it executing a shell script, but I put what the script was doing into the script itself

Edit: Here is the updated script https://termbin.com/q9vr6 I don't see any typo or anything to cause issue.

Edit2: OK vm is now crashing, so Got to figure out what caused it.

Edit3: Ok reverted the change I made, so next scripts update is fine.