Skip to main content ↑ ↓
Topic solved
This topic has been marked as solved and requires no further attention.
Topic: zaplocked installation recovery SELF-RESOLVED? (Read 1879 times) previous topic - next topic
0 Members and 3 Guests are viewing this topic.

zaplocked installation recovery SELF-RESOLVED?

I cannot for the life of me find the draft posting I started this with last night!  How do I get to it???

I had a filesystem corruption incident destroying the directory on a data drive from which I recovered reasonably well with testdisk but my Artix Plasma system ended up with an only partly populated /boot folder

memtest86+
amd-ucode.img
intel-ucode.img

It started with a lockup forcing a hard reset and whatever happened caused a journal replay on reboot on the data drive also hosting some of the home directories. I 'think' I made a mistake and rebooted to another system whereas in retrospect it might have been smarter to reboot Artix. Can't remember, anyway, I can't really tell what else is missing now. On this reboot I didn't suspect anything, yet, I did noticed the journal replay spooling the drive, but only found out that something was wrong when I could not log-in because my home folder was on the zapped data drive.Tried some kernel transplants to no avail (those used to work at one time). Long story short, I finally  chrooted from another system and did an update which seemed promising because it included a new kernel so the /boot folder got replenished as it were. Exited chroot and redid the grub thing, thinking I had it made. But then on reboot I got

"ERROR: Root device mounted succesfully, but /sbin/init does not exist.
 Bailing out, you are on yoiur own, Good luck."

Is there a reasonably easy way to salvage the installation? I'm no guru but can chew may way through a 2do list. 

TIA


Addendum 2022-06-30-2125edt:
===========================
To recap

I had a Linux system running on a 1tb ssd where other distros alo reside. This Linux system locked up (this happens not infrequently). I had to do a hard reset. On the next boot to a Linux system I noticed a 4tb DATA disk journal replay. This also happens not infrequently and maybe mostly after such lockups but IO cannot confirm that at this point. As this next boot got to the login prompt I could not log in, reason unknown, but experience told me that maybe my home folder was not accessible. So I logged in as a user whose home folderr in NOT on the data disk and soon confirmed that the data disk was gone, probably directory corruption.

Spent some time recovering the 4tb data disk using the testdisk recovery uti. My next boot which would have been to Artix bombed totally, on inspection from another system I observed the incomplete /boot folder stated in the OP above. Now I had to deal with another recovery to do.

I chrooted into the Artix system and did an update which also included a new kernel so that the /boot folder got repopulated. Upon trying top boot Artix I got a boort failure again, reference the m,issing /sbin/init.

A day goes by, I download the lates iso with the grim prospect of posssible having to reinstall.  But 1st I set out to try chroot again as per advice received in this thread. Upon seeing the boot menu I absent-mindedly hit the Artix entry thinking "well this ain't going to work because to chroot I need to be in another system to do it from.  In fact I almost aborted the boot to start all over  ..but somehow I didn't. Well, what do you know, Artix booted no problem, just I did another update and for all I know everything is back to normal.

Go figure...

Thank you all, I will look into this all the same:
https://forum.artixlinux.org/index.php/topic,4126.msg27091/topicseen.html#msg27091

Who, has loved us more?

Re: zaplocked installation recovery

Reply #1 –
if your home directory is on a separate drive than your root directory and you can't seem to recover your root directory you could always try reinstalling and just be sure to only format your root directory and NOT your home.

BUT before you do that you could try booting a usb/pendrive again with artix on it and try some filesystem recovery tools appropriate for your file system.  what format are you using?  ext4?  that would be ideal and probably easiest.  btrfs has its own tools also...
Cat Herders of Linux

Re: zaplocked installation recovery

Reply #2 –
Quote from: cat herders of linux – on
if your home directory is on a separate drive than your root directory and you can't seem to recover your root directory you could always try reinstalling and just be sure to only format your root directory and NOT your home.

BUT before you do that you could try booting a usb/pendrive again with artix on it and try some filesystem recovery tools appropriate for your file system.  what format are you using?  ext4?  that would be ideal and probably easiest.  btrfs has its own tools also...

No problem reinstalling, but it's all the apps and he setup where the work is. It's ext4, I think the filesystem is ok physically, I can mount it from another system and examine it, it could be missing things however. I'll start the iso download tomorrow, expect to be back at my desktop within 20 hours.

 
Who, has loved us more?

Re: zaplocked installation recovery

Reply #4 –
Quote from: badsector – on
I cannot for the life of me find the draft posting I started this with last night!  How do I get to it???

I had a filesystem corruption incident destroying the directory on a data drive from which I recovered reasonably well with testdisk but my Artix Plasma system ended up with an only partly populated /boot folder

memtest86+
amd-ucode.img
intel-ucode.img

It started with a lockup forcing a hard reset and whatever happened caused a journal replay on reboot on the data drive also hosting some of the home directories. I 'think' I made a mistake and rebooted to another system whereas in retrospect it might have been smarter to reboot Artix. Can't remember, anyway, I can't really tell what else is missing now. On this reboot I didn't suspect anything, yet, I did noticed the journal replay spooling the drive, but only found out that something was wrong when I could not log-in because my home folder was on the zapped data drive.Tried some kernel transplants to no avail (those used to work at one time). Long story short, I finally  chrooted from another system and did an update which seemed promising because it included a new kernel so the /boot folder got replenished as it were. Exited chroot and redid the grub thing, thinking I had it made. But then on reboot I got

"ERROR: Root device mounted succesfully, but /sbin/init does not exist.
 Bailing out, you are on yoiur own, Good luck."

Is there a reasonably easy way to salvage the installation? I'm no guru but can chew may way through a 2do list. 

TIA


>"ERROR: Root device mounted succesfully, but /sbin/init does not exist."

What init are you using? most likely it was a symlink to your actual init that you can re-create yourself.

Still, I would recommend to do a complete reinstall since even your init was not safe, seems like you might have more surprises waiting once you get over that hurdle.

Re: zaplocked installation recovery

Reply #5 –
As above it might be easier to install Artix again.
I'd suggest doing so to a separate partition or drive. Then you can still easily copy over configs and data to the new installation. With the caveat that you need to take care that what you are copying is not corrupt!

To help recreate your install you can take a list of all explicitly installed (by you) packages and then use the list to install them all.
https://wiki.archlinux.org/title/pacman/Tips_and_tricks#Install_packages_from_a_list

Just to see what happens I would try adding
Code: [Select]
init=/usr/bin/bash
to the end of your kernel command line from grub or rEFInd (or whatever boots your system) just to see if you get to a prompt.
As your corruption was definitely to at least your boot folder I'd be slightly suspicious that /boot/initramfs-linux.img may not be correct ?

Another thing to try is chroot in and install pacutils, if you don't have it, and use paccheck to check installed files against the database
Code: [Select]
paccheck --list-broken
(artix-branding-base changes it's own files on first run so don't worry about that one)
Of course this won't help if the pacman database is corrupted.

Re: zaplocked installation recovery

Reply #6 –
i just want to strongly assert, if this were my computer, i would be stress testing the hdd,ssd,nvme to see if there's some hardware failure imminent.  I would also first back up any important data that i could that i don't want lost should the stress test finish off said drive.
Cat Herders of Linux

Re: zaplocked installation recovery

Reply #7 –
Quote from: cat herders of linux – on
i just want to strongly assert, if this were my computer, i would be stress testing the hdd,ssd,nvme to see if there's some hardware failure imminent.  I would also first back up any important data that i could that i don't want lost should the stress test finish off said drive.

Any even marginally important data should be backed up already. But you're right OP should backup if not done as a matter of course.
S.M.A.R.T should give an idea of reliability.
For solid state drives especially all a stress test is going to do is bring the moment of failure closer. So I fail to see a benefit ?

Always expect a drive to fail imminently.

Re: zaplocked installation recovery

Reply #8 –
Quote from: gripped – on
Any even marginally important data should be backed up already. But you're right OP should backup if not done as a matter of course.
S.M.A.R.T should give an idea of reliability.
For solid state drives especially all a stress test is going to do is bring the moment of failure closer. So I fail to see a benefit ?

Always expect a drive to fail imminently.




All drives are not going to imminently fail.

Fine.  You fail to see the benefit.  Should your failure become ops failure?
Clearly if theres any defecit and how much, the benefit is op can see it for themselves.  Seeing somethi g for oneself has great benefit.  Do i really have to explain that?
Cat Herders of Linux

Re: zaplocked installation recovery

Reply #9 –
Quote from: cat herders of linux – on

All drives are not going to imminently fail.
Of course not. Well hopefully! But from a backup perspective it's best to treat them as though anyone of them could imminently fail.

If you stress test a drive what do you prove ? That it didn't fail during the test. Or that it did.
If it survived that does not mean it's a safe drive, a good drive. It only means it survived the test. It could start to fail on the very next write. Or in 1000 writes. Or 1 million. The stress test proves nothing as a user.
As a manufacturer you would stress test drives to failure. That has a purpose.
Stress testing CPU's, GPU's & memory over-clocks has a purpose.
Torturing hard drives will just make them die sooner.
If the drive has bad sectors replace it. If the S.M.A.R.T info shows problems replace it.

The OP didn't state how the corruption occurred. It may be a bad disk. It may have been a bad shutdown, power cut. IDK.

But regardless stress testing the drive won't help imho.
 

Re: zaplocked installation recovery

Reply #10 –
You can reinstall all your installed packages on the problem drive, also you could try to use whatever fsck applies to your fs to identify damaged files and / or some other method like paccheck mentioned above, working from a chroot if required. Often it can be fixed but not always. Essentially you need to find and replace all missing and damaged files in the OS, remember it could be config files too, which might complicate things. I couldn't offer any step by step guide, I usually just look up how to do these kind of tasks mentioned and try some stuff. Converting to a COW fs like BTRFS should  avoid this happening again on unclean shutdowns for future reference, something to consider (not everyone's a fan of BTRFS but this sort of thing is exactly why I started using it.) If the HDD is suspect, sure, replace and bin it after getting any useful data, but the first post specifically mentioned a lock up and hard power off which could cause corruption by itself on a non-COW fs, not every time, just if you are unlucky. Whether a reinstall is easier is a question though! Restoring from a backup could be another option if you had one, saving what you can from your current homedir.

Re: zaplocked installation recovery

Reply #11 –
Quote from: gripped – on
Of course not. Well hopefully! But from a backup perspective it's best to treat them as though anyone of them could imminently fail.

If you stress test a drive what do you prove ? That it didn't fail during the test. Or that it did.
If it survived that does not mean it's a safe drive, a good drive. It only means it survived the test. It could start to fail on the very next write. Or in 1000 writes. Or 1 million. The stress test proves nothing as a user.
As a manufacturer you would stress test drives to failure. That has a purpose.
Stress testing CPU's, GPU's & memory over-clocks has a purpose.
Torturing hard drives will just make them die sooner.
If the drive has bad sectors replace it. If the S.M.A.R.T info shows problems replace it.

The OP didn't state how the corruption occurred. It may be a bad disk. It may have been a bad shutdown, power cut. IDK.

But regardless stress testing the drive won't help imho.
 

Op clearly stated a hard reset corrupted the drive.  "It started with a lockup forcing a hard reset "

A stress test on a good drive will do it no harm.  On a bad drive it will def become apparent just how bad it is.
Cat Herders of Linux

Re: zaplocked installation recovery

Reply #12 –
Quote from: cat herders of linux – on
Op clearly stated a hard reset corrupted the drive.  "It started with a lockup forcing a hard reset "
Ok. So could still be a bad disk which caused a lockup, or just filesystem corruption caused by the hard reset.
Quote
A stress test on a good drive will do it no harm.  On a bad drive it will def become apparent just how bad it is.
I think we have a different definition of "Stress test a drive"? What do you mean by stress test a drive? How would you go about it ?

Re: zaplocked installation recovery

Reply #13 –
Quote from: gripped – on
Ok. So could still be a bad disk which caused a lockup, or just filesystem corruption caused by the hard reset.I think we have a different definition of "Stress test a drive"? What do you mean by stress test a drive? How would you go about it ?

I posted a link.  3rd reply

At this point though I'm wandering off topic and wondering why you seem less interested in helping op and more interested in just establishing me as wrong....
Cat Herders of Linux
 

Re: zaplocked installation recovery

Reply #14 –
Quote from: cat herders of linux – on
I posted a link.  3rd reply

At this point though I'm wandering off topic and wondering why you seem less interested in helping op and more interested in just establishing me as wrong....

That's not a stress test. That's a benchmark
I think I've made a few suggestions in this thread to try and give the OP a hand reviving their install.

When I see posts that have little merit I let it go most off the time. But when I see a post, part of which is both of little merit, and potentially harmful I comment.
I said you were right about the the backup. I said I didn't see the benefit of a stress test because there isn't a benefit and it just increases the wear to your drive. The only way to properly stress test a drive is to write and read the entire drive, mainly write. Many times.
And as already stated it proves nothing about a likely failure if it passes. Waste of time and shortens the drives life.