Skip to main content ↑ ↓
Topic: ran out of space on my 50gb ssd root partition and then i did a baaaaaaad thing (Read 1067 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

ran out of space on my 50gb ssd root partition and then i did a baaaaaaad thing

for whatever reason my 50 gig root partition ran out of space.  i cleaned pacman cache  and still had very little space left.  For the longest time i had like 25-30 gigs used but then it jumped up to the point that i couldn't upgrade my system.  so the very bad thing i did was go hack and slash to my var/log directory which seemed to contain some 37 gigs of log files.  now i cant upgrade my system at all because i some logs were necessary to tell the system where those files are on the system?

Code: [Select]
sudo pacman -S linux-lqx linux-lqx-headers


Packages (75) acl-2.3.1-3  attr-2.5.1-3  audit-3.1.2-1  bash-5.1.016-4  binutils-2.41-3  brotli-1.0.9-12  bzip2-1.0.8-5
              ca-certificates-20220905-1  ca-certificates-mozilla-3.92-1  ca-certificates-utils-20220905-1  coreutils-9.3-1
              curl-8.2.1-1  diffutils-3.10-1  e2fsprogs-1.47.0-1  expat-2.5.0-1  file-5.45-1  filesystem-2023.01.31-1
              findutils-4.9.0-3  gawk-5.2.2-1  gcc-libs-13.2.1-3  gdbm-1.23-2  glibc-2.38-3  gmp-6.3.0-1  grep-3.11-1
              hwdata-0.373-1  iana-etc-20230803-1  jansson-2.14-2  kbd-2.6.2-1  keyutils-1.6.3-2  kmod-30-3  krb5-1.20.1-1
              libarchive-3.7.1-1  libcap-2.69-1  libcap-ng-0.8.3-2  libelf-0.189-3  libevent-2.1.12-4  libffi-3.4.4-1
              libidn2-2.3.4-3  libldap-2.6.6-1  libnghttp2-1.55.1-1  libp11-kit-0.25.0-1  libpsl-0.21.2-1  libsasl-2.1.28-4
              libseccomp-2.5.4-2  libssh2-1.11.0-1  libtasn1-4.19.0-1  libtirpc-1.3.3-2  libudev-254.1-1  libunistring-1.1-2
              libutempter-1.2.1-3  libverto-0.3.2-4  libxcrypt-4.4.36-1  linux-api-headers-6.4-1  lz4-1:1.9.4-1  mkinitcpio-36-1
              mkinitcpio-busybox-1.36.1-1  mpfr-4.2.1-1  ncurses-6.4_20230520-1  openssl-3.1.2-1  p11-kit-0.25.0-1  pahole-1:1.25-4
              pam-1.5.3-3  pambase-20221020-1  pcre2-10.42-2  readline-8.2.001-2  shadow-4.13-2  tzdata-2023c-2  udev-254.1-1
              util-linux-2.39.2-1  util-linux-libs-2.39.2-1  xz-5.4.4-1  zlib-1:1.3-1  zstd-1.5.5-1  linux-lqx-6.4.12.lqx1-2
              linux-lqx-headers-6.4.12.lqx1-2

and so when i try to install all that mess i get a crap ton of errors telling me that the file exists on the system.  so many in gact that it would take a few hours maybe to scroll down the page to copy all the files.  here's a small example of just the linux api header files

https://pastebin.com/xh6iWyCX


is there a way to restore this damage i have caused that doesnt involve reinstalling my systemm?


Cat Herders of Linux

Re: ran out of space on my 50gb ssd root partition and then i did a baaaaaaad thing

Reply #1 –
Code: [Select]
sudo pacman -S linux-lqx linux-lqx-headers  --overwrite '*'

i ran that and it installed without errors

does that mean i diverted disaster?
Cat Herders of Linux

Re: ran out of space on my 50gb ssd root partition and then i did a baaaaaaad thing

Reply #2 –
in any case the last nvidia upgrade seemed to make liquorix not run... thats fine.  i can use artix kernel and be pleased with it.  no interest in making it work for sure.  i guess we'll see if i ever get another upgrade from pacman or if it always says up to date...  that's the test i guess if i averted disaster?
Cat Herders of Linux

Re: ran out of space on my 50gb ssd root partition and then i did a baaaaaaad thing

Reply #3 –
I don't think how anything bad happened from just wiping /var/log or cache, for pacman it's secret sauce is in /var/lib :-)
It's still strange how your logs are spammed that badly.

Re: ran out of space on my 50gb ssd root partition and then i did a baaaaaaad thing

Reply #4 –
testdisk / photorec is my favorite for recovery of deleted files. But for best results you should avoid doing anything with the partition in question (or dd an image to work on later) so the freed space with the lost files isn't reused. You usually need some kind of logrotate thing setup as a cron job to keep logs from growing too much, there are various possiblities, occasionally it stops working then logs grow, but it might also be something writing to the logs excessively.

Re: ran out of space on my 50gb ssd root partition and then i did a baaaaaaad thing

Reply #5 –
Quote from: ####### – on
testdisk / photorec is my favorite for recovery of deleted files. But for best results you should avoid doing anything with the partition in question (or dd an image to work on later) so the freed space with the lost files isn't reused. You usually need some kind of logrotate thing setup as a cron job to keep logs from growing too much, there are various possiblities, occasionally it stops working then logs grow, but it might also be something writing to the logs excessively.
if only there was a log file for that, that i hadn't deleted!
Cat Herders of Linux

Re: ran out of space on my 50gb ssd root partition and then i did a baaaaaaad thing

Reply #6 –
This is not to do with deleting /var/log/* but most likely to do with pacman database corruption

Is that pastebin all of the errors or have you snipped it ?

If all the first thing I would try is
Code: [Select]
sudo pacman -Rdd linux-api-headers
sudo pacman -S linux-api-headers

If that fails it's possible to use pacman's --overwrite optioin
You'd need to read the manpage (Under "upgrade options")  but I THINK
Code: [Select]
sudo pacman -S --overwrite=* linux-api-headers
might do it ?

I say think because I never do it that way for reasons that escape me but but I seem to remember it causing me problems in the past.

What I do do is
  • Copy the output of conflicting files into the Kate text editor
  • Replace all occurrences of "linux-api-headers: " with nothing
  • Replace all occurrences of "exists in filesystem" (leave the space before exists) with nothing
  • Switching Kate's replace tool to "Escape sequences" mode replace all occurrences of "\n" (newline) with nothing
  • Copy the one line of filenames that remains
  • Paste after sudo rm -v
  • sudo pacman -S linux-api-headers

After getting to the point you can do an update again I'd be tempted to reinstall everything
https://wiki.archlinux.org/title/Pacman/Tips_and_tricks#Reinstalling_all_packages


Re: ran out of space on my 50gb ssd root partition and then i did a baaaaaaad thing

Reply #7 –
thats just api errors.  the errors lst is too big to be copied...  anywat pacman seems to run i just have no updates available?

Code: [Select]
:: Starting full system upgrade...

 there is nothing to do
Cat Herders of Linux

Re: ran out of space on my 50gb ssd root partition and then i did a baaaaaaad thing

Reply #8 –
OK you've lost me now. They are not "api" errors they are file conflicts which suggests pacman has lost the details of which files belong to that and maybe other packages.
No list is too big to be copied ?

Does pacman -Q still produce a list of your installed packages ? If it does you can probably fix your system as I've outlined.
I don't know the damage though. Running out of drive space on the root drive can cause all sorts of problems depending on what's being processed when it happens.

Re: ran out of space on my 50gb ssd root partition and then i did a baaaaaaad thing

Reply #9 –
I would highly suggest to move your pacman cache directory onto another partition that has more space. The process is simple, just move the pacman cache directory and then edit /etc/pacman.conf to point to the new cache directory.

Re: ran out of space on my 50gb ssd root partition and then i did a baaaaaaad thing

Reply #10 –
Code: [Select]
[quote author=gripped date=1693494829 link=msg=36631]
OK you've lost me now. They are not "api" errors they are file conflicts which suggests pacman has lost the details of which files belong to that and maybe other packages.
No list is too big to be copied ?

Does pacman -Q still produce a list of your installed packages ? If it does you can probably fix your system as I've outlined.
I don't know the damage though. Running out of drive space on the root drive can cause all sorts of problems depending on what's being processed when it happens.
[/quote]






there were a so many errors i just pulled those out as 1 example.  there were probably 100 pages maybe more of errors


yah pacman -Q outputs this:
https://pastebin.com/8SiHJR0P



Code: [Select]
sudo pacman -S linux-lqx linux-lqx-headers  --overwrite '*'

is what i ran to make install work again without errors.  i can install files now i am just not sure that i am getting updates properly or not.

sudo pacman -Syu
reports everything is up to date
not sure if that's true though
Cat Herders of Linux

Re: ran out of space on my 50gb ssd root partition and then i did a baaaaaaad thing

Reply #11 –
Quote from: Lancia – on
I would highly suggest to move your pacman cache directory onto another partition that has more space. The process is simple, just move the pacman cache directory and then edit /etc/pacman.conf to point to the new cache directory.
when i deleted pacman cache it only freed up about 3 gb.  When i deleted the log files it freed up about 35 gb..  so maybe i should move /var to another drive?
Cat Herders of Linux

Re: ran out of space on my 50gb ssd root partition and then i did a baaaaaaad thing

Reply #12 –
first I recommend check what is using that much space with tool like "ncdu"
my "/var/log" only contain 51.1MiB" 35 Gb ain't normal

at least next time you face the problem

Re: ran out of space on my 50gb ssd root partition and then i did a baaaaaaad thing

Reply #13 –
Quote from: cat herders of linux – on
When i deleted the log files it freed up about 35 gb..  so maybe i should move /var to another drive?

Before you deleted the log files did you pay any intention to which file(s) were taking up the most space ?
Something is not right unless we are talking about many many years of logs.
Is logrotate setup and working ?

You could have an error or warning writing constantly to a logfile or maybe the log level is set to high eg debug for something ?
Even so logrotate should compress most repetition away.
For comparison my /var/log directory is 106MB's

Leave it a few days and see what has grown the most if you don't already know the culprit log. Then look inside it and try to figure out what the messages are and where they are coming from.

Edit: Ninja'd

Re: ran out of space on my 50gb ssd root partition and then i did a baaaaaaad thing

Reply #14 –
so my root has gone from 12 to 20 gig in a few days.  messages.log kernel.log and everything.log are each 2.4 gb in size.  when i try to open up any of them they crash geany.  here's a pic i managed to capture of what that looks like.
Cat Herders of Linux