Artix Linux Forum

How do I stop the lld linker from running out of memory?

12 July 2021, 15:17:17

I am currently trying to build a web browser more modern than Palemoon (which builds without problems but is severely out of date). The problem with all other browsers, besides profiling which ALWAYS fails, and having to download and unpack a tarball every time because there's no git repo, is the lld linker. I tried to build waterfox because its PKGBUILD has an option to turn profiling off, but the build still failed because the linker ran out of memory (I have 8GB of memory and enlarged my swapfile to 2GB, but I still ran out. This is insane.) Currently I can't use packaged browsers because I have eliminated wayland from my system to reduce complexity (see my xorg-server thread about my interface freeze issues. Currently I'm using mesa again, but I may still have to go back to glamor-only, which mostly works for me. And may I remind you that wayland is still experimental, and the last thing I need right now. Why is it hardcoded into packaged software? Let the wayland users, who presumably *enjoy* hacking, build their stuff from source!).

Do I really have to edit /etc/security/limits.conf? I never had to do that before, and compilers/linkers should know better than to crawl all over your memory. Would lld even respect limits.conf?

Re: How do I stop the lld linker from running out of memory?

Reply #1 – 12 July 2021, 18:52:59

You need more memory. Yes, modern web browsers are that insane. Make a stupidly big swap file (like 16gb or something) and if you wait long enough it should work.

Re: How do I stop the lld linker from running out of memory?

Reply #2 – 12 July 2021, 19:22:40

For some less important packages in AUR, I would pick *-bin version to install. Not worth building from source anymore.

Re: How do I stop the lld linker from running out of memory?

Reply #3 – 12 July 2021, 22:01:38

Try building with jobs=1 or whatever is the equivalent in your build options. That should be safe. The memory requirement will probably be a multiple of the threads used, the auto build settings often take the number of CPU cores as the default.
Also here is a not recommended idea, see discussion on thread, even swap isn't recommended for the same sort of reasons. You might create unexpected invisible problems and get pwned if you plan to use the result for more than testing. Getting more RAM is better than these ideas.
https://forum.palemoon.org/viewtopic.php?f=5&t=25205&hilit=building+memory
There are some easy workarounds for Pale Moon anyway, e.g. if you are having trouble on YouTube find an active Invidious instance and use the last part of the YouTube URL on the Invidious address.

Re: How do I stop the lld linker from running out of memory?

Reply #4 – 14 July 2021, 14:37:07

Thanks!
Succeeded in building waterfox. Incidentally, make -j1 won't cut it, I tried that before I posted (the build failed during the linking stage). But the 16GB swapfile and vm.swappiness=100 did the trick (according to top I used about 3GB of swap, though I wasn't present through most of the compilation). Good to know there are still solvable problems! (The palemoon thread is interesting but unneeded, since I never had any trouble building palemoon.)

Re: How do I stop the lld linker from running out of memory?

Reply #5 – 14 July 2021, 15:09:52

From my experience, it's usually the rust part of the build (I think waterfox is new enough to have this) that takes the most amount of memory. It's pretty insane.

Re: How do I stop the lld linker from running out of memory?

Reply #6 – 14 July 2021, 15:54:08

https://github.com/MrAlex94/Waterfox/commit/506d4e25e5892861b9ae6d9d08aaa3da867b069d
The Waterfox jobs config is quite complicated plus there are numerous mozconfig files, so I wonder if that was actually doing what you expected, because building Pale Moon only uses about 2GB RAM with 1 thread, although 8GB is recommended, so that's a huge difference for a related browser. Given how long that takes to build I'm not going to try Waterfox just to see though!
Also the linked idea was to strip unneeded symbols from the object files before linking which would equally apply for building other browsers although the method to do it could vary.
If you got the result you wanted then it must be fine though.