Skip to main content ↑ ↓
Topic: Create and inspect core dumps of service programs (Read 1009 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

Create and inspect core dumps of service programs

I am currently hacking at s6 support for ly, the TUI display manager. Unfortunately, the ly program crashes but it creates no core dumps. I verified that my user (and root user as well) can create core dumps but I cannot find any trace of a core dump from the crashing ly program. The only  trace of the crash is a string of logs in the dmesg output. Any ideas on how to get those cores when my program is launched via an exec ly in a service run script?

Re: Create and inspect core dumps of service programs

Reply #1 –
Any useful hints on here?
https://discuss.aerospike.com/t/enabling-coredumps-on-redhat-centos/713
I think I got coredumps working from those instructions when I needed them, although adapted a bit. Remember things like /etc/profile get read at boot so I guess you have to run the ulimit command before whatever service you are debugging has started somehow. I don't know exactly how that plays out regarding the crashing service, perhaps even make a little service of your own that runs before? The dumping cores in a specific location might help too?
Edit: checking what I actually did:
Created /etc/sysctl.conf:
Code: [Select]
kernel.core_pattern = coredumpfile
#kernel.core_pipe_limit = 0
kernel.core_uses_pid = 1
Modified /etc/security/limits.conf:
Code: [Select]
#*               soft    core            0
*               soft    core            unlimited
Tested, in one terminal run "top", then another:
Code: [Select]
$ sudo kill -11 `pidof top`
This makes e.g. coredumpfile.2945 which is quite uniquely named to search for, and you can do sudo updatedb and then locate coredumpfile, beware while you have them enabled everything that crashes anywhere will make a coredump so you might get them in unexpected places which will waste drive space. This was using openrc and you might be able to improve things a bit for your purposes.

Looking around some more, in openrc the "sysctl" service runs the config in /etc/sysctl.conf so anything running before then wouldn't work as expected. Also security/limits.conf might not work for init shells as it relates to pam so, possibly see the first link to run a ulimits cmd instead before or during the service?

Re: Create and inspect core dumps of service programs

Reply #2 –
Thanks for the good pointers. AFAICT, the sysctl service is set to run all scripts in /etc/system.d directory. I created a core.conf there to set relevant kernel params. Still, there will be no crash dump created when a service execs a program (like ly) if the ulimit was not already set by the time the service's s6-supervise process is run. As I am sure you are familiar each service has one such process running and that process is responsible for running the up and down scripts that actually launch the program that provides the service. As it happens, the s6-supervise process for my ly DM gets a ulimit of 0 for cores. Not all is lost though, the ulimit can be set after the fact, while the process is running, with prlimit. I changed the ulimit for the s6-supervise process for ly, up-ed the service, which crashed and I got the dump and found the source of the crash.

Is it at all possible to set the ulimit to unlimited before PID 1 (init) is running?

Re: Create and inspect core dumps of service programs

Reply #3 –
Laurent Bercot has responded on the issue of ulimit-s for s6 services on the skaware mailing list, here. The reply details how to enable unlimited core size before s6-linux-init process starts.

Re: Create and inspect core dumps of service programs

Reply #4 –
That's good you found a solution.
https://github.com/util-linux/util-linux/blob/master/sys-utils/prlimit.c
Looking at the code for prlimit it runs a kernel syscall, and I didn't find a kernel command line option or config option to change the default, so it looks like you either need to run the syscall or find where the default value is hardcoded into the kernel and build a custom kernel after switching it there. So it sounds from your link that s6 itself must include support to run the syscall.

Re: Create and inspect core dumps of service programs

Reply #5 –
Nice detective work. I've never had to do this kind of thing myself, but hopefully skarnet can make this easily configurable.
 

Re: Create and inspect core dumps of service programs

Reply #6 –
For future generations: based on Laurent Bercot's comment mentioned in reply 3, this is how to enable core dumps for single service:

Code: [Select]
# cat /etc/s6/sv/nginx-srv/run
#!/bin/execlineb -P
fdmove -c 2 1
s6-softlimit -c=               <---- inserted this line
exec nginx -g "daemon off;"

And also (see `man 5 core`):

Code: [Select]
# cat /etc/sysctl.d/99-dimgel-all.conf 
kernel.core_pattern = /tmp/core-%e.%p.%t