Solaris Core and Crash Dump

Crash dump Used to diagnose the problem.

Core files are just a process dump (image from physical memory) when the process terminates abnormally. —>application/DB  need to send these core files to application vendor to fix this issue.

Crash-dump is also similar to core-dump but instead of dumping for the process failure, it creates the memory dump for Operating system crash/failure. This crashdump file should be sent to Operating system vendor to find the root cause for the system crash.

# reboot -d   (To generate the crashdump)
When an operating system crashes, the savecore command is automatically executed during a boot. The savecore command retrieves
the crash dump from the dump device and then writes the crash dump to a pair of files in your file system:
The savecore command places kernel core information in the
/var/crash/nodename/vmcore.X file, where nodename is the name returned by uname -n, and X is an integer identifying the dump.
The savecore command places name list information and symbol table information in the
/var/crash/nodename/unix.X file.

###################################################################################
Dump considerations:
How much memory do you want dumped? all, kernel, kernel + active process
# dumpadm
Dump content: kernel pages
Dump device: /dev/dsk/c0t3d0s1 (swap)
Savecore directory: /var/crash/pluto ***(large enough to hold core)
Savecore enabled: yes

# dumpadm -c all -d /dev/dsk/c0t1d0s1 -m 10%
Dump content: all pages
Dump device: /dev/dsk/c0t1d0s1 (dedicated)
Savecore directory: /var/crash/pluto (minfree = 77071KB)
Savecore enabled: yes
savecore -L (live core dump, WATCH OUT, do not do a savecore -L to a dumpslot under volume
manager control)
###################################################################################

To view the current dump configuration

# dumpadm
Dump content: kernel pages
Dump device: /dev/dsk/c0t0d0s1 (swap)
Savecore directory: /var/crash/sys-02
Savecore enabled: yes

dumpadm -d /dev/dsk/c#t#d#s5
dumpadm -s /var/crash/sys-02
dumpadm -y

Run the sync command to flush all previously unwritten system buffers out to disk
# sync

Force the kernel to save a live snapshot of the running system and write out a new set of crash dump files
# savecore -L
configuration is located in the /etc/dumpadm.conf file:

# cat /etc/dumpadm.conf
#
# dumpadm.conf
#
# Configuration parameters for system crash dump.
# Do NOT edit this file by hand — use dumpadm(1m) instead.
#
DUMPADM_DEVICE=/dev/dsk/c0t0d0s1
DUMPADM_SAVDIR=/var/crash/sys-02
DUMPADM_CONTENT=kernel
DUMPADM_ENABLE=yes

——————————————————————————————————-
Note – Perform all modifications to the crash dump configuration by using the dumpadm command, rather than attempting to edit the
/etc/dumpadm.conf file. Editing the file might result in an inconsistent system dump configuration.
——————————————————————————————————-

The syntax of the dumpadm command is:

/usr/sbin/dumpadm [-nuy] [-c content-type] [-d dump-device] [-m mink | minm | min%] [-s savecore-dir] [-r root-dir]

where:
-n Modifies the dump configuration so it does not run the savecore command automatically on reboot.
-u Forcibly updates the kernel dump configuration based on the contents of the /etc/dumpadm.conf file.
-y Modifies the dump configuration so that the savecore command is run automatically on reboot. This is the default.
-c content-type Specifies the contents of the crash dump. The content-type can be kernel, all, or curproc. The curproc content type includes the kernel memory pages and the memory pages of the currently executing process.
-d dump-device Modifies the dump configuration to use the specified dump device. The dump device can be an absolute path name or swap

-m mink Creates a minfree file in the current savecore-dir directory indicating that the savecore command should maintain at least the specified
-m minm amount of free space in the file system in which the savecore-dir directory is located:
-m min%

• k – Indicates a positive integer suffixed with the unit k, specifying kilobytes.
• m – Indicates a positive integer suffixed with the unit m, specifying megabytes.
• % – Indicates a percent (%) symbol, indicating that the minfree value is computed as the specified percentage of the total, current size of the file system that contains the savecore-dir directory.
-r root-dir Specifies an alternative root directory relative to which the dumpadm command should create files. If the -r argument is not specified, the default root directory “/” is used.
-s savecore-dir Modifies the dump configuration to use the specified directory to save files written by the savecore command. The default savecore-dir directory is /var/crash/hostname, where hostname is the output of the uname -n command.


Forced Crash Dump

This is an useful procedure to create a forced crash dump when the system becomes unstable and no available “vmcore” file is there.. it will be created under /var/crash/`hostname`

karri2 # tty
/dev/console
karri2 # uadmin 5 0
panic[cpu2]/thread=30006212fc0: forced crash dump initiated at user request

000002a10182b850 genunix:kadmin+4a4 (b4, 0, 0, 11ef800, 5, 0)
%l0-3: 0000000001814000 00000000011ae000 0000000000000004 0000000000000004
%l4-7: 0000000000000438 0000000000000010 0000000000000004 0000000000000000
000002a10182b910 genunix:uadmin+110 (5, 0, 0, ff38e000, 18898, 5)
%l0-3: 0000000000000000 0000000000000000 00000000fab00000 000000000000fab0
%l4-7: 0000000000000001 0000000000000000 0000000000000000 000003003d8034c8

syncing file systems… 3 1 done
dumping to /dev/md/dsk/d20, offset 18468831232, content: kernel

1% done
2% done
3% done
4% done
5% done
=

98% done
99% done
100% done
100% done: 354845 pages dumped, compression ratio 4.27, dump succeeded
Program terminated
{2} ok
{2} ok

karri2 $ cd /var/crash/karri2
karri2 $ ls -l
total 6455810
-rw-r–r– 1 root root 2 May 27 17:54 bounds
-rw-r–r– 1 root root 2437568 May 27 17:51 unix.2
-rw-r–r– 1 root root 368127263 May 27 17:54 vmcore.2.gz
-rw——- 1 root other 2933155328 May 30 14:21 vmcore.2.tar
karri2 $


Core dumps:

You can see the application coredump location using the below command. To enable ,you can user -e option.

# coreadm
global core file pattern:
global core file content: default
init core file pattern: core
init core file content: default
global core dumps: disabled
per-process core dumps: enabled
global setid core dumps: disabled
per-process setid core dumps: disabled
global core dump logging: disabled

There are two patterns in the coreadm options.Those are global pattern & init pattern.

node1-CGI#coreadm -g /var/core/core-%n/core_%n_%f_%u_%g_%t_%p
node1-CGI#coreadm -i /var/core/core-%n/core_%n_%f_%u_%g_%t_%p
node1-CGI#coreadm
global core file pattern:/var/core/core-%n/core_%n_%f_%u_%g_%t_%p
    global core file content: default
init core file pattern: /var/core/core-%n/core_%n_%f_%u_%g_%t_%p
      init core file content: default
           global core dumps: enabled
      per-process core dumps: enabled
     global setid core dumps: enabled
per-process setid core dumps: enabled
     global core dump logging: enabled

Thank you for reading this article. Please leave a comment if you have any doubt. I will get back to you.

Forcing a core dump:

Let’s assume if you want to have an core dump of a running process on your system:

# ps -ef | grep “bash” | grep “hans-karri”
hans-karri 8345 3475 0 21:29:19 pts/11 0:10 bash
Okay, now we can trigger the core dump by using the process id of the process.
# gcore 8345
gcore: core.8345 dumped

You can analysis using  mdb command.
# mdb core.8345
Loading modules: [ libc.so.1 ld.so.1 ]
> $c