Performance issues-Solaris

If its comes to performance issue in Unix operating systems, everybody will be behind on CPU.But not all the time CPU will cause for the performance problem. Because sometimes CPU may waiting for resources like Disk I/O, Swapping ,Network.As sysadmin you should able to pin point where the problem.vmstat is very useful command to determined where it the problem.

After introduction Solaris zones, sysadmin job is become bit difficult to isolate the problems from other local zones and global zone. prstat is very useful command to find local zones CPU usage from global zone.
Solaris Support for Process Binding and Processor Sets

psrinfo Display CPU type and state

psradm Change CPU state

pbind Bind (or unbind) a process to a CPU

psrset Administer processor sets

mpstat Display CPU statistics, optionally on a processor set basis

ptree Display process hierarchy in which a process runs

The following options are supported:

-a Perform the action on all processors, or as many as possible.

-f Take the specified processors off-line.

SunOS 5.9 Last change: 10 Dec 1998 1

System Administration Commands psradm(1M)

-i Set the specified processors no-intr.

-n Bring the specified processors on-line.

-v Output a message giving the results of each attempted
operation.

The following example sets processors 2 and 3 off-line.

psradm -f 2 3

The following example sets processors 1 and 2 no-intr.

psradm -i 1 2

The following example sets all processors on-line.

psradm -a -n


VMSTAT

bash-3.00# vmstat 1
 kthr      memory            page            disk          faults      cpu
 r b w   swap  free  re  mf pi po fr de sr f0 s0 s1 s2   in   sy   cs us sy id
 0 0 0 1061088 86408 36  99  0  0  1  0  6 -0  2 -0  0  328  458  186  0  1 98
 1 0 0 954432 93008  15  49  0  0  0  0  0  0  0  0  0  546  583  307  0  2 97
 0 0 0 954352 92956   8   8  0  0  0  0  0  0  0  0  0  470  545  297  0  1 99
 0 0 0 954352 92956   8   8  0  0  0  0  0  0  0  0  0  566  563  328  0  2 98
 0 0 0 954352 92956   8   8  0  0  0  0  0  0  0  0  0  581  605  315  0  1 99
 0 0 0 954352 92956   8   8  0  0  0  0  0  0  0  0  0  553  564  301  0  1 99
 2 0 0 954352 92956   8   8  0  0  0  0  0  0  0  0  0  550  574  315  0  4 95
 1 0 0 954352 92956   8   8  0  0  0  0  0  0  0  0  0  604  682  340  0  1 98
 0 0 0 954352 92956   8   8  0  0  0  0  0  0  0  0  0  508  721  356  0  4 96
 1 0 0 954352 92956   8   8  0  0  0  0  0  0  0  0  0  519  575  321  0  2 97

The fields of vmstat’s display are
 procs 
r     in run queue
b     blocked for resources I/O, paging etc.
w     swapped

 memory (in Kbytes)
swap –  amount  of  swap   space   currently   available
free   – size of the free list     page ( in units per second).
re    page reclaims –  see  -S  option  for  how  this field is modified.
mf    minor faults –  see  -S  option  for  how    this field is modified.
pi    kilobytes paged in
po    kilobytes paged out
fr    kilobytes freed
de    anticipated short-term memory shortfall (Kbytes)
sr    pages scanned by clock algorithmdisk  ( operations per second )

There are  slots for up to four disks, labeled with a single letter and number.

The letter indicates  the  type  of  disk  (s = SCSI, i = IPI, etc) . The number is

the logical unit number.     faults
in    (non clock) device interrupts
sy    system calls
cs    CPU context switches cpu  –   breakdown of percentage usage of CPU  time.  On multiprocessors  this is an a

verage across all processors.
us    user time
sy    system time
id    idle time

Problem symptoms:

1.) If the number of processes in run queue (procs r) are consistently greater than the number of CPUs on the system it will slow down system as there are more processes then available CPUs .
2.) if this number is more than four times the number of available CPUs in the system then system is facing shortage of cpu power and will greatly slow down the processess on the system.
3.) If the idle time (cpu id) is consistently 0 and if the system time (cpu sy) is double the user time (cpu us) system is facing shortage of CPU resources.

 

Resolution :

Resolution to these kind of issues involves tuning of application procedures  to make efficient use of cpu  and as a last resort increasing the cpu power or adding more cpu to the system

Top:
Top is tool which is commonly available all the Unix/Linux.IN Solaris we need to install .To see top 10 process which are utilizing high CPU time,use below command.

bash-3.00# /usr/local/bin/top 10
last pid:5307;load avg:0.16,  1.25,  0.95; up 2+12:33:59 14:17:37                        
122 processes: 121 sleeping, 1 on cpu
CPU states: 99.0% idle,  0.5% user,  0.5% kernel,  0.0% iowait,  0.0% swap
Memory: 1024M phys mem, 91M free mem, 1124M total swap, 1054M free swap
   PID USERNAME LWP PRI NICE  SIZE   RES STATE    TIME    CPU COMMAND
 12321 root       1  59    0   67M 9688K sleep    3:07  0.17% mixer_applet2
  5306 root       1  52    0 3104K 1696K cpu/1    0:00  0.12% top
 12319 root       1  59    0   69M   11M sleep    1:19  0.07% gnome-netstatus
 12251 root       1  59    0   10M 6100K sleep    1:16  0.06% gconfd-2
 12226 root       1  59    0   25M 3596K sleep    1:15  0.06% vmtoolsd
 19353 noaccess  18  59    0  122M   97M sleep    2:16  0.04% java
  3961 noaccess  18  59    0  140M   87M sleep    6:17  0.03% java
  3869 root       1  59    0   73M   20M sleep    1:33  0.03% Xorg
  2015 root       1  59    0 7924K 3300K sleep    7:51  0.03% vmtoolsd
 12300 root       1  59    0   72M   12M sleep    0:39  0.03% gnome-panel


Prstat:
Prstat is in-built Solaris performance monitoring tool.Its very useful to find zone wise CPU utilization and also we can monitor amount of time threads wait for a processor using prstat -mL command in the LAT coloum.

bash-3.00# prstat -a
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
 18759 root     3856K 3144K cpu0    59    0   0:00:01 1.5% prstat/1
  2015 root     7572K 3744K sleep   59    0   0:05:25 0.2% vmtoolsd/1
 12226 root       25M 6696K sleep   59    0   0:00:28 0.2% vmtoolsd/1
 18497 root     6448K 3644K sleep   59    0   0:00:02 0.2% sshd/1
 12321 root       67M   10M sleep   59    0   0:00:52 0.2% mixer_applet2/1
     5 root        0K    0K sleep   99  -20   0:03:47 0.2% zpool-rpool/39
 12319 root       69M   13M sleep   59    0   0:00:23 0.1% gnome-netstatus/1
  3961 noaccess  140M   95M sleep   59    0   0:04:10 0.1% java/18
 12251 root       10M 7452K sleep   59    0   0:00:25 0.1% gconfd-2/1
  3869 root       73M   24M sleep   59    0   0:01:00 0.0% Xorg/1
 12300 root       72M   14M sleep   59    0   0:00:13 0.0% gnome-panel/1
 18508 root     2984K 1716K sleep   59    0   0:00:00 0.0% bash/1
   392 root     6156K 2776K sleep   59    0   0:00:07 0.0% nscd/33
 NPROC USERNAME  SWAP   RSS MEMORY      TIME  CPU
    84 root      172M  207M    20%   0:14:20 2.8%
     1 noaccess  134M  101M   9.9%   0:04:10 0.1%
     7 daemon   7528K 6816K   0.7%   0:00:12 0.0%
Total: 92 processes, 356 lwps, load averages: 0.08, 0.05, 0.04

bash-3.00# prstat -Z
   PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP
  4901 root     3808K 3052K cpu1     0    0   0:00:00 1.3% prstat/1
  4879 root     6448K 3876K sleep   59    0   0:00:02 0.4% sshd/1
  2015 root     7908K 3796K sleep   59    0   0:07:45 0.2% vmtoolsd/1
     5 root        0K    0K sleep   99  -20   0:05:30 0.2% zpool-rpool/39
 12321 root       67M 9892K sleep   59    0   0:03:04 0.2% mixer_applet2/1
 19353 noaccess  122M  105M sleep   59    0   0:02:13 0.1% java/18
  3961 noaccess  140M   94M sleep   59    0   0:06:14 0.1% java/18
 12319 root       69M   12M sleep   59    0   0:01:15 0.1% gnome-netstatus/1
  4890 root     3000K 1852K sleep   54    0   0:00:00 0.1% bash/1
 12226 root       25M 4480K sleep   59    0   0:01:11 0.1% vmtoolsd/1
 12251 root       10M 6684K sleep   59    0   0:01:15 0.1% gconfd-2/1
  3869 root       73M   21M sleep   59    0   0:01:26 0.0% Xorg/1
 12300 root       72M   13M sleep   59    0   0:00:38 0.0% gnome-panel/1
   398 daemon   4260K 2212K sleep   59    0   0:00:02 0.0% kcfd/3
  1889 root     3888K 1172K sleep   59    0   0:00:01 0.0% sshd/1
  4878 root     4800K 2640K sleep   59    0   0:00:00 0.0% sshd/1
 18497 root     6448K 3140K sleep   59    0   0:00:12 0.0% sshd/1
ZONEID    NPROC  SWAP   RSS MEMORY      TIME  CPU ZONE
     0       95  318M  242M    24%   0:32:03 2.7% global
    10       30  157M  141M    14%   0:03:04 0.1% sol1

bash-3.00# prstat -mL
   PID USERNAME USR SYS TRP TFL DFL LCK SLP LAT VCX ICX SCL SIG PROCESS/LWPID
 27258 root     4.2  17 0.0 0.0 0.0 0.0  78 0.2  76   0 48K   0 prstat/1
  1907 root     0.4 0.6 0.0 0.1 0.0 0.0  99 0.3  51   2  2K   0 vmtoolsd/1
 21975 root     0.2 0.1 0.0 0.0 0.0 0.0 100 0.1  30   1 193   0 mixer_applet/1
 21291 root     0.2 0.0 0.0 0.1 0.1 0.0 100 0.2  51   0 212   0 vmtoolsd/1
 21971 root     0.0 0.1 0.0 0.0 0.0 0.0 100 0.1  10   0  61   0 gnome-netsta/1
 22001 root     0.1 0.1 0.0 0.0 0.0  98 2.2 0.1   6   0  94   0 firefox-bin/18
 21289 root     0.1 0.1 0.0 0.0 0.0 0.0 100 0.1  20   1 161   0 gconfd-2/1
  3803 noaccess 0.1 0.1 0.0 0.0 0.0 0.0  99 0.4  97   0  96   0 java/14
 24052 root     0.1 0.1 0.0 0.0 0.1 0.0  99 0.8  11   0  42   0 gnome-termin/1
  3803 noaccess 0.1 0.0 0.0 0.1 0.0 0.1 100 0.1   0   0  33   0 java/35
 13099 root     0.1 0.0 0.0 0.0 0.1 0.0 100 0.1  25   0 100  11 Xorg/1
 21583 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.1  20   0 124   0 gnome-panel/1
     5 root     0.0 0.1 0.0 0.0 0.0 0.0 100 0.0   1   0   0   0 zpool-rpool/13
     5 root     0.0 0.1 0.0 0.0 0.0 0.0 100 0.0   1   0   0   0 zpool-rpool/39
 22001 root     0.0 0.0 0.0 0.3 0.3 0.0  99 0.0   3   0  14   0 firefox-bin/1
  3803 noaccess 0.0 0.0 0.0 0.0 0.0 100 0.0 0.1   5   0   5   0 java/7
  1597 daemon   0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   4   0 nfsmapid/4
     5 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   2   0   0   0 zpool-rpool/12
    67 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   7   0 vxconfigd/1
     5 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   2   0   0   0 zpool-rpool/38
     5 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   1   0   0   0 zpool-rpool/28
     5 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   1   0   0   0 zpool-rpool/19
     5 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   1   0   0   0 zpool-rpool/31
     5 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.1   1   0   0   0 zpool-rpool/30
     5 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.1   1   0   0   0 zpool-rpool/27
     5 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.1   1   0   0   0 zpool-rpool/21
 24884 root     0.0 0.0 0.0 0.0 0.0 0.0 100 0.0   0   0   2   0 sshd/1

SAR:

The below command will CPU statistics for every 5 seconds for five times.We can add SAR in crontab to monitor CPU usage all the time.The load averages can be monitored extended time periods by looking at run queue lengths and the amount of time that the run queue is occupied using sar -q. High number of system calls per second per processor can  be monitored using sar -c.

bash-3.00# sar -u 5 5 

SunOS sfos 5.10 Generic_142910-17 i86pc

 07/18/2012 

14:05:40      usr    %sys    %wio   %idle
14:05:45       0       3       0      97
14:05:50       1       2       0      98
14:05:55       0       2       0      97
14:06:00       1       3       0      96
14:06:05       0       2       0      98
Average        0       2       0      97
bash-3.00# sar -q
SunOS sfos 5.10 Generic_142910-17 i86pc    04/29/2013
10:37:41 runq-sz %runocc swpq-sz %swpocc
10:40:01     1.3      24    39.0     100
10:45:00     1.3      19    39.0     100
10:50:01     1.0      18    39.0     100
Average      1.2      20    39.0     100

bash-3.00# sar -c
SunOS sfos 5.10 Generic_142910-17 i86pc    04/29/2013
10:37:41 scall/s sread/s swrit/s  fork/s  exec/s rchar/s wchar/s
10:40:01     933     115      60    0.16    0.15   11689    8178
10:45:00     886      68      32    0.78    0.77  113195    8021
10:50:01     744      45      21    0.40    0.40   58097    6056

MPSTAT:

 

bash-3.00# mpstat
CPU minf mjf xcal  intr ithr  csw icsw migr smtx  srw syscl  usr sys  wt idl
  0   78   0    1   352  117  168    1    6    2    0   311    1   3   0  97
  1   71   0    1   127   29  158    2    6    3    0   328    1   2   0  97

iostat ( Input Output statistics )
iostat   reports terminal and disk  I/O  activity and  CPU utilization.  The first line of output is for the  time period  since boot  &  each subsequent line is for  the  prior  interval . Kernel maintains  a number of counters to keep track of  the  values.
iostat’s activity class options default  to  tdc  (terminal,  disk, and CPU). If any other option/s are specified,  this  default is completely overridden i.e.  iostat -d will report only statistics about the disks.

synctax : iostat  <options>   interval  count
option – let you specify the device for which information is needed like disk , cpu or terminal. (-d , -c , -t  or -tdc ) .  x options gives the extended statistics .
interval –  is time period in seconds between two samples . iostat  4  will give data at each 4 seconds interval.
count  – is the  number of times the data is needed .  iostat 4 5   will give data at 4 seconds interval   5 times

Example
$ iostat -xtc 5 2
extended disk statistics tty cpu
disk r/s w/s Kr/s Kw/s wait actv svc_t %w %b tin tout us sy wt id
sd0 2.6 3.0 20.7 22.7 0.1 0.2 59.2 6 19 0 84 3 85 11 0
sd1 4.2 1.0 33.5 8.0 0.0 0.2 47.2 2 23
sd2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
sd3 10.2 1.6 51.4 12.8 0.1 0.3 31.2 3 31

The fields have the following meanings:

disk name of the disk
r/s reads per second
w/s writes per second
Kr/s kilobytes read per second
Kw/s kilobytes written per second
wait average number of transactions waiting for service (Q length)
actv average number of transactions actively being serviced
(removed from the queue but not yet completed)
%w percent of time there are transactions waiting
for service (queue non-empty)
%b percent of time the disk is busy (transactions in progress)
svc_t Service time (svc_t)

Results and Solutions:
The values to look from the iostat output  are:
Reads/writes  per second (r/s , w/s)
Percentage busy (%b)
Service time (svc_t)
If a disk shows consistently high reads/writes along with , the percentage busy (%b) of the disks is greater than 5 percent, and the average service time  (svc_t) is greater than 30 milliseconds, then  one of the following action needs to be taken
1.)Tune the application to use disk i/o more efficiently  by modifying the disk queries and using available cache facilities of application servers .
2.) Spread the file system of the disk on to two or more disk  using disk striping feature of volume manager /disksuite  etc.
3.) Increase the system parameter values for  inode cache  , ufs_ninode ,  which is  Number of inodes to be held in memory. Inodes are cached globally (for UFS), not on a per-file system basis
4.) Move the file system to another faster disk /controller  or replace existing disk/controller to a faster
one.

Virtual Memory Statistics ( vmstat )
vmstat –  vmstat reports virtual memory statistics of   process, virtual memory, disk, trap, and CPU activity.

On multicpu systems , vmstat averages the number of CPUs  into  the  output. For per-process statistics .Without options, vmstat displays a one-line summary  of  the  virtual memory activity since the system was booted.

syntax:
Basic synctax is vmstat  <options>   interval  count
option – let you specify the type of information needed such as paging  -p , cache   -c ,.interrupt -i  etc.
if no option is specified  information about   process , memory , paging , disk ,interrupts & cpu  is displayed  .
interval  – is time period in seconds between two samples . vmstat   4  will give data at each 4 seconds interval.
count  – is the number of times the data is needed . vmstat 4   5   will give data at 4 seconds interval   5
times.
Example
The following command displays a summary of what the system
is doing every five seconds.

example% vmstat 5
procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr s0 s1 s2 s3 in sy cs us sy id
0 0 0 11456 4120 1 41 19 1 3 0 2 0 4 0 0 48 112 130 4 14 82
0 0 1 10132 4280 0 4 44 0 0 0 0 0 23 0 0 211 230 144 3 35 62
0 0 1 10132 4616 0 0 20 0 0 0 0 0 19 0 0 150 172 146 3 33 64
0 0 1 10132 5292 0 0 9 0 0 0 0 0 21 0 0 165 105 130 1 21 78
The fields of vmstat’s display are
procs
r     in run queue
b     blocked for resources I/O, paging etc.
w     swapped

memory (in Kbytes)
swap –  amount  of  swap   space   currently   available
free   – size of the free list

page ( in units per second).
re    page reclaims –  see  -S  option  for  how  this field is modified.
mf    minor faults –  see  -S  option  for  how    this field is modified.
pi    kilobytes paged in
po    kilobytes paged out
fr    kilobytes freed
de    anticipated short-term memory shortfall (Kbytes)
sr    pages scanned by clock algorithm

disk  ( operations per second )
There are  slots for up to four disks, labeled with a single letter and number.
The letter indicates  the  type  of  disk  (s = SCSI, i = IPI, etc) . The number is
the logical unit number.

faults
in    (non clock) device interrupts
sy    system calls
cs    CPU context switches

cpu  –   breakdown of percentage usage of CPU  time.  On multiprocessors  this is an average across all processors.
us    user time
sy    system time
id    idle time

Results and Solutions:
A.   CPU issues:
Following columns has to be watched to determine if there is any cpu issue
Processes in the run queue (procs r)
User time (cpu us)
System time (cpu sy)
Idle time (cpu id)
procs cpu
r b w us sy id
0 0 0 4 14 82
0 0 1 3 35 62
0 0 1 3 33 64
0 0 1 1 21 78
Problem symptoms:
1.) If the number of processes in run queue (procs r) are consistently greater than the number of CPUs on the system it will slow down system as there are more processes then available CPUs .
2.) if  this number is more than four times the number of available CPUs in the system then system is facing shortage of cpu power and will greatly slow down the processess on the system.
3.) If  the idle time (cpu id) is consistently 0 and if the system time (cpu sy) is double the user time (cpu us)  system is facing shortage of CPU resources.

Resolution :
Resolution to these kind of issues involves tuning of application procedures  to make efficient use of cpu  and as a last resort increasing the cpu power or adding more cpu to the system.

B.   Memory Issues:
Memory bottlenecks are determined by the scan rate (sr) . The scan rate is the pages scanned by the clock algorithm per second. If the scan rate (sr) is continuously over 200 pages per second then there is a memory shortage.

Resolution :
1. Tune the applications & servers to make  efficient use of memory and cache.
2. Increase system memory .
3. Implement priority paging in s in pre solaris 8 versions by adding line “set priority paging=1” in
/etc/system. Remove this line if upgrading from Solaris 7 to 8 & retaining old /etc/system file.

Network Statistics (netstat)
netstat displays the  contents  of  various  network-related  data structures in  depending on the options selected.
Syntax :
netstat  <option/s>
multiple options can be given at one time.
Options
-a – displays the state of all sockets.
-r – shows the system routing tables
-i – gives statistics on a per-interface basis.
-m – displays information from the network memory buffers. On Solaris, this shows statistics    forSTREAMS
-p [proto] – retrieves statistics for the specified protocol
-s – shows per-protocol statistics. (some implementations allow -ss to remove fileds with a value of 0 (zero) from the display.)
-D – display the status of DHCP configured interfaces.
-n do not lookup hostnames, display only IP addresses.
-d (with -i) displays dropped packets per interface.
-I [interface] retrieve information about only the specified interface.
-v be verbose
interval  –   number for continuous display of statictics.
Example :
$netstat -rn

Routing Table: IPv4
Destination           Gateway               Flags  Ref   Use   Interface
——————– ——————– —– —– —— ———
192.168.1.0         192.168.1.11           U        1   1444      le0
224.0.0.0             192.168.1.11           U        1   0            le0
default                  192.168.1.1           UG       1   68276
127.0.0.1             127.0.0.1               UH       1  10497     lo0
This shows the output on a Solaris machine who’s IP address is 192.168.1.11 with a default router at 192.168.1.1
Results and Solutions:
A.) Network availability
The command as above is mostly useful in troubleshooting network accessibility issues . When  outside network is not accessible from a machine check the following
1. if the default router ip  address is correct
2.  you can ping it from your machine.
3. If router address is incorrect  it can be changed  with route add  commnad . See man route  for more info .
route command examples:
$route add default <hostname>
$route add 192.0.2.32  <gateway_name>
If the router address is correct but still you can’t ping it  there may be some  network cable /hub/switch problem  and you have to try and eliminate the faulty component .

B.) Network Response

bash-3.2# netstat -i
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue
lo0 8232 loopback localhost 2527 0 2527 0 0 0
e1000g0 1500 karri1 karri1 56770 0 55934 0 0 0
e1000g1 1500 192.168.57.0 192.168.57.11 47290 0 44824 0 0 0
e1000g2 1500 192.168.58.0 192.168.58.11 47266 0 44790 0 0 0

bash-3.2#

This option is used to diagnose the network problems when  the connectivity is there but  it is slow in response .
Values to look at:
Collisions (Collis)
Output packets (Opkts)
Input errors (Ierrs)
Input packets (Ipkts)
The above values will give information to workout
i.  Network collision rate as follows :
Network collision rate = Output collision counts / Output packets
Network-wide collision rate greater than 10 percent  will indicate
Overloaded network,
Poorly configured network,
Hardware problems.
ii.  Input packet error rate as follows :
Input Packet Error Rate = Ierrs / Ipkts.
If the input error rate is high (over 0.25 percent), the host is dropping packets. Hub/switch cables etc needs to be checked for potential problems.
C. Network socket &  TCP Cconnection state
Netstat gives important   information about network socket and tcp state . This is very useful in finding out the open , closed and waiting network tcp connection .
Network states returned by  netstat are following :
CLOSED               —-  Closed.  The socket  is  not  being used.
LISTEN                 —-  Listening for incoming connections.
SYN_SENT           —-  Actively trying to  establish  connection.
SYN_RECEIVED  —- Initial synchronization of the connection under way.
ESTABLISHED     —-  Connection has been established.
CLOSE_WAIT      —-  Remote shut down; waiting  for  the socket to close.
FIN_WAIT_1        —-  Socket closed; shutting  down  connection.
CLOSING             —-  Closed,   then   remote   shutdown; awaiting acknowledgement.
LAST_ACK          —-   Remote  shut  down,  then   closed ;awaiting acknowledgement.
FIN_WAIT_2        —-  Socket closed; waiting for shutdown from remote.
TIME_WAIT         —-  Wait after close for  remote  shutdown retransmission.

Example:

 

bash-3.2# netstat -a

UDP: IPv4
 Local Address Remote Address State
-------------------- -------------------- ----------
 *.route Idle
 *.sunrpc Idle
 *.* Unbound
 *.32771 Idle
 *.* Unbound
 *.* Unbound
 *.* Unbound
 *.32772 Idle
 *.lockd Idle
 *.32773 Idle
 *.32774 Idle
 *.32775 Idle
 *.32776 Idle
 *.servicetag Idle
 *.syslog Idle
 *.16161 Idle
 *.32777 Idle
 *.32778 Idle
 *.32779 Idle
 *.32780 Idle
 *.32781 Idle
 *.32784 Idle
 *.32786 Idle
 *.snmpd Idle
 *.177 Idle
 *.32790 Idle
 *.* Unbound
 *.* Unbound
 *.* Unbound

UDP: IPv6
 Local Address Remote Address State If
--------------------------------- --------------------------------- ---------- -----
 *.servicetag Idle
 *.177 Idle

TCP: IPv4
 Local Address Remote Address Swind Send-Q Rwind Recv-Q State
-------------------- -------------------- ----- ------ ----- ------ -----------
 *.* *.* 0 0 49152 0 IDLE
karri1.32768 192.168.56.101.3260 23744 0 64240 0 ESTABLISHED
 *.sunrpc *.* 0 0 49152 0 LISTEN
 *.* *.* 0 0 49152 0 IDLE
 *.32772 *.* 0 0 49152 0 LISTEN
 *.32773 *.* 0 0 1048952 0 LISTEN
 *.lockd *.* 0 0 1048952 0 LISTEN
localhost.5987 *.* 0 0 49152 0 LISTEN
localhost.898 *.* 0 0 49152 0 LISTEN
localhost.32774 *.* 0 0 49152 0 LISTEN
localhost.5988 *.* 0 0 49152 0 LISTEN
localhost.32775 *.* 0 0 49152 0 LISTEN
 *.32776 *.* 0 0 49152 0 LISTEN
 *.fs *.* 0 0 49152 0 LISTEN
 *.shell *.* 0 0 49152 0 LISTEN
 *.32777 *.* 0 0 49152 0 LISTEN
 *.32778 *.* 0 0 49152 0 LISTEN
 *.32779 *.* 0 0 49152 0 LISTEN
 *.32780 *.* 0 0 49152 0 LISTEN
 *.32781 *.* 0 0 49152 0 LISTEN
 *.telnet *.* 0 0 49152 0 LISTEN
 *.dtspc *.* 0 0 49152 0 LISTEN
 *.ftp *.* 0 0 49152 0 LISTEN
 *.finger *.* 0 0 49152 0 LISTEN
 *.login *.* 0 0 49152 0 LISTEN
 *.servicetag *.* 0 0 49152 0 LISTEN
 *.ssh *.* 0 0 49152 0 LISTEN
 *.32782 *.* 0 0 49152 0 LISTEN
 *.32783 *.* 0 0 49152 0 LISTEN
 *.3260 *.* 0 0 49152 0 LISTEN
 *.3233 *.* 0 0 49152 0 LISTEN
 *.3233 *.* 0 0 49152 0 LISTEN
 *.14141 *.* 0 0 49152 0 LISTEN
 *.14150 *.* 0 0 49152 0 LISTEN
 *.32784 *.* 0 0 49152 0 LISTEN
 *.5634 *.* 0 0 49152 0 LISTEN
localhost.32794 localhost.32793 49152 0 49152 0 ESTABLISHED
localhost.32793 localhost.32794 49152 0 49152 0 ESTABLISHED
localhost.32000 *.* 0 0 49152 0 LISTEN
 *.6000 *.* 0 0 49152 0 LISTEN
karri1.ssh 192.168.56.1.50991 65280 51 49640 0 ESTABLISHED
 *.32818 *.* 0 0 49152 0 BOUND
localhost.6788 *.* 0 0 49152 0 LISTEN
localhost.6789 *.* 0 0 49152 0 LISTEN
localhost.smtp *.* 0 0 49152 0 LISTEN
localhost.submission *.* 0 0 49152 0 LISTEN
localhost.32806 *.* 0 0 49152 0 LISTEN
localhost.32952 localhost.3233 49152 0 49152 0 TIME_WAIT

h-3.00# iostat -Cxn
 extended device statistics
 r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device
 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0
 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 c0t0d0
 0.5 2.3 13.9 13.6 0.0 0.0 0.0 7.3 0 1 c1
 0.2 2.1 12.1 10.2 0.0 0.0 0.0 7.8 0 1 c1t0d0
 0.0 0.0 0.0 0.3 0.0 0.0 0.0 3.4 0 0 c1t1d0
 0.0 0.0 0.0 0.3 0.0 0.0 0.0 0.9 0 0 c1t2d0
 0.2 0.0 0.2 0.3 0.0 0.0 0.0 0.1 0 0 c1t3d0
 0.1 0.1 0.7 2.5 0.0 0.0 0.1 13.1 0 0 c1t4d0
 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 fd0
 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0
                 extended  disk  statistics  aggregated  by  controller id.
    -x          Report extended  disk  statistics.
    -n          Display names in descriptive format.  For  example, cXtYdZ, rmt/N, server:/export/path.
From the above output, we need to look at the below columns to identify the I/O bottle necks.
1.asvc_t    average service time of  active  transactions,  in milliseconds. If the average service time is exceeding more than 25, then there is issue in the specific disk and need attention.
2.wsvc_t average  service  time  in  wait  queue,  in  milliseconds. If the wait queue is crossed more than 10 then we need to check with SAN team regarding this.To more about the rest of the fields ,see man page of iostat.

prstat -u sr52666 -s cpu -n 20

prstat -s size -a -n 10

# /usr/dt/bin/sdtprocess & ------------>CDE Process Manager from the command line
prstat ------>command examines and displays information about active processes on the system

# prstat
PID USERNAME SIZE RSS STATE PRI NICE TIME CPU PROCESS/NLWP
1641 root 4864K 4520K cpu0 59 0 0:00:00 0.5% prstat/1
1635 root 1504K 1168K sleep 59 0 0:00:00 0.3% ksh/1
9 root 6096K 4072K sleep 59 0 0:00:29 0.1% svc.configd/11
566 root 82M 30M sleep 29 10 0:00:36 0.1% java/14

Default Column Heading
Description
PID
The PID number of the process.
USERNAME
The login name or UID of the owner of the process.
SIZE
The total virtual memory size of the process.
RSS
The resident set size of the process in kilobytes, megabytes, or
gigabytes.
STATE
The state of the process:
• cpu – The process is running on the CPU.
• sleep – The process is waiting for an event to complete.
• run – The process is in the run queue.
• zombie – The process terminated, and the parent is not waiting.
• stop – The process is stopped.
PRI
The priority of the process.
NICE
The value used in priority computation.
TIME
The cumulative execution time for the process.
CPU
The percentage of recent CPU time used by the process
PROCESS/NLWP
The name of the process/the number of lightweight processes
(LWPs) in the process.

prstat

Option Description
-a Displays separate reports about processes and users at the same time.
-c Continuously prints new reports below previous reports.
-n nproc Restricts the number of output lines.
-p pidlist Reports only on processes that have a PID in the given list.
-s key Sorts output lines by key in descending order. The five possible keys include: cpu, time, size, rss, and ri. You can use only one key at a time.
-S key Sorts output lines by key in ascending order.
-t Reports total usage summary for each user.
-u euidlist Reports only processes that have an effective user ID (EUID) in the given list.
-U uidlist Reports only processes that have a real UID in the given list


pgrep -lf sendmail
pkill sendmail
$ /usr/lib/sendmail -bd -q 15 m

-bd ---> Runs the sendmail daemon in the background
-q 15 --> Checks the mail queue every 15 minutes for new message to send

pkill -u kumar ps

 kill or pkill commands to terminate one or more processes.

kill -signal PID
pkill -signal Process
# pgrep -l mail
241 sendmail
240 sendmail
#
# pkill sendmail