Solaris Manual Patching with SVM Root Disk

1.We need to prepare patch document procedure with back-out plan, and needs to get the approval by change management. Once we get the approval, we can start our change in scheduled window
2.Download suitable our platform bundle patch(Recommended Solaris Cluster Patch) from https://support.oracle.com
#isainfo -b [ to check the bit version]
3.Take the necessary backups of the server.
4.Check console and server access.
5.Check if any issues are tere in that server

Take Below outputs

# uname –a 
# df –k 
# metastat –p 
# metadb –i 
# netastat –nr
# ifconfig –a 
# prtconf 
# prtdiag –v 
# metastat -ac 
# metastat –t
# cp -p /etc/mail/sendmail.cf /etc/mail/sendmail.cf.ddmmyy
# cp -p /etc/mail/submit.cf /etc/mail/submit.cf.ddmmyy
# cp -p /etc/ssh/sshd_config /etc/ssh/sshd_config.ddmmyy
# eeprom #echo | format 
# cp -p /etc/system /etc/system.ddmmyy 
# cp -p /etc/vfstab /etc/vfstab.ddmmyy

NOTE: Better to run SUN Explorer before preceding this activity
NOTE: Take the outputs before detaching the mirror else you would have to take the backups separately for your detached disk
NOTE: Zones must be halted, while doing patch on active boot enviromnet
#zoneadm list –vc
NOTE: Before proceeding with patching, make sure our application & data base are down
6. Break the mirror, and it will isolate from root disk.
c1t0d0 —> Root disk
c1t1d0 —> Root-mirror disk

take the lay-out of root disk and root mirror.

prtvtoc -h /dev/rdsk/c1t10d0s2 >/var/tmp/CONFIG.`hostname`.`date '+%d%m%y'`/prtvtoc.c1t10d0
prtvtoc -h /dev/rdsk/c0t8d0s2 > /var/tmp/CONFIG.`hostname`.`date '+%d%m%y'`/prtvtoc.c0t8d0

Check Zones are available or not, if the zones are in cluster

# zoneadm list -cv
# zoneadm -z zonename halt
# zoneadm -z zonename detach

# df -n /zones/<server name >
/zones/<server name >: ufs

##### For Rollback of zones ############

# mkdir -p /var/crash/<server name >
# ufsdump 0uf /var/crash/<server name >/<server name >_root.dump /zones/<server name >

Create pdo.conf file under /etc/patch dir, if it is not available on system
vi /etc/patch/pdo.conf (and add the below content “num_proc=6”)
num_proc=6 (Depended on number of Local Zones) after saving this file Please change the permissions of pod.conf to 755

# vi /etc/patch/pdo.conf
num_proc=x (Where x=6 is no of local zones running on global zone)

# chmod 755 /etc/patch/pdo.conf

First delete state database replicas from root-mirror disk

#metadb –i
a p luo 16 8192 /dev/dsk/c1t1d0s7
#metadb –d –f /dev/dsk/ c1t1d0s7

• Proceed with the mirror detach for your root-mirror disk

# metastat –p
 d1 -m d20 d21 1
d20 1 1 c1t0d0s1
d21 1 1 c1t1d0s1
d0 -m d10 d11 1
d10 1 1 c1t0d0s0
d11 1 1 c1t1d0s0
#metadetach d1 d21
#metadetach d0 d11
#metaclear d21
#metaclear d11
#metastat –p
 d1 -m d20
d20 1 1 c1t0d0s1
 d0 -m d10
d10 1 1 c1t0d0s0

Mount detached root-mirror disk on mount point /mnt and edit /mnt/etc/vfstab & /mnt/etc/vfstab

#mount /dev/dsk/c1t1d0s0 /mnt
#vi /mnt/etc/vfstab [change md entries to native entries]
/dev/md/dsk/d1  /dev/dsk/c1t1d0s1
/dev/md/dsk/d0  /dev/dsk/c1t1d0s0
# grep -i md /mnt/etc/system
* Begin MDD root info (do not edit)
rootdev:/pseudo/md@0:0,0,blk [comment with *]
* End MDD root info (do not edit)

7. Unmount root-mirror disk and fsck

#umount /mnt
#fsck /dev/rdsk/c1t1d0s0

NOTE:
1.) Please keep in mind that all changes are applicable to root-mirror disk. Do not do any changes on other disk (i.e. root-disk).
2.) It is recommended to run fsck on root-mirror disk (slice 0, rootFS slice) so that it won’t give any errors while booting.
3.) Before proceeding with the patching make sure that /var FS have enough free space as all logs will be written there. else you will get errors while patching.

8. Now our root-disk is under SVM control and root-mirror disk is under native raw Solaris disk. By doing this step we are making sure that in case if anything goes against our action plan then we will be able to recover our server from other disk.
9. We have to test the booting of our server from both the disks to make sure that the server will come up with both of the disks without any issues.

#init 0
Ok>devalias root-mirror
/pci@1f,0/pci@1/scsi@8/disk@1,0:a
Ok>boot root-mirror 
Once system come-up with root-mirror, needs do the post check like df –h, ifconfig etc
Now boot the server from root-disk as well
#init 0
Ok>devalias rootdisk
/pci@1f,0/pci@1/scsi@8/disk@0,0:a
Ok>boot rootdisk

Once system come-up with rootdisk, needs do the post check like df –h, ifconfig etc

9. Now we can go ahead with patching, but server should be single user mode

#cd /var/tmp
# digest -a md5 -v /var/tmp/10_x86_Recommended.zip [will check md5 value]
#unzip -q 10_Recommended.zip
NOTE: for unzipping of recommended patch, we need 5 GB of disk space
#grep PASSCODE 10_Recommended.README
*************** PASSCODE **************
PASSCODE: s10patchset
#shutdown -g0 -y –i6
#who –r
. run-level S Oct 25 12:39 S 0 3
NOTE: For installation recommended patch, we need 3 GB of disk space
#./installpatchset --s10patchset 
Setup ..........
Recommended OS Patchset Solaris 10 x86 (2011.09.14)

The patch set will complete installation in this session. No intermediate
reboots are required.

Application of patches started : 2011.09.16 20:24:45
Applying 120901-03 ( 1 of 302) ... skipped
Applying 121334-04 ( 2 of 302) ... skipped
Applying 119255-81 ( 3 of 302) ... skipped
.
. <similar output omitted>
.
Applying 147379-01 (300 of 302) ... success
Applying 147435-01 (301 of 302) ... success
Applying 147441-01 (302 of 302) ... success

NOTE: A symlink to ‘installpatchset’ is provided in place of the legacy ‘installcluster’ script for now

10. Once patching has done, needs to run reconfiguration reboot on server and check the kernel version

#touch /reconfigure
#reboot -- -r
#uname -a
SunOS Release 5.10 Version Generic_147441-01 32-bit
NOTE: Do not re-mirror the root-mirror disk immediately; wait for 5 to 7 days to notice any issues with the OS or application. If all is good then re-mirror the disks

NOTE: If patch fails, for roll-back you need to boot from the root-mirror disk and create all the SVM mirrors back on root-mirror disk from start and then re-mirror root-disk back with root-mirror disk.

Re-Attaching the Local zones

# Zoneadm -z zonename attach -U

# zoneadm -z zonename ready

# zoneadm -z zonename boot

# zoneadm list -cvv

Complete all post verifications tests attach the mirror with approvals of app team

# metattach d100 d10
# metattach d103 d13

# metastat | grep -i sync
# metastat

Custom Fix or Rollback of patching

1.  Shutdown and detach the zones

# shutdown -y -g0 -i0
OK devalias
OK boot rootmirror –s

Cleared the rootdisk's metadevices along with main mirror.

# metaclear -fr d100
# metaclear -fr d103
# metstat -p d100 d103 (there should not be any d100,d103)
# metainit d20 1 1 c0t8d0s0 (on rootdisk)
# metainit d23 1 1 c1t10d0s0

Recreated the main mirror with rootmirror disk

# metainit d100 -m d10
# metainit d103 -m d13
# metaroot d100 
# vi /etc/vfstab (make sure / and /var are of d0 and d3 respectively. Edit as needed)

# shutdown -y -g0 -i0
OK boot rootmirror

Perform TVT
Handover servers to Apps Support for application startup and PVT

Reattach mirrors on hosts
*** Only do this AFTER
root cause has been identified ***

# metattach d100 d20
# metattach d103 d12
# metastat | grep %

Restore the UFS dump for local zones roll back

# cd /zones/<zone name >; rm -rf

do for all the zones

# ufsrestore -xvf /var/crash/<zone name>/<zone name>_root.dump .

Set the auto boot true for Zones

Boot the zones

Patching is completed succesfully