Patching Linux Server

In this blog we are going to Patch Linux Machine Using up2date and yum. We are going to take backup of important fine and necessary steps after patching, backout plan if system crashed

Backup of your important files

  • Take the back-up of the following files/commands.
  • Common for all revisions:
  • uname -a
  • ifconfig –a
  • fdisk -l
  • uptime
  • cat /etc/hosts
  • cat /etc/fstab
  • df -h
  • cat /etc/grub.conf
  • cat /etc/sysctl.conf
  • rpm -qa > /packagelist_beforePatch_May2011.txt
  • cat /packagelist_beforePatch_May2011.txt
  • cat /etc/selinux/config
  • cat /etc/resolv.conf
  • chkconfig –list

RHEL 4

  • #cat /etc/sysconfig/rhn/up2date
  • #up2date –l
  • #up2date –configure
  • #more /etc/sysconfig/rhn/up2date

RHEL 5

  • more /etc/yum.conf
  • yum check-update
  • The below document takes all the details of the remaining system-files as a part of taking backup of system configuration:

After-Patching:

  • rpm -qa > /packagelist_afterPatch_May2011.txt
  • cat/packagelist_afterPatch_May2011.txt
  • First, you must update the up2date utility do to havening problems not being able to boot up after patching.

#up2date up2date

  • this will download and install the latest up2date utility
  • After verifying that up2date is at the latest revision and the development and production environment are the same you must first down load the patches on the all the servers that are being patched and install patches on the development servers for testing.

#up2date –dry-run Or   #up2date -l   Or #up2date   –nodownload

  • This will show you the updated patches/packages that are available for download.
    Fetching Obsoletes list for channel: rhel-i386-es-4…

Fetching rpm headers…

########################################

Name                                    Version             Rel               Arch

  • —————————————————————————————-
  • 4Suite                                 1.0                 3.el4_8.1           i386
  • PyXML                                  0.8.3               6.el4_8.2           i386
  • acpid                                   1.0.3               2.el4_7.1           i386
  • apr                                     0.9.4               24.9.el4_8.2       i386
  • apr-util                              0.9.4               22.el4_8.2         i386
  • audit                                   1.0.16             4.el4_8.1           i386
  • audit-libs                             1.0.16             4.el4_8.1           i386
  • bash                                    3.0                 21.el4_8.2         i386
  • bind-libs                               9.2.4               30.el4_8.5         i386
  • bind-utils                             9.2.4               30.el4_8.5         i386
  • compat-openldap                        2.1.30             12.el4_8.2         i386
  • cpio                                   2.5                 16.el4_8.1         i386
  • cpp                                     3.4.6               11.el4_8.1         i386
  • wget                                    1.10.2             1.el4_8.1           i386
  • xmlsec1                                 1.2.6               3.1                 i386
  • xmlsec1-openssl                         1.2.6               3.1                 i386
  • Testing package set / solving RPM inter-dependencies…
  • ########################################
  • Name                                     Version             Rel               Arch
  • —————————————————————————————-
  • 4Suite                                 1.0                 3.el4_8.1           i386
  • PyXML                                   0.8.3               6.el4_8.2           i386
  • acpid                                   1.0.3               2.el4_7.1          i386
  • bind-utils                             9.2.4               30.el4_8.5         i386
  • compat-openldap                         2.1.30             12.el4_8.2         i386
  • gd                                       2.0.28               5.4E.el4_8.1      i386
  • glibc                                   2.3.4               2.43.el4_8.3       i686
  • The following Packages were marked to be skipped by your configuration:
  • Name                                     Version             Rel                     Reason
  • ————————————————————————————————
  • kernel                                 2.6.9               89.0.26.EL         Pkg name/pattern
  • kernel-smp                             2.6.9              89.0.26.EL         Pkg name/pattern
  • kernel-utils                           2.4                 20.el4             Pkg name/pattern

#more /etc/sysconfig/rhn/up2date

# Automatically generated Red Hat Update Agent config file, do not edit.

# Format: 1.0

useNoSSLForPackages [comment] =Use the noSSLServerURL for package, package list, a

nd header fetching

useNoSSLForPackages=0

storageDir[comment]=Where to store packages and other data when they are retrieved

storageDir=/var/spool/up2date

[comment]=Remote server URL without SSL

noSSLServerURL=http://xmlrpc.rhn.redhat.com/XMLRPC

networkRetries[comment]=Number of attempts to make at network connections before

giving up

networkRetries=5

pkgsToInstallNotUpdate[comment]=A list of provides names or package names of pack

ages to install not update

pkgsToInstallNotUpdate=kernel;kernel-modules;kernel-devel;

#up2date –configure

Select the required options ( keepAfterInstall & pkgskipList and etc) to change the Configuration of Up2date Agent.

  1. debug             No
  2. rhnuuid             38e8d384-589b-11d7-9124-00096be0a8c5
  3. isatty             Yes

showAvailablePacka No

  1. depslist           [ ]
  2. networkSetup       Yes
  3. retrieveOnly       No
  4. enableRollbacks   No

8.pkgSkipList       [‘kernel*’]

9.storageDir         /var/spool/up2date

#up2date -d or up2date -duk /var/spool (any dir)

  • This will download and save only the updates/packages in /var/spool/up2date or what is defined in line 9 of up2date-config file
  • Run only if packages are downloaded into non-default directories

Example:

#up2date –iuk /var/spool

  • This will download patches/rpm into a custom directory. The default download directory is /var/spool/up2date. If the updates/packages have already been downloaded, use this option below to install the downloaded updates/packages.
  • After patches are installed

#rpm -qa > /packagelist_afterPatch_10182010.txt

  • A new listing should be done after patching for future reference.

#cat newpatchlist.txt

  • Onece you fine this command then you will get below mention output

vim-enhanced-6.3.046-0.40E.7

vim-minimal-6.3.046-0.40E.7

vixie-cron-4.1-50.el4

vsftpd-2.0.1-6.el4

vte-0.11.11-12.el4

vte-0.11.11-12.el4

wget-1.10.2-0.40E

which-2.16-4

wireless-tools-28-0.pre16.3.3.EL4

words-3.0-3.2

wvdial-1.54.0-3

Xaw3d-1.5-24

# shutdown [OPTION]… TIME [MESSAGE] The shutdown command format.

# shutdown -r 0

# Broadcast message from root@RH5

(/dev/pts/1) at 14:10 …

The system is going down for reboot NOW!

  • AFTER THE PREDEFINDED, TESTING PEIORED THE UPDATES/PATCHES WILL NEED TO BE MOVED TO THE PRODUCTION ENVIROMENT.

PRODUCTION SERVER

  • Take the back-up of the following files/commands.

#uname -a

#ifconfig –a

#cat /etc/hosts

#cat /etc/fstab

#df -h

#cat /etc/sysconfig/rhn/up2date

#cat /etc/grub.conf

#cat /etc/sysctl.conf

#rpm -qa > /packagelist_10152010.txt#cat /packagelist_10152010.txt

#cat /etc/selinux/config

#up2date –configure

  • Select the required options ( keepAfterInstall & pkgskipList and etc) to change the Configuration of Up2date Agent.
  1. debug             No
  2. rhnuuid             38e8d384-589b-11d7-9124-00096be0a8c5
  3. isatty             Yes
  4. showAvailablePacka No
  5. depslist           [ ]
  6. networkSetup       Yes
  7. retrieveOnly       No
  8. enableRollbacks   No

8.pkgSkipList       [‘kernel*’]

9.storageDir         /var/spool/up2date

(Run only if packages are downloaded into non-default directories)

up2date -iuk (custom directory)

Example:

#up2date –iuk /var/spool

  • This will check for downloaded patches first before downloading from the RHN. The default download directory is /var/spool/up2date. If the updates/packages have already been downloaded, use this option to install the downloaded updates/packages first before checking the RHN for updates/packages.

#rpm -qa > newpatchlist.txt

  • A new listing should be done after patching for future reference.

#cat newpatchlist.txt

vim-enhanced-6.3.046-0.40E.7

vim-minimal-6.3.046-0.40E.7

vixie-cron-4.1-50.el4

vsftpd-2.0.1-6.el4

vte-0.11.11-12.el4

vte-0.11.11-12.el4

wget-1.10.2-0.40E

which-2.16-4

wireless-tools-28-0.pre16.3.3.EL4

words-3.0-3.2

wvdial-1.54.0-3Xaw3d-1.5-24

# shutdown [OPTION]… TIME [MESSAGE] The shutdown command format.

shutdown -r 0

# Broadcast message from root@RH5
(/dev/pts/1) at 14:10 …
The system is going down for reboot NOW!

 RHEL 5 patching Steps

  • (RH5): How to download and install patches/Updates for a development/production environment:

Take the back-up of the following files/commands.

#uname -a

#ifconfig –a

#fdisk -l

#cat /etc/hosts

#cat /etc/fstab

#df -h

#cat /etc/yum.conf

#cat /etc/grub.conf

#cat /etc/sysctl.conf

#rpm -qa > /packagelist_10152010.txt

#cat /packagelist_10152010.txt

#cat /etc/selinux/config

  • First, you must install the yum downloadonly utility to give yum the ability to download patches/rpm.

#yum install yum-downloadonly

  • this will download and install the downloadonly utility
  • After verifying that yum download utility is installed and the development and production environment are the same you must first down load the patches on the all the servers that are being patched and install patches on the development server for testing
  • In addition, you will need to clear the yum cache.

yum clean all

  • This will clean the yum chache when you again fire the command then it will search all repository for updated packages.

 yum check-update

  • Once you fine check-update command then This will give you a list of updated patches/packages available for download.

 

kpartx.i386                                                                         0.4.7-34.el5_5.1                                                           rhel-i386-server-5

krb5-libs.i386                                                                      1.6.1-36.el5_5.4                                                           rhel-i386-server-5

krb5-workstation.i386                                                               1.6.1-36.el5_5.4                                                         rhel-i386-server-5

libsmbclient.i386                                                                 3.0.33-3.29.el5_5                                                         rhel-i386-server-5

lvm2.i386                                                                          2.02.56-8.el5_5.4                                                         rhel-i386-server-5mkinitrd.i386                                                                       5.1.19.6-61.el5_5.1                                                      rhel-i386-server-5

nash.i386                                                                           5.1.19.6-61.el5_5.1                                                       rhel-i386-server-5

net-snmp-libs.i386                                                                1:5.3.2.2-9.el5_5.1                                                       rhel-i386-server-5

nscd.i386                                                                           2.5-49.el5_5.2                                                            rhel-i386-server-5

yum update -–downloadonly or yum update –downloaddir= (custom directory)

  • This will download and install updates/packages. This may update several packages on server including kernel.
  • Yum will download rpm files to the default download directory /var/cache/yum.

 more /etc/yum.conf

[main]

cachedir=/var/cache/yum

keepcache=0

debuglevel=2

logfile=/var/log/yum.log

distroverpkg=redhat-release

tolerant=1

exactarch=1

obsoletes=1

gpgcheck=1

plugins=1

# Note: yum-RHN-plugin doesn’t honor this.

metadata_expire=1h

# Default.

# installonly_limit = 3

# PUT YOUR REPOS HERE OR IN separate files named file.repo

# in /etc/yum.repos.d

yum localinstall <path/filename> [<path/filename>]

  • Example:#yum localinstall   /var/cache/yum/rhel-i386-server-5/packages/*
  • Software testing is built into the yum command

rpm -qa | sort >newpatchlist.txt

  • A new listing should be done after patching for future reference.

cat newpatchlist.txt

vim-enhanced-6.3.046-0.40E.7

vim-minimal-6.3.046-0.40E.7

vixie-cron-4.1-50.el4

vsftpd-2.0.1-6.el4

vte-0.11.11-12.el4

vte-0.11.11-12.el4

wget-1.10.2-0.40E

which-2.16-4

wireless-tools-28-0.pre16.3.3.EL4

words-3.0-3.2

wvdial-1.54.0-3

Xaw3d-1.5-24
shutdown {OPTION}.. TIME {MESSAGE} the shutdown command format

shutdown -r 0

  • # Broadcast message from root@RH5
    (/dev/pts/1) at 14:10 …

    • The system is going down for reboot NOW!
  • AFTER THE PREDEFINDED, TESTING PEIORED THE UPDATES/PATCHES WILL NEED TO BE MOVED TO THE PRODUCTION ENVIROMENT.

Production Server

  • Take the back-up of the following files/commands.

#uname -a

#ifconfig –a

#cat /etc/hosts

#cat /etc/fstab

#df -h

#cat /etc/yum.conf

#cat /etc/grub.conf

#cat /etc/sysctl.conf

#rpm -qa > /packagelist_10152010.txt

#cat /packagelist_10152010.txt

#cat /etc/selinux/config

yum localinstall   <path/filename> [<path/filename>]

  • Example:#yum localinstall /var/cache/yum/rhel-i386-server-5/packages/*
  • Software testing is built into the yum command

rpm -qa | sort >newpatchlist.txt

  • A new listing should be done after patching for future reference.

#cat newpatchlist.txt

vim-enhanced-6.3.046-0.40E.7

vim-minimal-6.3.046-0.40E.7

vixie-cron-4.1-50.el4

vsftpd-2.0.1-6.el4

vte-0.11.11-12.el4

vte-0.11.11-12.el4

wget-1.10.2-0.40E

which-2.16-4

wireless-tools-28-0.pre16.3.3.EL4

words-3.0-3.2

wvdial-1.54.0-3

Xaw3d-1.5-24

# shutdown [OPTION]… TIME [MESSAGE] The shutdown command format.

shutdown -r 0

# Broadcast message from root@RH5

(/dev/pts/1) at 14:10 …

The system is going down for reboot NOW

Backout plan to boot the server into old kernel:

  • Boot the server from old kernel through GRUB.
  • Edit the grub configuration file under /etc/grub.conf. (Delete the new kernel entry, make the old-kernel as default)

Backout plan to boot the server into old kernel in case if the GRUB is corrupted:

  • If the patching corrupts the present kernel which corrupts the GRUB, then perform the below tasks:
  • The GRUB build will be corrupted as OS is corrupted. So, insert OS-CD on the machine and boot from CD.
  • Proceed to the OS from the rescue mode, and select grub.conf.
  • Make appropriate changes to the file, which reflects old-kernel to be booted as default. (This makes the server to boot from it.)
  • Restart the server and Boot the server from the old-kernel.

 Backout Plan in case of server crash:

  • If the old-kernel and new-kernel both are crashed while patching the machine, then we shall need to rebuild the server. Follow the below mentioned steps for rebuild:
  • Insert CD into the cd-rom.
  • Boot the machine from CD and proceed with installation.
  • After the installation, work with changing the system configuration files. (Screen-shot of the system files is taken before patching)
  • Work on restoration of files from the recent backup.
  • Work with Nimsoft-tier on getting the machine into monitoring.
  • Restart the machine and make sure that the machine is back UP to the normal state as before. (Monitors should work as normal as before after this reboot