Upgrade Issue on UCS Manager
++ UCSM and FI B is upgraded successfully but FI A is stuck at 65% with the following Message
UCS pre-upgrade check failed. Free space in the file system is below threshold
Note: None of UCS have stuck with this Message Unlinke N5K, N3K. This issue is similar to CSCun79792
Load the debug Plugin and Perform the following (CHeck How to Load Debug Plugin at the End og this Section)
Linux(debug)# cd /var/tmp
Upgrade Issue on UCS Manager:
+++++++++++++++++++++++++++++
Versions Affected: https://www.cisco.com/c/en/us/support/docs/field-notices/640/fn64094.html
++ UCSM and FI B is upgraded successfully but FI A is stuck at 65% with the following Message
UCS pre-upgrade check failed. Free space in the file system is below threshold
Note: None of UCS have stuck with this Message Unlinke N5K, N3K. This issue is similar to CSCun79792
Load the debug Plugin and Perform the following (CHeck How to Load Debug Plugin at the End og this Section)
Linux(debug)# cd /var/tmp
Linux(debug)# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 300M 255M 46M 85% /
none 2.0M 4.0K 2.0M 1% /post
none 300M 255M 46M 85% /var
none 3.0G 1.1G 2.0G 35% /isan
none 600M 180M 421M 30% /var/tmp <————
Note: Check How much Percentage is /var/tmp used? In this case, 30% is used.
Most of the time, it is smm.log and auto_file_deletion_log.txt that eats up the space.
Here is an example. After you have run “ df –h “ , issue the following command to see which files are eating the space (below output shows SMM.LOG is eating 1.3M of space)
Linux(debug)# ls -lSh
total 2.2M
-rw-rw-rw- 1 root root 1.3M Apr 22 14:20 smm.log <<<———
-rw-rw-rw- 1 root root 245K Apr 21 03:28 afm_srv_15.log.0
-rw-r–r– 1 root root 196K Apr 8 11:47 sam_bridge_boot.log
-rw-rw-rw- 1 root root 63K Apr 22 14:18 afm_srv_15.log
-rw-rw-rw- 1 root root 60K Apr 8 11:50 syslogd_errlog.4886_4886
-rw-rw-rw- 1 root root 21K Apr 8 11:47 fm_debug.log.vdc_1.4919
-rw-rw-rw- 1 root root 18K Apr 8 11:49 fcoe_mgr_init.log
-rw-rw-rw- 1 root root 5.2K Apr 8 11:49 first_setup.log
-rwxr-xr-x 1 root root 4.4K Apr 8 12:25 iptab.sh
Since the files are in /var/tmp they are assumed to be safe to be deleted. Rather than fully deleting the files, it is safer to echo 0 into the file to empty and reduce the file size, this will ensure any service that is dependent upon being able to write to this log file still can.
Linux(debug)# echo 0 > /var/tmp/smm.log
Linux(debug)# echo 0 > /var/tmp/auto_root_file_deletion_log.txt
After this, you can df -h again to confirm that /var/tmp is now less than 10% utilzation. The upgrade should now be able to proceed.
++
After this workaround, I waited for 10 mins and still the FI is stuck at 65% and I reboot the FI and waited to 10 more mins after it came up – No Luck. So, I went ahead and re-activated the FI firmware image to 2.2(6c) with force option and the upgrade went successful.
Loading Debug Plugin in UCS
=======================
First Get the debug Plugin from : https://cspg-releng.cisco.com/
Note: You will need to select the appropriate version for the Plugin.
Below is the example for 2.2(6c) version
SELECT and CLICK on 2.2.6 Elcap_MR5 UNDER Release Builds on the lefthand side (Dont select Commits)
NOW You will select FCSc and click on BUNDLES and then you will select DEBUG.
Note: You can upload this image to UCSM same as you upload Upgrade Images. Once you uplaod it to UCSM, login to CLI and then Perfomr the following steps
Load debug plugin
FI-A(local-mgmt)# copy debug_plugin/ucs-dplug.5.0.3.N2.2.26c.gbin x
FI-A(local-mgmt)# load x
###############################################################
Warning: debug-plugin is for engineering internal use only!
For security reason, plugin image has been deleted.
###############################################################
Successfully loaded debug-plugin!!!
Detail List of Commands:
+++++++++++++++++++++
Connect Local-mgmt
FI-A(local-mgmt)# copy debug_plugin/ucs-dplug.5.0.3.N2.2.26c.gbin x
FI-A(local-mgmt)# load x
Linux(debug)# cd /var/tmp
Linux(debug)# df -h
Linux(debug)# ls -lSh
Linux(debug)# echo 0 > /var/tmp/smm.log
Linux(debug)# echo 0 > /var/tmp/auto_root_file_deletion_log.txt
CHECK THE HARDWARE VERSION :
+++++++++++++++++++++++++++++++
5596# show sprom sup | inc H/W
If hardware version in 1.1:
Copy dplugin and update file, Load Plugin
Copy the ucd-update.tar to the bootflash
Copy the debug plugin to the bootflash
Load the image via the load command
>>You might have to call support for the update.tar script
Linux(debug)# cp /bootflash/ucd-update.tar /tmp
Linux(debug)# cd /tmp/
Linux(debug)# tar -xvf ucd-update.tar
ucd.bin
Linux(debug)#
Step 2
———————————-
Run the ucd.bin file:
Linux(debug)# ./ucd.bin
You will see the prompt about updated version 1.1.
SAFESHUT:
++++++++++
>>Have to contact support for the safeshut.tar script
Copy tftp ://10.96.60.154/safeshut.tar workspace://safeshut.tar
Untar safeshut.tar (tar -xvf safeshut.tar)
Linux(debug)# ./safeshut.sh
md5sum for ./klm_sup_ctrl_mc.klm does not match – corrupted tarball?
Linux(debug)# ./safeshut.sh
md5sum for ./rebootsys.bin does not match – corrupted tarball?
Cd bootflash/ ————————————>Checks if the safeshut is updated ,look for the change in time of reboot it should be the current time
Cd /sbin
ls -la |grep “reboot”LS
Cd/bootflash
Ls -la |grep -I “klm_sup_ctrl_mc.klm”
Repeat for the peer FI
OPTIONAL:
++++++++++
echo 7 7 7 7 > /proc/sys/kernel/printk
Linux(debug)# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 300M 255M 46M 85% /
none 2.0M 4.0K 2.0M 1% /post
none 300M 255M 46M 85% /var
none 3.0G 1.1G 2.0G 35% /isan
none 600M 180M 421M 30% /var/tmp <————
Note: Check How much Percentage is /var/tmp used? In this case, 30% is used.
Most of the time, it is smm.log and auto_file_deletion_log.txt that eats up the space.
Here is an example. After you have run “ df –h “ , issue the following command to see which files are eating the space (below output shows SMM.LOG is eating 1.3M of space)
Linux(debug)# ls -lSh
total 2.2M
-rw-rw-rw- 1 root root 1.3M Apr 22 14:20 smm.log <<<———
-rw-rw-rw- 1 root root 245K Apr 21 03:28 afm_srv_15.log.0
-rw-r–r– 1 root root 196K Apr 8 11:47 sam_bridge_boot.log
-rw-rw-rw- 1 root root 63K Apr 22 14:18 afm_srv_15.log
-rw-rw-rw- 1 root root 60K Apr 8 11:50 syslogd_errlog.4886_4886
-rw-rw-rw- 1 root root 21K Apr 8 11:47 fm_debug.log.vdc_1.4919
-rw-rw-rw- 1 root root 18K Apr 8 11:49 fcoe_mgr_init.log
-rw-rw-rw- 1 root root 5.2K Apr 8 11:49 first_setup.log
-rwxr-xr-x 1 root root 4.4K Apr 8 12:25 iptab.sh
Since the files are in /var/tmp they are assumed to be safe to be deleted. Rather than fully deleting the files, it is safer to echo 0 into the file to empty and reduce the file size, this will ensure any service that is dependent upon being able to write to this log file still can.
Linux(debug)# echo 0 > /var/tmp/smm.log
Linux(debug)# echo 0 > /var/tmp/auto_root_file_deletion_log.txt
After this, you can df -h again to confirm that /var/tmp is now less than 10% utilzation. The upgrade should now be able to proceed.
++ After this workaround, I waited for 10 mins and still the FI is stuck at 65% and I reboot the FI and waited to 10 more mins after it came up – No Luck. So, I went ahead and re-activated the FI firmware image to 2.2(6c) with force option and the upgrade went successful.
Loading Debug Plugin in UCS
=======================
First Get the debug Plugin from : https://cspg-releng.cisco.com/
Note: You will need to select the appropriate version for the Plugin.
Below is the example for 2.2(6c) version
SELECT and CLICK on 2.2.6 Elcap_MR5 UNDER Release Builds on the lefthand side (Dont select Commits)
NOW You will select FCSc and click on BUNDLES and then you will select DEBUG.
Note: You can upload this image to UCSM same as you upload Upgrade Images. Once you uplaod it to UCSM, login to CLI and then Perfomr the following steps
Load debug plugin
FI-A(local-mgmt)# copy debug_plugin/ucs-dplug.5.0.3.N2.2.26c.gbin x
FI-A(local-mgmt)# load x
###############################################################
Warning: debug-plugin is for engineering internal use only!
For security reason, plugin image has been deleted.
###############################################################
Successfully loaded debug-plugin!!!
VMwareBlogs 