Skip to main content

Cisco Catalyst 4500 Series Zero-Downtime IOS Upgrade Process for Supervisor Engine 7-E, 7L-E, 6L-E and V-10GE Redundant Configurations

Article Reads:158786

The Supervisor Engine is the heart of the Cisco Catalyst 4500 and 6500 series switches. It is the equivalent of an engine in a car. Every packet that enters or exits your switch passes through the Supervisor Engine and, naturally, the Cisco IOS is the software that brings this beast to life.

Cisco makes an enormous effort to bring new features to its products through new versions of its popular IOS software, especially for the enterprise-class series switches such as the 4500 and 6500 Catalyst switches. Unlike other vendor operating systems, it is always advisable to keep your Cisco Catalyst operating system up to date by fitting it with the latest IOS image.

To be eligible to download an IOS image from Cisco’s website, one must have a valid contract support with Cisco Systems. This contract not only makes you eligible for all software upgrades for the duration of your contract, but also binds Cisco in delivering Top-Class support for your covered device(s).

Note: Information on installing the Cisco 4507R-E/4507R+E Catalyst Switch, Supervisor Engines, Line cards and power supplies can be found in our following articles which, combined, contain over 30 pictures of all switch components, including the passive backplane, fantray and more:

Are Network Services Affected During Or After The Catalyst Supervisor Engine(s) Upgrade?

This is perhaps one of the most frequently asked question that troubles engineers, administrators and IT Managers who have not dealt with the Supervisor Engine upgrade process before.  Who wouldn’t be? This is, after all, the network backbone switch and when it comes to large campus, Enterprise networks or networks that operate around the clock, downtime is not an option.

At this point, we should note that when upgrading a Supervisor Engine the engine must be reloaded (rebooted) in order for the system to load the new IOS image. For the Cisco Catalyst 4503 and 4506 series, which can only accept a single Supervisor Engine, this means network service interruption is unavoidable as the single Supervisor Engine must reboot. On the other hand, the Cisco Catalyst 4507R and 4510R series are capable of accepting up to two Supervisor Engines (hence the ‘R’ – Redundancy), therefore in case of a 4507R/4510R with two Supervisor Engines installed, IOS upgrades can be performed with guaranteed Zero-Service-Interruption

Note: The process we are about to describe was performed on a live 4507R+E with two Supervisor Engines 7L-E, on a network of 120+ users and 11 servers connected to our 4507R+E via dual 10Gbps fiber optic links (one 10Gbps link on each Supervisor Engine).

Quick Overview Of The Supervisor Engine Upgrade Process

Before we dive into the upgrade process, let’s take a quick look at the steps to be followed. This will help understand the process and caveats of each step.

When upgrading a system with redundant Supervisor Engines the upgrade process has to be performed in a specific way as each Supervisor Engine is upgraded in turn.  

Following is a brief overview of the upgrade steps:

  • Load the new IOS image on to the Active Supervisor Engine (SE1)
  • Copy IOS image to Standby Supervisor Engine (SE2)
  • Configure Supervisor Engines to load the new image upon reboot
  • Set Configuration-Register variable to ensure newest image is loaded upon bootup
  • Force reload of Standby Supervisor Engine (SE2) & Switchover to Standby Supervisor Engine (SE2). This now becomes the new Active Supervisor Engine
  • Force reload of previously Active Supervisor Engine (SE1).

Step 1: Loading The New IOS Image On To The Active Supervisor Engine (SE1)

The first step is to copy the new IOS image on to the active Supervisor Engine (SE1). For this, a TFTP server is required for the file transfer. Users can download a selection of Free TFTP Servers from our FTP/TFTP Servers & Clients download section.

Once the TFTP server is ready, we issue the necessary command to initiate the file transfer:

4507R# copy tftp bootflash:
Address or name of remote host []? 10.0.0.76
Source filename []? cat4500e-universal.SPA.03.04.00.SG.151-2.SG.bin
Destination filename [cat4500e-universal.SPA.03.04.00.SG.151-2.SG.bin]?
Accessing tftp://10.0.0.76/cat4500e-universal.SPA.03.04.00.SG.151-2.SG.bin...
Loading cat4500e-universal.SPA.03.04.00.SG.151-2.SG.bin from 10.0.0.36 (via Vlan2): !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[OK - 119576292 bytes]
119576292 bytes copied in 356.708 secs (335222 bytes/sec)

Note that the new IOS image is saved to the bootflash: file system.  Cisco Supervisor Engines use the bootflash: file system rather than the flash: file system which most of us are used to.

If in doubt, simply make use of the show file system command that will reveal the file systems on your Catalyst switch:

4507R# show file system
File Systems:
     Size(b)          Free(b)      Type  Flags  Prefixes
*    831541248     718979072      disk     rw   bootflash:
     141433856     139310080       rom     ro   crashinfo:
      16915456      16660992      disk     rw   kinfo:
           -           -          disk     rw   slot0:
           -           -          disk     rw   usb0:
           -           -        opaque     rw   system:
           -           -        opaque     rw   tmpsys:
        524284        523248     flash     rw   cat4000_flash:
        524284        510196     nvram     rw   slavenvram:
     820875264     708313088     flash     rw   slavebootflash:
        524284        523248     flash     rw   slavecat4000_flash:
           -           -         flash     rw   slaveslot0:
           -           -         flash     rw   slaveusb0:
     139940864     137817088    opaque     ro   slavecrashinfo:
           -           -        opaque     rw   slavercsf:
      16915456      16660992     flash     rw   slavekinfo:
           -           -        opaque     rw   null:
           -           -        opaque     ro   tar:
           -           -       network     rw   tftp:
           -           -        opaque     wo   syslog:
        524284        510196     nvram     rw   nvram:
           -           -       network     rw   rcp:
           -           -       network     rw   http:
           -           -       network     rw   ftp:
           -           -        opaque     ro   cns:
           -           -        opaque     rw   revrcsf:

Step 2: Copy IOS Image To Standby Supervisor Engine (SE2)

Once the IOS image is loaded on to the active Supervisor Engine (SE1), it must be copied to the standby Supervisor Engine (SE2). For this, we use the copy bootflash: slavebootflash: command. Note that the slavebootflash: file system refers to the bootflash: of the standby Supervisor Engine, regardless of which physical engine is on standby mode.

4507R# copy bootflash: slavebootflash:
Source filename []? cat4500e-universal.SPA.03.04.00.SG.151-2.SG.bin
Destination filename [cat4500e-universal.SPA.03.04.00.SG.151-2.SG.bin]?
Copy in progress...CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
119576292 bytes copied in 99.404 secs (1202932 bytes/sec)

4507R#

To verify the image is correctly loaded on both Supervisor Engine bootflash systems, use the show bootflash & show slavebootflash commands:

4507R# show bootflash:
-#- --length-- ---------date/time--------- path
  1  112337548 Feb 26 2013 08:44:27 +00:00 cat4500e-universal.SPA.03.03.00.SG.151-1.SG.bin
  2  119576292 Mar 21 2013 03:07:21 +00:00 cat4500e-universal.SPA.03.04.00.SG.151-2.SG.bin
557035520 bytes available (232263680 bytes used)
Filesystem: bootflash
Mounted: Unknown

4507R# show slavebootflash:
-#- --length-- ---------date/time--------- path
  1  112337548 Feb 26 2013 06:46:26 +00:00 cat4500e-universal.SPA.03.03.00.SG.151-1.SG.bin
  2  119576292 Mar 21 2013 01:09:33 +00:00 cat4500e-universal.SPA.03.04.00.SG.151-2.SG.bin
546914304 bytes available (232263680 bytes used)

Note:  Notice that the first IOS image listed is the previous version (03.03.00.SG.151-1.SG). We’ll need to keep this information in mind for our next step.

Step 3: Configure Supervisor Engines To Load The New Image Upon Reboot

In our next step, we configure the active Supervisor Engine to load the new IOS image when it reboots. This is easily done using the boot system flash command as shown below:

4507R(config)# boot system flash bootflash:cat4500e-universal.SPA.03.04.00.SG.151-2.SG.bin

It is highly likely the boot system flash bootflash command already exists in the system’s configuration for the previous IOS image, so we’ll need to remove the command from the configuration to ensure the newest IOS image we just uploaded (03.04.00.SG.151-2.SG) is the only one referenced:

4507R(config)# no boot system flash bootflash:cat4500e-universal.SPA.03.03.00.SG.151-1.SG.bin

Next, we must ensure the configuration is saved to the startup-configuration:

4507R# wr mem
Building configuration...
% VRF table-id 0 not activeCompressed configuration from 12839 bytes to 3791 bytes[OK]
*Mar 21 01:15:28.356: %C4K_REDUNDANCY-5-CONFIGSYNC: The private-config has been successfully synchronized to the standby supervisor
*Mar 21 01:15:29.098: %C4K_REDUNDANCY-5-CONFIGSYNC: The startup-config has been successfully synchronized to the standby supervisor

Readers wondering about the % VRF table-id 0 not activeCompressed message when saving the running-configuration should not be alarmed as this is a cosmetic bug and can be safely ignored. If no such message was reported when saving the configuration, it means that the IOS currently running does not have this cosmetic bug.

On another note, every time we save our configuration to startup-config the system will immediately synchronize the configuration changes to the standby supervisor. When this happens, we’ll receive a message, similar to the one show above, confirming the synchronization has been successful.

Step 4: Set Configuration-Register Variable To Ensure Newest Image Is Loaded Upon Bootup

Cisco Supervisor Engines, by factory default, have their configuration register set to 0x2101. While this value is a combination of settings, we will focus on two specific values: 0x2101 & 0x2102.  The value 0x2101 instructs the system to boot the first system image in the onboard flash memory (bootflash). This is usually the oldest image in the flash. The value of 0x2102 instructs the system to use the image specified in the BOOT environment variable, which is essentially whatever was specified in the previous step (No.3) using the boot system flash bootflash: command.

To view the environment variables on both Supervisor Engines, use the show bootvar command:

4507R# show bootvar
BOOT variable = bootflash:cat4500e-universal.SPA.03.04.00.SG.151-2.SG.bin,1;
CONFIG_FILE variable does not exist
BOOTLDR variable does not exist
Configuration register is 0x2101
Standby BOOT variable = bootflash:cat4500e-universal.SPA.03.04.00.SG.151-2.SG.bin,1;
Standby CONFIG_FILE variable does not exist
Standby BOOTLDR variable does not exist
Standby Configuration register is 0x2101

Notice how the newly uploaded IOS image cat4500e-universal.SPA.03.04.00.SG.151-2.SG.bin is set for the BOOT variable, however the Configuration register value is 0x2101, which means it will force the Supervisor Engine to boot the first image it is going to find on the bootflash. Which image is that?  Let’s refresh our memory:

4507R# show bootflash: all
-#- --length-- ---------date/time--------- path
  1  112337548 Feb 26 2013 08:44:27 +00:00 cat4500e-universal.SPA.03.03.00.SG.151-1.SG.bin
  2  119576292 Mar 21 2013 03:07:21 +00:00 cat4500e-universal.SPA.03.04.00.SG.151-2.SG.bin

4507R# show slavebootflash:
-#- --length-- ---------date/time--------- path
  1  112337548 Feb 26 2013 06:46:26 +00:00 cat4500e-universal.SPA.03.03.00.SG.151-1.SG.bin
  2  119576292 Mar 21 2013 01:09:33 +00:00 cat4500e-universal.SPA.03.04.00.SG.151-2.SG.bin

As we can see, despite the boot system flash command being correctly set to load the second IOS image, the configuration register setting of 0x2101 will force the Supervisor Engine to boot the first image found, that is: cat4500e-universal.SPA.03.03.00.SG.151-1.SG.bin.

To overcome this problem, we have two options: 

1) Delete all older IOS images from the Bootflash 

2) Set the configuration register to 0x2102

We decided to set the configuration register just to be on the safe side:

4507R(config)# config-register 0x2102

*Mar 21 01:19:55.542: %C4K_REDUNDANCY-5-CONFIGSYNC: The config-reg has been successfully synchronized to the standby supervisor
 
4507R# wr mem
Building configuration...
% VRF table-id 0 not activeCompressed configuration from 12849 bytes to 3791 bytes[OK]
*Mar 21 01:20:08.352: %C4K_REDUNDANCY-5-CONFIGSYNC: The private-config has been successfully synchronized to the standby supervisor
*Mar 21 01:20:09.091: %C4K_REDUNDANCY-5-CONFIGSYNC: The startup-config has been successfully synchronized to the standby supervisor

Take note that the system is confirming that both configuration register and startup-config have been successfully synchronized with the standby supervisor.

…and there’s that cosmetic bug again :)

Step 5: Force Reload Of Standby Supervisor (SE2) & Switchover To Standby Supervisor Engine (SE2)

At this point we are ready to force the standby Supervisor Engine(SE2) to reload. Once this happens, the Supervisor Engine (SE2) will load the new IOS. Once we confirm the new IOS is loaded, we can then make the standby Supervisor Engine (SE2) the active Supervisor Engine.

To force the reload of the standby Supervisor Engine (SE2), use the redundancy reload peer command:

4507R# redundancy reload peer
Reload peer [confirm]
4507R#
*Mar 21 01:21:02.318: %RF-5-RF_RELOAD: Peer reload. Reason: Unknown Reason
*Mar 21 01:21:05.314: %C4K_REDUNDANCY-3-COMMUNICATION: Communication with the peer Supervisor has been lost
*Mar 21 01:21:05.327: %C4K_REDUNDANCY-3-SIMPLEX_MODE: The peer Supervisor has been lost

The second (standby) Supervisor Engine is now restarting. This process will take a couple of minutes and will have no negative impact on the 4507R switch.

Once the IOS has loaded and the restart process is complete we will receive a message similar to the following:

*Mar 21 01:24:54.312: %C4K_REDUNDANCY-6-DUPLEX_MODE: The peer Supervisor has been detected
*Mar 21 01:25:36.066: %C4K_IOSMODPORTMAN-6-MODULEONLINE: Module 4 (WS-X45-SUP7L-E S/N: CAT1714L4T4 Hw: 1.1) is online
*Mar 21 01:25:36.094: %C4K_REDUNDANCY-6-MODE: ACTIVE supervisor initializing for sso mode
*Mar 21 01:25:36.344: %C4K_REDUNDANCY-3-COMMUNICATION: Communication with the peer Supervisor has been established
*Mar 21 01:25:37.098: %C4K_REDUNDANCY-6-MODE: ACTIVE supervisor initializing for sso mode

The above messages indicate that the standby Supervisor Engine has successfully completed its reboot and is fully synchronized with the active Supervisor Engine. The system shows the product ID (WS-X45-SUP7L-E), serial number and hardware revision of the Supervisor Engine that has just established communication with the active Supervisor Engine.

To verify that the standby Supervisor Engine is running the latest and greatest IOS image we just loaded, use the show module command:

4507R# show module      
Chassis Type : WS-C4507R+E
Power consumed by backplane : 40 Watts
Mod Ports Card Type                              Model              Serial No.
---+-----+--------------------------------------+------------------+-----------
 1    48  10/100/1000BaseT Premium POE E Series  WS-X4748-RJ45V+E   CAT1754L4C7
 2    48  10/100/1000BaseT Premium POE E Series  WS-X4648-RJ45V+E   JAE171515SY
 3     6  Sup 7L-E 10GE (SFP+), 1000BaseX (SFP)  WS-X45-SUP7L-E     CAT1712L2X5
 4     6  Sup 7L-E 10GE (SFP+), 1000BaseX (SFP)  WS-X45-SUP7L-E     CAT1714L4T4
 5    48  10/100/1000BaseT Premium POE E Series  WS-X4648-RJ45V+E   JAE170525D3

 M MAC addresses                    Hw  Fw           Sw               Status
--+--------------------------------+---+------------+----------------+---------
 1 4c4e.352e.a2cc to 4c4e.352e.a2fb 1.3                               Ok       
 2 0006.f6e9.7520 to 0006.f6e9.754f 3.1                               Ok      
 3 0006.f620.2e80 to 0006.f620.2e85 1.1 15.0(1r)SG3  03.03.00.SG      Ok      
 4 0006.f620.2e86 to 0006.f620.2e8b 1.1 15.0(1r)SG3  03.04.00.SG      Ok      
 5 0006.f673.51d4 to 0006.f673.5203 3.1                               Ok      

Mod  Redundancy role     Operating mode      Redundancy status
----+-------------------+-------------------+----------------------------------
 3   Active Supervisor   SSO                 Active                           
 4   Standby Supervisor  SSO                 Standby hot     

The show module command provides a generous amount of information including all line cards installed, Supervisor Engine models, IOS software versions, serial numbers, operational status and redundancy mode (SSO by default).

We’ve highlighted our standby Supervisor Engine which happens to be in slot No.4. Notice the software image the system is reporting for both Supervisor Engines. Our current active Supervisor Engine is running version 03.03.00.SG, while our standby Supervisor Engine is running version 03.04.00.SG – the new IOS loaded!

We’ve now confirmed that the standby Supervisor Engine has loaded the new IOS and is fully operational, waiting eagerly to manage our 4507R switch and all its data!!

Step 6: Force Reload Of Previously Active Supervisor Engine (SE1)

On our final step, we will need to force the reload of the active Supervisor Engine (SE1) so it too can boot using the new IOS image. Forcing a Supervisor Engine switch over is an experience when you consider what’s really happening inside the switch at that moment. During the process of the switchover we don’t get LEDs lighting up, flashing like crazy. The SUP ACTIVE LED will simply switch off from the current active Supervisor Engine and switch on on our previously standby Supervisor Engine, indicating it is now the new active engine.

To initiate the Supervisor Engine switchover, use the redundancy force-switchover command as shown below:

4507R# redundancy force-switchover
This will reload the active unit and force switchover to standby[confirm]
Preparing for switchover..
*Mar 21 01:27:57.007: %SYS-5-SWITCHOVER: Switchover requested by Virtual Exec. Reason: Stateful Switchover.

Once the command is entered and we’ve confirmed by hitting ENTER, we will lose our telnet session to the 4507R. This is normal expected behaviour – do not be alarmed!  Apart from the switch cutting our telnet session, users will not notice any service disruption – the switchover will be completely transparent to them no matter the network load during the switchover.

To reconnect to the switch, simply telnet back into the same IP address. If connected via console cable it will be necessary to connect it to the new active supervisor engine in order to continue controlling the switch.

Engineers who would like to monitor, via telnet or direct console cable connection (to the new active engine), the reload progress of the Supervisor Engine can use the show module command. By typing the command we will see the 4507R identifying a Supervisor Engine in slot 3 (that’s SE1 that is reloading), however, further down we will see that no information about the engine’s MAC address or IOS software is provided due to the fact that it has not fully booted into its new IOS. In addition the redundancy status of the first engine is Disabled – an expected result since the Supervisor Engine (SE1) has not booted yet.

4507R# show module
Chassis Type : WS-C4507R+E
Power consumed by backplane : 40 Watts
Mod Ports Card Type                              Model              Serial No.
---+-----+--------------------------------------+------------------+-----------
 1    48  10/100/1000BaseT Premium POE E Series  WS-X4748-RJ45V+E   CAT1754L4C7
 2    48  10/100/1000BaseT Premium POE E Series  WS-X4648-RJ45V+E   JAE171515SY
 3        Supervisor
 4     6  Sup 7L-E 10GE (SFP+), 1000BaseX (SFP)  WS-X45-SUP7L-E     CAT1714L4T4
 5    48  10/100/1000BaseT Premium POE E Series  WS-X4648-RJ45V+E   JAE170525D3
 
M MAC addresses                    Hw  Fw           Sw               Status
--+--------------------------------+---+------------+----------------+---------
 1 4c4e.352e.a2cc to 4c4e.352e.a2fb 1.3                               Ok      
 2 0006.f6e9.7520 to 0006.f6e9.754f 3.1                               Ok      
 3 Unknown                              Unknown      Unknown          Other   
 4 0006.f620.2e86 to 0006.f620.2e8b 1.1 15.0(1r)SG3  03.04.00.SG      Ok      
 5 0006.f673.51d4 to 0006.f673.5203 3.1                               Ok      

Mod  Redundancy role     Operating mode      Redundancy status
----+-------------------+-------------------+----------------------------------
 3   Standby Supervisor  SSO                 Disabled                         
 4   Active Supervisor   SSO                 Active    

At this point, we can continue to issue the show module command and monitor the output changes. At some point the system will show the Supervisor Engine (SE1) loaded with the new IOS, however, the redundancy status will go through the following phases until it is ready (Standby hot):

Standby Supervisor Redundancy Status Cycle:

  1. Disabled
  2. In progress to Standby cold
  3. Standby cold
  4. In progress to Issu negotiation la
  5. In progress to Standby config
  6. In progress to Standby bulk  
  7. Standby hot

When the Supervisor Engine reaches Standby hot status, it is ready to take over in the event the active Supervisor Engine fails.

Following is the expected output when the Supervisor Engine IOS is loaded and fully synced with the active Supervisor Engine:

4507R# show module
Chassis Type : WS-C4507R+E
Power consumed by backplane : 40 Watts

Mod Ports Card Type                              Model              Serial No.
---+-----+--------------------------------------+------------------+-----------
 1    48  10/100/1000BaseT Premium POE E Series  WS-X4748-RJ45V+E   CAT1754L4C7
 2    48  10/100/1000BaseT Premium POE E Series  WS-X4648-RJ45V+E   JAE171515SY
 3     6  Sup 7L-E 10GE (SFP+), 1000BaseX (SFP)  WS-X45-SUP7L-E     CAT1712L2X5
 4     6  Sup 7L-E 10GE (SFP+), 1000BaseX (SFP)  WS-X45-SUP7L-E     CAT1714L4T4
 5    48  10/100/1000BaseT Premium POE E Series  WS-X4648-RJ45V+E   JAE170525D3

 M MAC addresses                    Hw  Fw           Sw               Status
--+--------------------------------+---+------------+----------------+---------
 1 4c4e.352e.a2cc to 4c4e.352e.a2fb 1.3                               Ok     
 2 0006.f6e9.7520 to 0006.f6e9.754f 3.1                               Ok     
 3 0006.f620.2e80 to 0006.f620.2e85 1.1 15.0(1r)SG3  03.04.00.SG      Ok     
 4 0006.f620.2e86 to 0006.f620.2e8b 1.1 15.0(1r)SG3  03.04.00.SG      Ok     
 5 0006.f673.51d4 to 0006.f673.5203 3.1                               Ok     

Mod  Redundancy role     Operating mode      Redundancy status
----+-------------------+-------------------+----------------------------------
 3   Standby Supervisor  SSO                Standby hot                     
 4   Active Supervisor   SSO                 Active

Summary

This article explained the importance of upgrading the Cisco Supervisor Engine IOS to the latest version, the process that needs to be followed for the upgrade and how the Catalyst switch’s operations are affected during or after the upgrade. We also covered the IOS upgrade process for redundant Supervisor Engine configurations, how to force a Supervisor Engine switchover and monitoring the Supervisor Engine bootup and sync process.

More articles covering Cisco Catalyst switches and LAN switching technologies can be found in our Cisco Knowledgebase Switching category.

Your IP address:

3.231.222.84

FREE Hyper-V & VMware Backup

FREE Hyper-V & VMware Backup

Wi-Fi Key Generator

Generate/Crack any
WEP, WPA, WPA2 Key!

Network and Server Monitoring

Network and Server Monitoring

Follow Firewall.cx

Cisco Password Crack

Decrypt Cisco Type-7 Passwords on the fly!

Decrypt Now!

Bandwidth Monitor

Bandwidth Monitor

Free PatchManager

Free PatchManager

EventLog Analyzer

ManageEngine Eventlog Analyzer

Firewall Analyzer

zoho firewall analyzer