Nexus 2248TP FEX connected to a Nexus 7000: part 1 basic connection

Cisco published a configuration guide for connecting a Nexus 2248TP FEX to a Nexus 7000. I’ll explain the configuration process to configure the FEX for basic connectivity.

A FEX is a Nexus 2000 series switch. In a very high level explanation, this switch is a switchport module in a separate 1U chassis, which is configured and controlled from a Nexus 5000/7000. There is 1 drawback: the switchports on the FEX can only be used for host ports. It’s not possible to connect other switches to a FEX port because of the fact that BPDUguard is enabled by default. It’s not possible to disable BPDUguard. Switchports on the FEX can be used for layer 2 and layer 3 connections. For more information about the FEX itself I’ll refer to this link

All configuration is done on the Nexus 7000 with NX-OS 6.0(1). In this scenario, the 2248TP FEX is connected to switchport ethernet 1/1  (10GE port) of the Nexus 7000 with a twinax cable.

First, let’s configure a switchport for the FEX:
switch(config)# int ethernet 1/1
switch(config-if)# switchport mode fex-fabric
Error: feature-set fex is not enabled
Okay, we have to install (!) and enable the fex feature before we can continue
switch(config)# install feature-set fex
switch(config)# feature-set fex
And try again to configure the fex-fabric mode:
switch(config)# int ethernet 1/1
switch(config-if)# switchport mode fex-fabric
switch(config-if)# fex associate 101
switch(config-if)# no shut
This fex associate number defines the physical FEX to the FEX configuration. The range is 101 – 199. In this case, the FEX connected to switchport 1/1 will be numbered as 101. It’s possible to assign a FEX serial number to the FEX configuration. With this configuration, only the FEX with that particular serialnumber can be connected to these ports, and receives this configuration.
switch(config-if)# sh int status

Port          Name               Status    Vlan      Duplex  Speed   Type
mgmt0         --                 connected routed    full    100     --
Eth1/1        --                 intFailEr 1         full    auto    SFP-H10GB-A
As you see, the switchport is in error mode. Let’s first finish the FEX configuration before we troubleshoot the connection.
switch(config)# fex 101
switch(config-fex)# description FEX-test
switch(config-fex)# type n2248t
Let’s check the logging:
switch(config)# sh logging
2012 Feb 15 08:42:50 switch %ETHPORT-5-IF_DOWN_ERROR_DISABLED: Interface Ethernet1/1 is down (Error disabled. Reason:Port is not a port-channel member)
The error message self-explaining: make the switchport member of a port channel:
switch(config)# int eth1/1
switch(config-if)# channel-group 1
command failed: port not compatible [Ethernet Layer]

** You can use force option to override the port's parameters
** (e.g. "channel-group X force")
** Use "show port-channel compatibility-parameters" to get more information on failure

switch(config-if)# channel-group 1 force
The configuration looks like:
interface port-channel1
  switchport mode fex-fabric
  fex associate 101
Now it’s time to check the portchannel:
switch(config-if)# sh interface port-channel 1 fex-intf
Fabric           FEX
Interface        Interfaces
Grab a coffee and wait a few minutes.
switch(config)# sh fex detail
FEX: 101 Description: FEX-test   state: Online Sequence
  FEX version: 6.0(1) [Switch version: 6.0(1)]
  FEX Interim version: 6.0(0.4)
  Switch Interim version: 6.0(1)
  Extender Model: N2K-C2248TP-1GE,  Extender Serial: FOC1549621C
  Part No: 73-13232-01
  Card Id: 99, Mac Addr: d8:24:bd:fc:25:02, Num Macs: 64
  Module Sw Gen: 12594  [Switch Sw Gen: 21]
pinning-mode: static    Max-links: 1
  Fabric port for control traffic: Eth1/1
  Fabric interface state:
    Po1 - Interface Up. State: Active
    Eth1/1 - Interface Up. State: Active
  Fex Port        State  Fabric Port
       Eth101/1/1  Down         Po1
       Eth101/1/2  Down         Po1
       Eth101/1/3  Down         Po1
       Eth101/1/4  Down         Po1
       Eth101/1/5  Down         Po1
       Eth101/1/6  Down         Po1
       Eth101/1/7  Down         Po1
       Eth101/1/8  Down         Po1
       Eth101/1/9  Down         Po1
      Eth101/1/10  Down         Po1
      Eth101/1/11  Down         Po1
      Eth101/1/12  Down         Po1
      Eth101/1/13  Down         Po1
      Eth101/1/14  Down         Po1
      Eth101/1/15  Down         Po1
      Eth101/1/16  Down         Po1
      Eth101/1/17  Down         Po1
      Eth101/1/18  Down         Po1
      Eth101/1/19  Down         Po1
      Eth101/1/20  Down         Po1
      Eth101/1/21  Down         Po1
      Eth101/1/22  Down         Po1
      Eth101/1/23  Down         Po1
      Eth101/1/24  Down         Po1
      Eth101/1/25  Down         Po1
      Eth101/1/26  Down         Po1
      Eth101/1/27  Down         Po1
      Eth101/1/28  Down         Po1
      Eth101/1/29  Down         Po1
      Eth101/1/30  Down         Po1
      Eth101/1/31  Down         Po1
      Eth101/1/32  Down         Po1
      Eth101/1/33  Down         Po1
      Eth101/1/34  Down         Po1
      Eth101/1/35  Down         Po1
      Eth101/1/36  Down         Po1
      Eth101/1/37  Down         Po1
      Eth101/1/38  Down         Po1
      Eth101/1/39  Down         Po1
      Eth101/1/40  Down         Po1
      Eth101/1/41  Down         Po1
      Eth101/1/42  Down         Po1
      Eth101/1/43  Down         Po1
      Eth101/1/44  Down         Po1
      Eth101/1/45  Down         Po1
      Eth101/1/46  Down         Po1
      Eth101/1/47  Down         Po1
      Eth101/1/48  Down         Po1
02/15/2012 08:55:16.78268: Module register received
02/15/2012 08:55:16.79093: Image Version Mismatch
02/15/2012 08:55:16.79695: Registration response sent
02/15/2012 08:55:16.79854: Requesting satellite to download image
02/15/2012 09:03:01.684836: Image preload successful.
02/15/2012 09:03:01.901494: Deleting route to FEX
02/15/2012 09:03:01.906381: Module disconnected
02/15/2012 09:03:01.910050: Module Offline
02/15/2012 09:03:01.918355: Deleting route to FEX
02/15/2012 09:03:01.923126: Module disconnected
02/15/2012 09:03:01.925539: Offlining Module
02/15/2012 09:04:14.606783: Module register received
02/15/2012 09:04:14.608763: Registration response sent
02/15/2012 09:04:15.411239: Module Online Sequence
In case you’ve a lot of FEX’s in your datacenter and you have to locate this particular FEX, it’s possible to start a beacon (blinking blue LED on the FEX) with this command:
switch(config)# fex 101
switch(config-fex)# beacon
 We’re done! In a upcoming blog post I’ll cover multiple uplinks to the FEX and the pinning configuration.