This section introduces you to the most common troubleshooting techniques for PEs in EoMPLS. You first learn the commands and outputs for the Cisco router PEs and then learn to troubleshoot Cisco 7600 series switches.
The first common step in troubleshooting problems is attempting to discover failure by verifying the status of a VC by issuing the show mpls l2transport vc command.
Three conditions must be met so that the VC is UP:
The disposition interfaces are programmed if the VC has been configured and the CE interface is UP.
If the IGP label exists, it can be implicit null in a back-to-back configuration.
The imposition interface is programmed if the disposition interface is programmed and you have a remote VC label and an IGP label (LSP to the peer).
If the status field is marked DOWN (that is, the VC is not ready to carry traffic between the two VC endpoints), as shown in the output of Example 7-34, execute the show mpls l2transport vc detail command seen in Example 7-35 for more in-depth information.
NewYork#show mpls l2transport vc Local intf Local circuit Dest address VC ID Status ------------- ----------------------- --------------- ---------- ---------- Et0/0 Ethernet 192.168.1.102 100 DOWN
NewYork#show mpls l2transport vc detail
Local interface: Et0/0 up, line protocol up, Ethernet up
Destination address: 192.168.1.102, VC ID: 100, VC status: down
Preferred path: not configured
Default path: active
Tunnel label: 16, next hop point2point
Output interface: Se5/0, imposed label stack {16 16}
Create time: 00:18:10, last status change time: 00:03:51
Signaling protocol: LDP, peer 192.168.1.102:0 up
MPLS VC labels: local 16, remote 16
Group ID: local 0, remote 0
MTU: local 1500, remote unknown
Remote interface description:
Sequencing: receive disabled, send disabled
VC statistics:
packet totals: receive 0, send 0
byte totals: receive 0, send 0
packet drops: receive 0, send 78
Table 7-4 describes some of the significant fields of the show mpls l2transport vc detail command output.
Destination address
| The IP address of the remote router specified for this VC as part of the mpls l2transport route or xconnect command.
|
VC ID
| The virtual circuit identifier assigned to the interface on the router.
|
VC status
| The status of the VC. The status can be one of the following:
|
UPThe VC is in a state in which it can carry traffic between the two VC endpoints. A VC is UP when both imposition and disposition interfaces are programmed.
| |
DOWNThe VC is not ready to carry traffic between the two VC endpoints.
| |
ADMIN DOWNA user has disabled the VC.
| |
Tunnel label
| An IGP label that routes the packet over the MPLS backbone to the destination router with the egress interface.
|
Output interface
| The interface on the remote router that has been enabled to transmit and receive Layer 2 packets.
|
Imposed label stack
| A summary of the MPLS label stack that directs the VC to the PE router.
|
Signaling protocol
| The type of protocol that sends the MPLS labels. The output also shows the status of the peer router.
|
MPLS VC labels
| The local VC label is a disposition label, which determines the egress interface of an arriving packet from the MPLS backbone. The remote VC label is a disposition VC label of the remote peer router.
|
MTU
| The maximum transmission unit specified for the local and remote interfaces.
|
Sequencing
| This field describes whether sequencing of out-of-order packets is enabled or disabled.
|
In the show mpls l2transport vc detail command output, pay attention to the remote unknown next to the local MTU. One of the possible causes is a remote interface down or an MTU mismatch. Verify to make sure that MTU on each side is the same. If an EoMPLS tunnel is still down after this and you cannot pass traffic, perform another check by issuing the show mpls forwarding-table command, as demonstrated in Example 7-36.
NewYork#show mpls forwarding-table Local Outgoing Prefix Bytes tag Outgoing Next Hop tag tag or VC or Tunnel Id switched interface 17 Untagged 10.1.1.0/24 0 Se5/0 point2point 18 Untagged 192.168.1.101/32 0 Se5/0 point2point 19 Untagged 192.168.1.102/32 0 Se5/0 point2point 20 Untagged l2ckt(100) 4592 Et0/0.100 point2point
The Untagged result in the Outgoing tag or VC field indicates that an MPLS label might not be exchanged between the PE and P (Denver) routers.
A couple possible causes exist. Either the mpls ip has not been enabled per interface, or CEF is disabled (or not enabled) on the P or PE router. To verify, issue the show mpls ldp discovery command, as in Example 7-37.
NewYork#show mpls ldp discovery
Local LDP Identifier:
192.168.1.103:0
Discovery Sources:
Targeted Hellos:
192.168.1.103 -> 192.168.1.102 (ldp): active/passive, xmit/recv
LDP Id: 192.168.1.102:0
The output shows whether you have a direct LDP session open between directly connected MPLS-enabled interfaces. By observing this behavior, you can conclude that MPLS indeed was not enabled per interface facing the core. To solve this problem, enable MPLS on an interface or check whether CEF is enabled.
Two common issues result when the circuit does not come up:
The remote port is down or not configured.
The MTU is mismatched.
Examples 7-38 and 7-39 display the output of the show mpls l2transport vc vciddetail command with the two conditions, respectively.
NewYork#show mpls l2transport vc 10 detail
Local interface: FastEternet0/0.10 up, line protocol up, Eth VLAN 10 up
Destination address: 192.168.1.102, VC ID: 10, VC status: down
Tunnel label: not ready
Output interface: unknown, imposed label stack {}
Create time: 22:31:53, last status change time: 04:02:56
Signaling protocol: LDP, peer 192.168.1.102:0 up
MPLS VC labels: local 19, remote unassigned
Group ID: local 0, remote unknown
MTU: local 1500, remote unknown
Remote interface description:
Sequencing: receive disabled, send disabled
VC statistics:
packet totals: receive 1650, send 1743
byte totals: receive 552557, send 550044
packet drops: receive 0, send 7
NewYork#show mpls l2transport vc 10 detail
Local interface: FastEternet0/0.10 up, line protocol up, Eth VLAN 10 up
Destination address: 192.168.1.102, VC ID: 10, VC status: down
Tunnel label: not ready
Output interface: unknown, imposed label stack {}
Create time: 22:36:10, last status change time: 00:00:20
Signaling protocol: LDP, peer 192.168.1.102:0 up
MPLS VC labels: local 19, remote 21
Group ID: local 0, remote 0
MTU: local 1500, remote 1000
Remote interface description: *** To SanFran ***
Sequencing: receive disabled, send disabled
VC statistics:
packet totals: receive 1880, send 1901
byte totals: receive 168476, send 155436
packet drops: receive 0, send 13
The highlighted portions of Examples 7-38 and 7-39 call attention to the faulty conditions. Compare the output to output of the same command in an operational environment from Example 7-40.
NewYork#show mpls l2transport vc 10 detail
Local interface: FastEternet0/0.10 up, line protocol up, Eth VLAN 10 up
Destination address: 192.168.1.102, VC ID: 10, VC status: up
Preferred path: not configured
Default path: active
Tunnel label: 17, next hop 10.1.1.202
Output interface: Et1/0, imposed label stack {17 21}
Create time: 23:06:37, last status change time: 00:30:47
Signaling protocol: LDP, peer 192.168.1.102:0 up
MPLS VC labels: local 19, remote 21
Group ID: local 0, remote 0
MTU: local 1500, remote 1500
Remote interface description: *** To SanFran ***
Sequencing: receive disabled, send disabled
VC statistics:
packet totals: receive 1683, send 1777
byte totals: receive 565455, send 563328
packet drops: receive 0, send 7
Example 7-41 presents the verification and configuration sequence of enabling MPLS on the Serial 5/0 interface.
NewYork#show mpls interfaces Interface IP Tunnel Operational Se5/0 No Yes No configure terminal interface serial 5/0 mpls ip NewYork#show mpls interfaces Interface IP Tunnel Operational Se5/0 Yes Yes Yes
Next, check whether MPLS is directly receiving labels and exchanging LDP between the PE and the P by reissuing the show mpls forwarding-table command. The output is provided in Example 7-42.
NewYork#show mpls forwarding-table Local Outgoing Prefix Bytes tag Outgoing Next Hop tag tag or VC or Tunnel Id switched interface 17 Pop tag 10.1.1.0/24 0 Se5/0 point2point 18 Pop tag 192.168.1.101/32 0 Se5/0 point2point 19 16 192.168.1.102/32 0 Se5/0 point2point 20 Untagged l2ckt(100) 5310 Et0/0.100 point2point
To ensure that LDP xmit/recv (transmit/receive) is occurring in both directions, use the show mpls ldp discovery command again, as in Example 7-43.
NewYork#show mpls ldp discovery
Local LDP Identifier:
192.168.1.103:0
Discovery Sources:
Interfaces:
Serial5/0 (ldp): xmit/recv
LDP Id: 192.168.1.101:0
Targeted Hellos:
192.168.1.103 -> 192.168.1.102 (ldp): active/passive, xmit/recv
LDP Id: 192.168.1.102:0
Example 7-44 shows that the VC is now ready and operational and should be able to send traffic from CE to CE.
NewYork#show mpls l2transport vc Local intf Local circuit Dest address VC ID Status ------------- ----------------------- --------------- ---------- ---------- Et0/0 Ethernet 192.168.1.102 100 UP
Verify with the show mpls l2transport vc detail command output (shown in Example 7-45) that the packets are being sent and received.
NewYork#show mpls l2transport vc detail
Local interface: Et0/0 up, line protocol up, Ethernet up
Destination address: 192.168.1.102, VC ID: 100, VC status: up
Preferred path: not configured
Default path: active
Tunnel label: 16, next hop point2point
Output interface: Se5/0, imposed label stack {16 16}
Create time: 00:18:10, last status change time: 00:03:51
Signaling protocol: LDP, peer 192.168.1.102:0 up
MPLS VC labels: local 16, remote 16
Group ID: local 0, remote 0
MTU: local 1500, remote 1500
Remote interface description:
Sequencing: receive disabled, send disabled
VC statistics:
packet totals: receive 56, send 56
byte totals: receive 6019, send 6014
packet drops: receive 0, send 78
You can use several useful debugging commands to verify and troubleshoot EoMPLS operation on PE routers.
For dot1Q operation, look for VC type 4 in output generated by the debug mpls l2transport signaling message command, as shown in Example 7-46.
NewYork#debug mpls l2transport signaling message NewYork(config)#interface ethernet 0/1.100 NewYork(config-subif)#shutdown 00:19:51: AToM LDP [192.168.1.102]: Sending label withdraw msg vc type 4, cbit 1, vc id 100, group id 0, vc label 20, status 0, mtu 1500 00:19:51: AToM LDP [192.168.1.102]: Received label release msg, id 78 vc type 4, cbit 1, vc id 100, group id 0, vc label 20, status 0, mtu 0 NewYork(config-subif)#no shutdown 00:21:56: AToM LDP [192.168.1.102]: Sending label mapping msg vc type 4, cbit 1, vc id 100, group id 0, vc label 20, status 0, mtu 1500
In troubleshooting the port-based EoMPLS operation, look for VC type 5 in the debug mpls l2transport signaling message command output, as shown in Example 7-47.
NewYork#debug mpls l2transport signaling message AToM LDP message debugging is on ! NewYork(config)#interface ethernet 0/0 NewYork(config-if)#shutdown 00:08:39: AToM LDP [192.168.1.102]: Sending label withdraw msg vc type 5, cbit 1, vc id 100, group id 0, vc label 16, status 0, mtu 1500 00:08:39: AToM LDP [192.168.1.102]: Received label release msg, id 34 vc type 5, cbit 1, vc id 100, group id 0, vc label 16, status 0, mtu 0 00:08:41: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to administratively down NewYork(config-if)#no shutdown 00:08:42: AToM LDP [192.168.1.102]: Sending label mapping msg vc type 5, cbit 1, vc id 100, group id 0, vc label 20, status 0, mtu 1500 00:08:44: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up 00:08:45: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to up
Another helpful command is debug acircuit event for information on all attachment circuits, as illustrated in Example 7-48.
NewYork#debug acircuit event NewYork(config)#int e 0/0 NewYork(config-if)#no shutdown 00:12:59: ACLIB [192.168.1.102, 100]: SW AC interface UP for Ethernet interface Et0/0 00:12:59: ACLIB [192.168.1.102, 100]: pthru_intf_handle_circuit_up() calling acmgr_circuit_up 00:12:59: ACLIB [192.168.1.102, 100]: Setting new AC state to Ac-Connecting 00:12:59: ACLIB: Update switching plane with circuit UP status 00:12:59: ACLIB [192.168.1.102, 100]: SW AC interface UP for Ethernet interface Et0/0 00:12:59: ACLIB [192.168.1.102, 100]: pthru_intf_handle_circuit_up() ignoring up event. Already connected or connecting. 00:12:59: Et0/0 ACMGR: Receive <Circuit Up> msg 00:12:59: Et0/0 ACMGR: circuit up event, SIP state chg fsp up to connected, action is p2p up forwarded 00:12:59: ACLIB: pthru_intf_response hdl is 8C000002, response is 2 00:12:59: ACLIB [192.168.1.102, 100]: Setting new AC state to Ac-Connected 00:12:59: AToM LDP [192.168.1.102]: Sending label mapping msg vc type 5, cbit 1, vc id 100, group id 0, vc label 16, status 0, mtu 1500 00:12:59: Et0/0 ACMGR: Rcv SIP msg: resp peer-to-peer msg, hdl 8C000002, sss_hdlB4000003 00:12:59: Et0/0 ACMGR: remote up event, SIP connected state no chg, action is ignore 00:13:01: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up 00:13:02: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to up
Use the debug mpls l2transport vc event command to see the AToM event messages about the VCs, as shown in Example 7-49. Watch how the messages reflect the shutdown of the interface and the healthy recovery.
NewYork#debug mpls l2transport vc event NewYork(config)#interface ethernet 0/0 NewYork(config-if)#shutdown 00:14:37: AToM MGR [192.168.1.102, 100]: Local end down, vc is down 00:14:37: AToM MGR [192.168.1.102, 100]: Unprovision SSM segment 00:14:37: AToM SMGR: Submit Imposition Update 00:14:37: AToM SMGR: Submit Disposition Update 00:14:37: AToM SMGR [192.168.1.102, 100]: Event Imposition Disable 00:14:37: AToM SMGR [192.168.1.102, 100]: State [Imposition/Disposition Rdy-> Disposition Rdy] 00:14:37: AToM SMGR [192.168.1.102, 100]: Event Disposition Disable 00:14:37: AToM SMGR [192.168.1.102, 100]: State [Disposition Rdy->Provisioned] 00:14:37: AToM SMGR: Submit SSM event 00:14:37: AToM SMGR: Event SSM event 00:14:37: AToM SMGR [192.168.1.102, 100]: sucessfully teardown sss switch for pwid 5A000000 00:14:37: AToM SMGR [192.168.1.102, 100]: sucessfully processed ssm unprovisioning for pwid 5A000000 00:14:39: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to administratively down 00:14:40: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to down NewYork(config-if)#no shutdown 00:15:16: AToM MGR [192.168.1.102, 100]: Local end up 00:15:16: AToM MGR [192.168.1.102, 100]: Validate vc, activating data plane 00:15:16: AToM SMGR: Submit Imposition Update 00:15:16: AToM SMGR: Submit Disposition Update 00:15:16: AToM SMGR [192.168.1.102, 100]: Event Imposition Enable, imp-ctrlflag 83, remote vc label 16 00:15:16: AToM SMGR [192.168.1.102, 100]: Imposition Programmed, Output Interface: Se5/0 00:15:16: AToM SMGR [192.168.1.102, 100]: State [Provisioned->Imposition Rdy] 00:15:16: AToM SMGR [192.168.1.102, 100]: Event Disposition Enable, disp-ctrlflag 3, local vc label 16 00:15:16: AToM SMGR [192.168.1.102, 100]: State [Imposition Rdy->Imposition/ Disposition Rdy] 00:15:16: AToM SMGR: Submit SSM event 00:15:16: AToM SMGR: Event SSM event 00:15:16: AToM SMGR [192.168.1.102, 100]: sucessfully processed ssm provision request pwid 5A000000 00:15:16: AToM SMGR [192.168.1.102, 100]: Send COMPLETE signal to SSM 00:15:16: AToM SMGR [192.168.1.102, 100]: sucessfully setup sss switch for pwid 5A000000 00:15:16: AToM SMGR: Submit SSM event 00:15:16: AToM SMGR: Event SSM event 00:15:16: AToM SMGR [192.168.1.102, 100]: sucessfully processed ssm bind for pw id 5A000000 00:15:16: AToM MGR [192.168.1.102, 100]: Receive SSM dataplane up notification 00:15:16: AToM MGR [192.168.1.102, 100]: Dataplane activated 00:15:18: %LINK-3-UPDOWN: Interface Ethernet0/0, changed state to up 00:15:19: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0, changed state to up
To verify the flow of packets across the Layer 2 tunnel, use the debug mpls l2transport packet data command, as shown in Example 7-50.
Caution
Make sure to use this testing technique in a lab environment only.
NewYork#debug mpls l2transport packet data 00:17:44: ATOM disposition: in Se5/0, size 60, seq 0, control word 0x0 00:17:44: 00 00 0C 00 6C 00 00 00 0C 00 6C 00 90 00 00 00 ^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^ ^^^^^ Dest. Address Source Address Etype = 0x9000 = LOOP 00:17:44: 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00:17:44: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00:17:44: 00 00 00 00 00 00 00 00 00 00 00 00 00:17:44: ATOM disposition: in Se5/0, size 114, seq 0, control word 0x0 00:17:44: 00 00 0C 00 6F 00 00 00 0C 00 6C 00 08 00 45 00 ^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^ ^^^^^ ^^^^^... Dest. Address Source Address | Begins IP packetEtype = 0x0800 = IP 00:17:44: 00 64 00 0A 00 00 FF 01 72 3A C0 A8 64 01 C0 A8 00:17:44: 64 02 08 00 28 0D 0E D9 13 FC 00 00 00 00 00 10 00:17:44: 33 58 AB CD AB CD AB CD AB CD AB CD AB CD AB CD 00:17:44: AB CD AB CD AB CD AB CD AB CD AB CD AB CD AB CD 00:17:44: AB CD AB CD AB CD AB CD AB CD AB CD AB CD AB CD 00:17:44: AB CD AB CD AB CD AB CD AB CD AB CD AB CD AB CD 00:17:44: AB CD 00:17:44: ATOM imposition: out Se5/0, size 130, EXP 0x0, seq 0, control word 0x0 00:17:44: 0F 00 88 47 00 01 00 FF 00 01 01 02 00 00 00 00 ^^^^^ ^^^^^ ^^^^^^^^^^^ ^^^^^^^^^^^ ^^^^^^^^^^^ HDLC
Tunn. Label VC Label Ctrl-word
Label=16 Label=16
S=0 S=1
TTL=255 TTL=2 etype = MPLS Unicast 00:17:44: 00 00 0C 00 6C 00 00 00 0C 00 6F 00 08 00 45 00 ^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^ ^^^^^ ^^^^^... Dest. Address Source Address
Begins IP packet Etype = 0x0800 = IP 00:17:44: 00 64 00 0A 00 00 FF 01 72 3A C0 A8 64 02 C0 A8 00:17:44: 64 01 00 00 30 0D 0E D9 13 FC 00 00 00 00 00 10 00:17:44: 33 58 AB CD AB CD AB CD AB CD AB CD AB CD AB CD 00:17:44: AB CD AB CD AB CD AB CD AB CD AB CD AB CD AB CD 00:17:44: AB CD AB CD AB CD AB CD AB CD AB CD AB CD AB CD 00:17:44: AB CD AB CD AB CD AB CD AB CD AB CD AB CD AB CD 00:17:44: AB CD
Note
Note that in Example 7-50, the offline hand decoding of the packets is shown in bold.
You can see in Example 7-50 that the disposition packets show only the EoMPLS Payload, whereas the imposition packets also include the EoMPLS header.
In the first packet, the Ethertype of 0x9000 indicates a loopback packet. For this reason, the source and destination MAC addresses are the same. The second packet shows an IP packet transported in EoMPLS. Finally, in the third packet with the imposition operation, you can also see the Tunnel, MPLS, and AToM headers. Specifically, the Layer 2 is Cisco HDLC with an HDLC type of 0x8847, indicating that MPLS follows. A two-level MPLS stack includes the Tunnel label of 16 and a VC label of 16. Note that these two values do not need to be the same. The label stack is followed by a 4-byte control word and finally an Ethernet frame with an Ethertype of 0x0800 transporting an IP datagram with an ICMP packet.
Troubleshooting commands on switches are, for the most part, the same as those on routers. However, the output might be different, as you learn in this section. For instance, use the output of the command show mpls l2transport vc from Example 7-51 to get information about the VCs just as you would use this command on routers.
Metro-switch#show mpls l2transport vc Transport Client VC Trans Local Remote Tunnel VC ID Intf State Type VC Label VC Label Label 200 Vl200 UP vlan 215 215 implc-null
Table 7-5 explains the most pertinent fields of the show mpls l2transport vc command.
VC ID
| The VC ID configured by the mpls l2 route or xconnect command; must match on both ends.
|
Client Intf
| Indicates which Layer 2 interface is being used.
|
VC State
| The UP state shows whether the VC ever saw traffic.
|
Trans Type
| The available results include vlan for VLAN based and Ether for port based.
|
Local VC label
| Indicates that the local VC label is being used. For Cisco 7600 series switches, the local label is derived from the VLAN (local label = VLAN + 15).
|
Remote VC label
| The label advertised by the remote PE and used by this PE to reach the CE on the other end of the LSP.
|
Tunnel Label
| Identifies the outer label used to switch the packets between the PEs. An implicit-null label is shown for the back-to-back connection.
|
For more in-depth information about the VCs, use the show mpls l2transport vc detail command, as shown in Example 7-52.
Metro-switch# show mpls l2transport vc detail vcid: 200, local groupid: 24, remote groupid: 102 (vc is up) client: Vl200 is up, destination: 1.2.2.1, Peer LDP Ident: 1.2.1.1:0 local label: 215, remote label: 215, tunnel label: implc-null outgoing interface: s0/0, next hop: point2point Local MTU: 1500, Remote MTU: 1500 Remote interface description: Vlan 200 imposition: LC Programmed current imposition/last disposition slot: 2/32 Packet totals(in/out): 6246/6159 byte totals(in/out): 536999/444722
Table 7-6 describes the various output fields.
groupid
| AToM group ID advertised in the VC FEC
|
destination
| IP address of the targeted LDP session
|
outgoing interface
| Indication of which interface is being used to transmit
|
next hop
| MAC address of the next hop or point2point
|
remote interface description
| Interface on remote PE that connects to the CE
|
imposition
| Indication of whether the line card has been programmed (that is, imposition rewrite resolved)
|
current imposition
| Indication of the slot performing imposition for this VLAN
|
last disposition
| Indication of the last slot performing disposition*
|
packet or bytes in
| Per VC counters for disposition
|
packet or bytes out
| Per VC counters for imposition
|
*7600 supports per-destination load sharing. If multiple connections to the MPLS cloud exist, the imposition traffic can be transmitted on one interface, and the disposition traffic for the same VLAN can be received in another interface.
|