This case mainly talks about the ICMP delay caused by the packet processing mechanism of the CE switch.
Problem Description
The CE12808 switch is connected to the CRS_Router Cisco switch through four Layer 3 interfaces. After the interconnection, it's found that the ICMP delay between the two devices is long.
To check whether the link is normal, we make an ICMP test with large Packets and see the following on Huawei & Cisco.
1. From Huawei to Cisco :
~Huawei]ping -m 30 -t 8000 -c 100 -s 9152 -a 192.168.23.2 -f 192.168.23.1
PING 192.168.23.1: 9152 data bytes, press CTRL_C to break
Reply from 192.168.23.1: bytes=9152 Sequence=1 ttl=255 time=5 ms
Reply from 192.168.23.1: bytes=9152 Sequence=2 ttl=255 time=6 ms
Reply from 192.168.23.1: bytes=9152 Sequence=3 ttl=255 time=3 ms
Reply from 192.168.23.1: bytes=9152 Sequence=4 ttl=255 time=3 ms
Reply from 192.168.23.1: bytes=9152 Sequence=9 ttl=255 time=17 ms
Reply from 192.168.23.1: bytes=9152 Sequence=10 ttl=255 time=59 ms
Reply from 192.168.23.1: bytes=9152 Sequence=11 ttl=255 time=93 ms
Reply from 192.168.23.1: bytes=9152 Sequence=12 ttl=255 time=93 ms
Reply from 192.168.23.1: bytes=9152 Sequence=13 ttl=255 time=93 ms
Reply from 192.168.23.1: bytes=9152 Sequence=14 ttl=255 time=92 ms
Reply from 192.168.23.1: bytes=9152 Sequence=15 ttl=255 time=93 ms
...
Reply from 192.168.23.1: bytes=9152 Sequence=24 ttl=255 time=92 ms
Reply from 192.168.23.1: bytes=9152 Sequence=25 ttl=255 time=93 ms
--- 192.168.23.1 ping statistics ---
100 packet(s) transmitted
100 packet(s) received
0.00% packet loss
round-trip min/avg/max = 3/84/94 ms
2. From Cisco to Huawei :
Cisco# ping 192.168.24.2 source 192.168.24.1 size 9180 donotfrag count 30000 interval 5
Fri Jul 1 03:50:57.844 GMT
Type escape sequence to abort.
Sending 30000, 9180-byte ICMP Echos to 192.168.24.2, timeout is 2 seconds:
Success rate is 100 percent (153/153), round-trip min/avg/max = 1/90/94 ms
There seems to be some ISSUE on the link interconnect because the delay should not exceed 15 or 20 ms.
Handling Process
Before checking the issue we need to be clear about MTU for Cisco & Huawei:
A. Cisco Interface MTU is 9194: The ICMP test packet length is 9180 bytes through Cisco CRS Ios XR not consider 14 bytes for Ethernet on ICMP Test.
B. Huawei Interface MTU is 9180: The ICMP test packet length is 9152 bytes through Huawei does not consider 14 bytes for Ethernet, 8 bytes for ICMP, and 6 bytes for VLAN.
1. At the beginning of the issue, we thought the issue was on Optical Fibers and GE 10G Transceivers, so our plan was to verify physical links and made physical loops for verifying the Transceiver. After this, the issue continues.
[~Huawei]display interface 10GE8/0/46 transceiver verbose | in DBm
Current RX Power (dBm) :-2.79
Default RX Power High Threshold (dBm) :2.50
Default RX Power Low Threshold (dBm) :-16.40
Current TX Power (dBm) :-3.75
Default TX Power High Threshold (dBm) :2.50
Default TX Power Low Threshold (dBm) :-10.20
Cisco Ten 0/4/0/0 <<>> Huawei 10GE5/0/46
Cisco#sh controllers tenGigE 0/4/0/0 phy | in dBm
Fri Jul 1 03:18:00.095 GMT
Tx power: -3 dBm (4307 in units of 0.1 uW)
Rx power: -5 dBm (2597 in units of 0.1 uW)
<Huawei>display interface 10GE5/0/46 transceiver verbose | in dBm
Current RX Power (dBm) :-3.90
Default RX Power High Threshold (dBm) :2.50
Default RX Power Low Threshold (dBm) :-16.40
Current TX Power (dBm) :-3.26
Default TX Power High Threshold (dBm) :2.50
Default TX Power Low Threshold (dBm) :-10.20
2. Verify the packets for ICMP statistics and check if there is some issue with processing :
<Huawei>disp current-configuration configuration include-default | i fast
undo arp fast-reply disable
undo icmp echo-reply fast disable
[~Huawei]display icmp fast-reply statistics slot 5
------------------------ Display ICMP Statistics -------------------------------
Received packets:
request packets: 0
invalid request packets: 0
failed to get vrf: 0
destination is not host ip: 0
failed to get ctrl word: 0
Send packets:
successful reply packets: 0
failed reply packets: 0
[~Huawei]display icmp fast-reply statistics slot 8
------------------------ Display ICMP Statistics -------------------------------
Received packets:
request packets: 0
invalid request packets: 0
failed to get vrf: 0
destination is not host ip: 0
failed to get ctrl word: 0
Send packets:
successful reply packets: 0
failed reply packets: 0
[~Huawei]display icmp fast-reply statistics slot 1
------------------------ Display ICMP Statistics -------------------------------
Received packets:
request packets: 0
invalid request packets: 0
failed to get vrf: 0
destination is not host ip: 0
failed to get ctrl word: 0
Send packets:
successful reply packets: 0
failed reply packets: 0
--------------------------------------------------------------------------------
[~Huawei]display icmp fast-reply statistics slot 4
------------------------ Display ICMP Statistics -------------------------------
Received packets:
request packets: 0
invalid request packets: 0
failed to get vrf: 0
destination is not host ip: 0
failed to get ctrl word: 0
Send packets:
successful reply packets: 0
failed reply packets: 0
--------------------------------------------------------------------------------
Root Cause
The Ping Packet Processing on a CE Switch is like follows :
1. The green line in the preceding figure shows ping packet processing. The forwarding plane sends ping packets to the control plane through the CPU. The processing latency mainly includes the latency on the forwarding plane and the latency on the control plane. The latency on the forwarding plane is at the microsecond level and can be ignored. The latency on the control plane is determined by the CPU usage. If the CPU usage is high, the latency increases.
2. The red lines show service data flow processing. Service data packets are directly forwarded by the forwarding plane and are not processed by the control plane. The latency is at the microsecond level.
Analysis of Latency and Jitter During Ping Packet Transmission Between CE and Cisco Switches :
Ping packet processing is as follows:
1. The packet receiving process on the CE switch is as follows:
Control plane of the Cisco switch -> packet sending by the CPU of the Cisco switch -> forwarding plane of the Cisco switch -> forwarding plane of the CE switch -> packet receiving by the CPU of the CE switch -> control plane of the CE switch
2. The packet forwarding process on the CE switch is as follows:
Control plane of the CE switch -> packet sending by the CPU of the CE switch -> forwarding plane of the CE switch -> forwarding plane of the Cisco switch -> packet receiving by the CPU of the Cisco switch -> control plane of the Cisco switch.
Solution
The latency of ping packets mainly includes the processing latency on the Cisco switch and the processing latency on the CE switch. The processing latency on the CE switch mainly includes the latency on the forwarding plane and the latency on the control plane.
The latency on the forwarding plane is at the microsecond level and can be ignored. The latency on the control plane is determined by the CPU usage. If the CPU usage is high, the latency increases.
Because of the CPU performance and software calculation priority of the CE switch, a little latency and jitter may occur for ping packets that need to be processed by the CPU. However, data packet forwarding is not affected on the CE switch because data packets are forwarded by the hardware and software processing is not involved.
Suggestions and Summary
As we mentioned this issue is just related to IMCP Test, not with Traffic of Services.
That's true because when we put traffic on the Internet there is no issue with users.
Before the ICMP test, ensure that the test does not affect service traffic processing.
No comments:
Post a Comment