CSR1000v Troubleshooting

References:
BRKSPG-2519 - SP Use Cases for NFV and vCPE - Enabling Service Agility via CSR1000V
http://www.cisco.com/c/dam/en/us/td/docs/routers/csr1000/technical_references/CSR-Packet-Flow-Troubleshooting-Guide.pdf

No dedicated crypto engine, leverages the Intel AES-NI instruction set to provide hardware crypto assist. FFP (Fast Forwarding Processor) is the software implementation of the QFP from ASR1000s, which uses DPDK.

Packet path within CSR 1000v:
1. Ethernet driver (ingress)
2. Rx thread
3. PPE Thread (packet processing)
4. HQF Thread (egress queuing)
5. Ethernet driver (egress)

Control Plane and Data Plane are mapped to vCPUs. There are four main threads in the CSR1000v: Control Plane, Data Plane PPE, Data Plane HQF, Data Plane Rx processing.

1 vCPUs: vCPE0 =  Control Plane, Data Plane PPE, Data Plane HQF, Data Plane Rx processing
2 vCPUs: vCPU0 = Control Plane, vCPU1 = Data Plane PPE, Data Plane HQF, Data Plane Rx processing
4 vCPUs: vCPU0 = Control Plane, vCPU1&2 = Data Plane PPE, vCPU3 = Data Plane HQF, Data Plane Rx processing
8 vCPUs: vCPU0 = Control Plane, vCPU1-5 = Data Plane PPE, vCPU6 = Data Plane HQF, vCPU7 = Data Plane Rx processing

Official throughput stats with 1500 byte packets:

License:     IP Base   SEC       AppX      AX

Throughput | vCPUs/RAM
1   Gbps     1vCPU/4GB 1vCPU/4GB 1vCPU/4GB 2vCPU/4GB
2.5 Gbps     1vCPU/4GB 1vCPU/4GB 4vCPU/4GB 4vCPU/4GB
5   Gbps     1vCPU/4GB 2vCPU/4GB 8vCPU/4GB NA
10  Gbps     2vCPU/4GB NA        NA        NA

 

System Overview

# Is this definitely a virtual router?
show platform hardware qfp active infrastructure chipset 0 version

# Memory allocated to the VM
show platform software system memory
# Show vCPUs allocated to VM and their usage
show platform software system all
show platform hardware throughput level
show platform hardware qfp active infrastructure chipset 0 capabilities

CPU/Memory/Processes

# Show process, CPU and memory stats inside IOS-XE
show processes cpu sorted
show processes memory sorted

# Show process, CPU and memory stats on the device
show platform software status control-processor

# Run Linux "top" for IOS-XE
monitor platform software process r0
# Run Linux "top" for IOSd
monitor platform software process RP active
# Might need this first: terminal terminal-type vt100

# Show DRAM and IRAM usage; together they add up to the total ammount allocated to the virtual QFP in the CSR1000v
show platform hardware qfp active infrastructure exmem statistics
# Check for specific process that is hogging DRAM/IRAM memory
show platform hardware qfp active infrastructure exmem statistics user

# Punt stats
show platform hardware qfp active infrastructure punt statistics qfp 0 type queue-stats
show platform hardware qfp active infrastructure punt statistics qfp 0 type per-cause
show platform hardware qfp active datapath infrastructure sw-ipc

# Punt issues
show platform software infrastructure lsmpi driver
show platform hardware qfp active datapath infrastructure sw-lsmpi

 

Interfaces, Data Plane and Throughput Stats

# Show interface to vNIC mapping
show platform software vnic-if interface-mapping
# Show interace rx-uidb index
show platform hardware qfp active datapath infrastructure rx-uidb summary
# Show NIC types in CSR1000v (PMD/VF/etc)
show platform hardware qfp active datapath infrastructure sw-nic

# When using SR-IOV we can see VF stats with
show interface gi1 controller

# Show interface packet stats
show interface gi1
show interfaces gigabitEthernet 3 stats
show interfaces gigabitEthernet 3 accounting
show interfaces gigabitEthernet 3 summary
show platform hardware qfp active interface if-name gi3
show platform hardware qfp active interface if-name gi3 statistics

# Show data plane stats
show platform hardware qfp active datapath utilization
show platform hardware qfp active statistics drop
# Show data plane stats per interface
show platform hardware qfp active interface if-name gi3 statistics
show platform hardware qfp active datapath infrastructure rx-uidb detail <rx-uidb number>
# Check the data path Tx/Rx process per interface
show platform hardware qfp active datapath infrastructure sw-nic

 

KVM Host:

# Show interfaces bound to a VM
virsh --connect qemu:///system domiflist <vm>

# Check interface MTU with
ip l show dev <if_name>
# Check VFs with
ip l show dev <pf_name>
# Disable spoofing check with
sudo ip link set dev <pf_name> vf <vf_number> spoofchk off
# Packet statistics for physical NIC packets/bytes/drops etc
sudo ethtool -S <ethN>
# Check current Rx/Tx queue size on physical NIC
sudo ethtool -g <ethN>

# Check interface queue count
cat /proc/interrupts
# Or...
ls -l /sys/class/net/<dev>queues/
# Show live interface stats
sar -n EDEV 1
# Show live CPU core usage stats (such as interrupts)
mpstat -P ALL 1

---