Simulation Results for the Basic Scenario

We simulated a very basic scenario to verify the simulation, get some performance estimates and conclusions.

Conclusions

  1. These basic cases show expected results: minimum latency, expected throughput, no congestion and almost no queueing. We verify the behavior of all components is as expected (bonded links, routing, generation, etc).
  2. It is interesting (not inmediattely expected) to observe a small queuing effect at the ouput of the felix servers. This is because an exponential period distribution is used, so if a packet is generated before the previous one is sent, it suffers a minimum queuing delay (which can be observed also in the small dispertion in the latency. Generation busts are absorbed by the felix out buffers and flows don't share ports, so no buffering will ever happen at the switches in these scenario.
  3. Performance: In both scenarios it took ~0.035ms to simulate each packet. So a rough estimation to simulate 10 servers generating 40M packets/s (ej: @40Gbps with packets of 1Kb) each during 1 minute: 14000s = ~4 hours.

Future Steps

  1. Add a more realistic Felix generation distribution. As described in Characterization of Felix server traffic.
  2. Add monitoring traffic. As described here
Nevertheless, with the conclusions observed in this simple case, with these 2 modifications we expect to see similar results: no queueing in the switches as data is buffers in the servers

Simulated Topology

We simulate 2 Felix and 2 SWROD servers connected using 2 switches (via 2 bonded links each server). The communication pattern is 1:1. Buffer capacity is set to infinite (no discards). We simulate 1s.

In the different scenarios below we changed 1) the generation rate (to estimate performance) and 2) the capacity link of links (to observe queuing patterns).

We observe 1) the throughput at the SWRODs, 2) the link usage and 3) the queue size at the switches.

NOTE: ports 0 and 1 of the switches are connected to the core routers but they were removed for simplicity as they are not used in this scenario.

Low generation rate (1Gbps), 40Gbps links

Configuration

  • generation rate at each of the servers was set to 1Gbps (1M packets of 1Kb each):
  • link capacity was set to 40Gbps:
    • LINK_BW_BITS_S = 40 * G # 40 Gbps

Results

backup: /afs/cern.ch/work/m/mbonaven/public/SimuResults/PhaseI/LArSlice_1_to_1/LAr_slice_1_to_1_1Gbps
git commit: f79d19d9c41714ad6581dace3325b3ad5a030235

Throughput at the SWROD

totalPackets_swrod_0 = 1999408.
totalPackets_swrod_1 = 2000368.

Each SWROD receives ~125 MB/s (1Gbps). It can be seen that there is not much dispersion (123 - 128 MB/s) and it corresponds to the dispersion of the generated traffic (no dispersion due to queuing).

Latency

The time to transmit 1K packet through a 40Gbps link is 0.025us. The packet should traverse 2 links, so the expected latency is 0.05us.

The plot shows that the latency at both swrods is as expected (with mean 0.05us). It also shows a very little dispersion (0.00001us), which suggests there must be a minimum queuing somewhere in the path (switches are modeled as perfect cut-through so they add no delay if there is no queuing).

Link Usage

Felix servers

Each felix server generates traffic in both it outports (bonded links equally distributed), ~62.5 MB/s in each port (1Gbps = 125MBps = 2 * 62.5MBps).
Generation rate follows an exponential distribution.

switches

Traffic goes through both switches. Each switch received traffic from both felix servers (felix0 -> port2 ; felix1 -> port3). Ports 0 and 1 are used to connect with the core routers and no traffic goes through the cores.

Queue sizes

Note on the queue plots: the figures plot the MAXIMUM queue size in a given sampling period (in this case samplingPeriod=0.01s). This is because we are interested in the queue required to achieve no discards. In the legends of the figures, the TIME _ AVERAGE is shown: this is the queue size average taking time into consideration => queueSize{i} / totalTimeWithSize{i}. See SamplerLogger and TimeAvg definition for more details.

Felix servers

Felix servers do show a minimum queueing (maximum of 125B = 1Kb = 1 packet). This is because an exponential period distribution is used, so if a packet is generated before the previous one is sent, it suffers a minimum queuing delay. This explains the small dispertion in the latency.

Switches
Switches show no queueing. This is because switches are cut-through (immediately forwards packets without applying any delay), and because the flows don't share ports (incoming traffic from a 40Gbps link going to a 40Gbps link). Generation busts are absorbed by the felix out buffers.

Performance

Number of generated packets: 2M ==> 2 servers generating 1MHz packets during 1 second
Total metric points (not including T) logged to Scilab: 2023766

Simulation execution:

  • Initialization TOTAL time: 24384 (ms)
  • Simulation time (not including init): 73334 (ms)
  • TOTAL execution time: 98540 (ms)

=> ~0.036ms of execution per simulated packet (not including init time)

Medium generation rate (40Gbps), 40Gbps links

40Gbps is medium generation rate as the network can absorb upto 80Gbps (2*40Gbps bonded links).

Configuration

  • generation rate at each of the servers was set to 40Gbps (40M packets of 1Kb each):
  • link capacity was set to 40Gbps:
    • LINK_BW_BITS_S = 40 * G # 40 Gbps

Results

backup: /afs/cern.ch/work/m/mbonaven/public/SimuResults/PhaseI/LArSlice_1_to_1/LAr_slice_1_to_1_40Gbps
git commit: e61f36d7

Throughput at the SWROD

totRcvPackets_swrod_0 = 39998741.
totRcvPackets_swrod_1 = 39997433.

Each SWROD receives ~5000 MB/s (40Gbps). It can be seen that there is some dispersion (49K - 51k MB/s) and it corresponds to the dispersion of the generated traffic (no dispersion due to queuing).

Latency

the expected latency is 0.05us.

The plot shows that the latency at both swrods is as expected (with mean 0.054us). It also shows a very little dispersion (0.001us) but more than in the low BW case (0.00001us).

Link Usage

Felix servers

Each felix server generates traffic in both it outports (bonded links equally distributed), ~2500 MB/s in each port (40Gbps = 5000MBps = 2 * 2500MBps).
Generation rate follows an exponential distribution.

switches

Traffic goes through both switches. Each switch received traffic from both felix servers (felix0 -> port2 ; felix1 -> port3). Ports 0 and 1 are used to connect with the core routers and no traffic goes through the cores.

Queue sizes

Note on the queue plots: the figures plot the MAXIMUM queue size in a given sampling period (in this case samplingPeriod=0.01s). This is because we are interested in the queue required to achieve no discards. In the legends of the figures, the TIME _ AVERAGE is shown: this is the queue size average taking time into consideration => queueSize{i} / totalTimeWithSize{i}. See SamplerLogger and TimeAvg definition for more details.

Felix servers

Felix servers show some queueing (maximum of ~625B = 5Kb = 5 packets). This is because an exponential period distribution is used, so if a packet is generated before the previous one is sent, it suffers a minimum queuing delay. This explains the small dispertion in the latency.

Switches

Switches show no queueing. This is because switches are cut-through (immediately forwards packets without applying any delay), and because the flows don't share ports (incoming traffic from a 40Gbps link going to a 40Gbps link). Generation busts are absorbed by the felix out buffers.

Performance

Number of generated packets: 80M ==> 2 servers generating 40MHz packets during 1 second

Total metric points (not including T) logged to Scilab:: 80020056

Simulation execution:

  • Initialization TOTAL time: 24797 (ms)
  • Simulation time (not including init): 2824890 (ms)
  • TOTAL execution time: 2869745 (ms)
=> ~0.035ms of execution per simulated packet (not including the init time)

High generation rate (70Gbps), 40Gbps links

Configuration

  • generation rate at each of the servers was set to 1Gbps (1M packets of 1Kb each):
  • link capacity was set to 40Gbps:
    • LINK_BW_BITS_S = 40 * G # 40 Gbps

Results

backup: /afs/cern.ch/work/m/mbonaven/public/SimuResults/PhaseI/LArSlice_1_to_1/LAr_slice_1_to_1_40Gbps
git commit: f79d19d9c41714ad6581dace3325b3ad5a030235

Throughput at the SWROD

totRcvPackets_swrod_0 = 69992603.
totRcvPackets_swrod_1 =69996343.

Each SWROD receives ~8750MB/s (70Gbps). It can be seen that there is some dispersion (87.3 - 87.9 MB/s) and it corresponds to the dispersion of the generated traffic (no dispersion due to queuing).

Latency

The time to transmit 1K packet through a 40Gbps link is 0.025us. The packet should traverse 2 links, so the expected latency is 0.05us.

The plot shows that the latency at both swrods is higher than expected (with mean 0.09us). It also shows a dispersion of 0.02us, which suggests queuing somewhere in the path (switches are modeled as perfect cut-through so they add no delay if there is no queuing).

Link Usage

Felix servers

Each felix server generates traffic in both it outports (bonded links equally distributed), ~4375.5 MB/s in each port (70Gbps = 8750MBps = 2 * 4375MBps).
Generation rate follows an exponential distribution.

switches

Traffic goes through both switches. Each switch received traffic from both felix servers (felix0 -> port2 ; felix1 -> port3). Ports 0 and 1 are used to connect with the core routers and no traffic goes through the cores.

Queue sizes

Note on the queue plots: the figures plot the MAXIMUM queue size in a given sampling period (in this case samplingPeriod=0.01s). This is because we are interested in the queue required to achieve no discards. In the legends of the figures, the TIME _ AVERAGE is shown: this is the queue size average taking time into consideration => queueSize{i} / totalTimeWithSize{i}. See SamplerLogger and TimeAvg definition for more details.

Felix servers

Felix servers do show queueing (maximum of 3800B = 30Kb = 30 packets). This is because an exponential period distribution is used, so if a packet is generated before the previous one is sent, it suffers queuing delay. This explains the dispertion in the latency.

Switches
Switches show no queueing. This is because switches are cut-through (immediately forwards packets without applying any delay), and because the flows don't share ports (incoming traffic from a 40Gbps link going to a 40Gbps link). Generation busts are absorbed by the felix out buffers.

Performance

Number of generated packets: 140M ==> 2 servers generating 70MHz packets during 1 second
Total metric points (not including T) logged to Scilab: 2023766

Simulation execution:

  • Initialization TOTAL time: 25994 (ms)
  • Simulation time (not including init): 4925393 (ms)
  • TOTAL execution time: 4970050 (ms) = 1h 22min
=> ~0.035ms of execution per simulated packet (not including init time)

-- MatiasAlejandroBonaventura - 2016-11-21

Edit | Attach | Watch | Print version | History: r9 < r8 < r7 < r6 < r5 | Backlinks | Raw View | WYSIWYG | More topic actions
Topic revision: r9 - 2016-11-29 - MatiasAlejandroBonaventura
 
    • Cern Search Icon Cern Search
    • TWiki Search Icon TWiki Search
    • Google Search Icon Google Search

    Main All webs login

This site is powered by the TWiki collaboration platform Powered by PerlCopyright & 2008-2019 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback