Skip to content

industry-standards.md

Latest commit

 

History

History
105 lines (79 loc) · 4.51 KB

industry-standards.md

File metadata and controls

105 lines (79 loc) · 4.51 KB
Raw

Industry Standard Benchmarking Methodologies

The iperf3 Soucy methodology in this repo is practical and reproducible on commodity hardware. Formal testing labs and vendor certifications use the IETF standards below. Understanding these helps interpret published results and design more rigorous tests.

RFC 2544 — Benchmarking Methodology for Network Interconnect Devices

The foundational standard. Defines four mandatory test types for any network device benchmark. Tools like TRex and Ostinato have built-in RFC 2544 test suites.

flowchart TD
    rfc["RFC 2544 Test Suite"]
    rfc --> t["Throughput\nMax forwarding rate\nwith 0% frame loss"]
    rfc --> l["Latency\nRound-trip delay at\nmax throughput rate"]
    rfc --> fl["Frame Loss Rate\nPackets dropped vs\npackets sent across\nload range 0–100%"]
    rfc --> bb["Back-to-Back Frames\nMax burst size before\nframe loss occurs\n(buffer depth test)"]
Loading
Test What it measures Why it matters
Throughput Maximum forwarding rate with zero frame loss True line-rate capacity
Latency Round-trip delay at throughput rate Real-time and voice traffic sensitivity
Frame Loss Rate % frames dropped across 0–100% load range Behaviour under overload
Back-to-Back Frames Maximum burst size before loss Buffer depth / burst tolerance

Key difference from iperf3: RFC 2544 uses UDP with exact frame counts to measure loss precisely. TCP (used by iperf3) retransmits lost frames, which hides loss but measures effective transfer rate — the right choice for understanding user experience, but not for hardware characterisation.

Required packet sizes per RFC 2544: 64, 128, 256, 512, 1024, 1280, 1518 bytes (Ethernet payload sizes, not IP sizes).

Running RFC 2544 with TRex

# TRex has a built-in RFC 2544 automation script
cd /opt/trex/v3.x/
./t-rex-64 -f cap2/imix_64.yaml --cfg /etc/trex_cfg.yaml -d 60

ITU-T Y.1564 — Ethernet Service Activation Test Methodology

Designed for modern Ethernet services. An improvement over RFC 2544 for SLA verification — tests multiple traffic flows simultaneously rather than sequentially, which better represents production conditions.

Feature RFC 2544 ITU-T Y.1564
Traffic flows Single flow Multiple simultaneous flows
Test approach Sequential per metric Service configuration + performance
Primary use Device characterisation SLA validation
Duration Variable Two-phase (config + 15-min performance)

Two phases:

  1. Service configuration test — ramps traffic to CIR (Committed Information Rate) to confirm no frame loss at the committed rate
  2. Service performance test — 15-minute sustained test measuring throughput, frame loss, latency, and jitter simultaneously

RFC 3511 — Benchmarking Methodology for Firewall Performance

Firewall-specific. Extends RFC 2544 with additional metrics that matter for stateful devices. Throughput alone is insufficient for firewalls because connection state tracking is the dominant bottleneck.

Metric Description
Throughput Standard forwarding rate (as RFC 2544)
Concurrent connections Maximum simultaneous tracked sessions
Connection setup rate New connections per second (TCP SYN/ACK/data)
Connection teardown rate Connections closed per second
HTTP transaction rate Complete HTTP request/response cycles per second

Why concurrent connections matter for VyOS: A stateful firewall tracking 100,000 concurrent connections has a very different memory and CPU profile than one tracking 1,000,000. Marketing specs often state maximum throughput at a very low concurrent connection count.

Summary: which standard for which purpose

Purpose Standard Tooling
Hardware characterisation / vendor comparison RFC 2544 TRex, Ostinato
Ethernet service SLA validation ITU-T Y.1564 TRex, Ixia
Firewall capacity planning RFC 3511 TRex ASTF mode
Practical throughput for Linux routers Soucy / iperf3 iperf3 (this repo)

Further reading