Skip to content

Commit f5d8853

Browse files
DeagleGrossDeagleGross
authored
Configure openssl.cnf EC, Ciphers for TLS benchmarks (#2122)
* configs / instructions * correct setup for EC * more data * setup * docs on crontab usage * update for azurelinux3
1 parent 40cf535 commit f5d8853

9 files changed

Lines changed: 487 additions & 3 deletions

File tree

‎src/BenchmarksApps/TLS/Kestrel/Program.cs‎

Lines changed: 17 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -173,11 +173,15 @@ bool AllowAnyCertificateValidationWithLogging(X509Certificate2 certificate, X509
173173
{
174174
logged = true;
175175

176-
var tlsHandshakeFeature = context.Features.GetRequiredFeature<ITlsHandshakeFeature>();
176+
var tlsFeature = context.Features.GetRequiredFeature<ITlsHandshakeFeature>();
177177

178178
Console.WriteLine("Request details:");
179179
Console.WriteLine("-----");
180-
Console.WriteLine("TLS: " + tlsHandshakeFeature.Protocol);
180+
Console.WriteLine($"Protocol: {tlsFeature.Protocol}");
181+
Console.WriteLine($"CipherSuite: {tlsFeature.NegotiatedCipherSuite}");
182+
Console.WriteLine($"CipherAlgorithm: {tlsFeature.CipherAlgorithm}");
183+
Console.WriteLine($"KeyExchangeAlgorithm: {tlsFeature.KeyExchangeAlgorithm}");
184+
Console.WriteLine("TLS: " + tlsFeature.Protocol);
181185
Console.WriteLine("-----");
182186
}
183187

@@ -218,7 +222,7 @@ bool AllowAnyCertificateValidationWithLogging(X509Certificate2 certificate, X509
218222
}
219223

220224
app.MapGet("/hello-world", () =>
221-
{
225+
{
222226
return Results.Ok("Hello World!");
223227
});
224228

@@ -246,6 +250,16 @@ bool AllowAnyCertificateValidationWithLogging(X509Certificate2 certificate, X509
246250
{
247251
Console.WriteLine($"\tenabled logging stats to console");
248252
}
253+
254+
if (!(OperatingSystem.IsLinux() || OperatingSystem.IsMacOS()))
255+
{
256+
#pragma warning disable CA1416 // Validate platform compatibility
257+
Console.WriteLine($"OpenSSL: {System.Security.Cryptography.SafeEvpPKeyHandle.OpenSslVersion}");
258+
#pragma warning restore CA1416 // Validate platform compatibility
259+
}
260+
261+
Console.WriteLine($"OPENSSL_CONF: {Environment.GetEnvironmentVariable("OPENSSL_CONF")}");
262+
Console.WriteLine($"LD_LIBRARY_PATH: {Environment.GetEnvironmentVariable("LD_LIBRARY_PATH")}");
249263
Console.WriteLine($"\tlistening endpoints: {listeningEndpoints}");
250264
Console.WriteLine("--------------------------------");
251265

‎src/BenchmarksApps/TLS/README.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
# Useful stuff to test TLS behavior
2+
3+
### Docker images
4+
Crank agent comes with its own dockerfile, and its own dependencies. Here we are interested in some low-level setups of TLS parameters on OS level as well.
5+
For that reason in [crank/agent](./crank/agent/) you can find a replica of dockerfiles from [crank](https://github.com/dotnet/crank/tree/main/docker/agent).
6+
7+
### Analysis of TLS parameters on request
8+
To lookup TLS behavior you can install npcap/wireshark on win machine,
9+
and collect a network dump (note: using a custom port requires to use `Analyze->DecodeAs` and set TCP / TLS port on dump data). There in `Client Hello` or `Server Hello` TLS parameters can be found.
10+
11+
However, easier way is to simply perform a curl request
12+
```bash
13+
curl -v https://<ip>:<port>/<endpoint> --tlsv1.3 --tls-max 1.2 --insecure --curves [P-256/P-384/P-521/X25519]
14+
```
15+
where:
16+
- `--insecure` skips certificate check (but still runs with TLS)
17+
- `--tlsv1.3` or `--tlsv1.2` sets a minimum tls version
18+
- `--tls-max 1.3` or `--tls-max 1.2` sets a maximum tls version (does not allow client-server to lift up a version)
19+
- `--curves ...` forces a specific curve.
20+
21+
In output you nee to find SSL connection:
22+
```
23+
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / secp521r1 / RSASSA-PSS
24+
```
25+
26+
### Network dump collection and analysis
27+
Most probably benchmarks are run via CI setup, and client will send it's own request (as set). So you want to collect the network dump on the server to ensure request/response has correct TLS parameters.
28+
29+
In order to collect network dump (via `tcpdump`) use this command. Change the port accordingly.
30+
```bash
31+
sudo tcpdump -i any -w capture.pcap port 5000
32+
```
33+
34+
then you can analyze it via `tshark`
35+
```bash
36+
tshark -r capture.pcap -Y "tls.handshake.type == 2" -d tcp.port==5000,tls -c 300 -V
37+
```
38+
Arguments:
39+
- `-Y "tls.handshake.type == 2"` filters only `Server Hello` packets.
40+
- `-d tcp.port==5000,tls` changes the port for tcp/tls if client/server does not communicate via standard ports.
41+
- `-c 300` looks into only first 300 packets. Otherwise too hard to see in a single cmd window
42+
- `-V` gives verbose infomation about packet (you can see EC, CipherSuite used etc)
43+
44+
### Machine setup
45+
You could use [set-fips-compliant-tls-config](./set-fips-compliant-tls-config.ps1) to configure machine. It may not work (registry on windows does not apply always).
46+
47+
You can set TLS CipherSuite and ECC Curve order in Windows UI:
48+
- Local Group Policy Editor -> Computer Configuration > Administrative Templates > Network > SSL Configuration
49+
- Values can be taken from https://learn.microsoft.com/en-us/windows/win32/secauthn/tls-elliptic-curves-in-windows-10-1607-and-later
50+
51+
### Verify machine setup
52+
53+
#### Windows
54+
- Look cipher suite priority list in registry:
55+
```powershell
56+
(Get-ItemProperty 'HKLM:\SOFTWARE\Policies\Microsoft\Cryptography\Configuration\SSL\00010002' -Name 'Functions').Functions -split ',' | ForEach-Object { "{0,3}. {1}" -f ($_.ReadCount), $_ }
57+
```
58+
59+
- Look eliptic curves priority list in registry:
60+
```powershell
61+
Get-ItemProperty -Path 'HKLM:\SYSTEM\CurrentControlSet\Control\Cryptography\Configuration\Local\SSL\00010002' -Name 'EccCurves' -ErrorAction SilentlyContinue
62+
```

‎src/BenchmarksApps/TLS/crank/agent/Dockerfile‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build-env
2+
3+
COPY . .
4+
5+
ENV DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1
6+
ENV DOTNET_CLI_TELEMETRY_OPTOUT=1
7+
8+
# Build self contained
9+
RUN dotnet publish -c Release src/Microsoft.Crank.Agent --output /app --framework net8.0
10+
11+
# Build runtime image
12+
# FROM mcr.microsoft.com/dotnet/aspnet:8.0
13+
# Use SDK image as it is required for the dotnet tools
14+
FROM mcr.microsoft.com/dotnet/sdk:8.0
15+
16+
ARG CPUNAME=x86_64
17+
ARG ENABLE_FIPS_MODE=false
18+
ARG OPENSSL_CIPHER_STRING=TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256
19+
ARG OPENSSL_GROUPS=P-384:P-256:P-521
20+
21+
# Install dotnet-symbols
22+
RUN dotnet tool install -g dotnet-symbol
23+
ENV PATH="${PATH}:/root/.dotnet/tools"
24+
25+
# Install dependencies
26+
RUN apt-get update \
27+
&& apt-get install -y --no-install-recommends \
28+
git \
29+
procps \
30+
cgroup-tools \
31+
curl \
32+
wget \
33+
nano \
34+
# dotnet performance repo microbenchmark dependencies
35+
libgdiplus \
36+
# libmsquic requirements
37+
gnupg2 \
38+
software-properties-common \
39+
# NativeAOT requirements
40+
clang \
41+
zlib1g-dev \
42+
libkrb5-dev \
43+
# .NET 9.0 requirement
44+
libc6
45+
46+
# Install HTTP/3 support
47+
RUN curl -LO https://packages.microsoft.com/keys/microsoft.asc && \
48+
echo 2fa9c05d591a1582a9aba276272478c262e95ad00acf60eaee1644d93941e3c6 microsoft.asc| sha256sum --check - && \
49+
apt-key add microsoft.asc && \
50+
rm microsoft.asc && \
51+
echo deb https://packages.microsoft.com/debian/12/prod bookworm main >> /etc/apt/sources.list.d/microsoft.list && \
52+
apt-get update && \
53+
apt-get install -y libmsquic && \
54+
rm -rf /var/lib/apt/lists/*
55+
56+
# Build and install h2load. Required as there isn't a way to distribute h2load as a single file to download
57+
RUN apt-get update \
58+
&& apt-get install -y --no-install-recommends \
59+
g++ make binutils autoconf automake autotools-dev libtool pkg-config \
60+
zlib1g-dev libcunit1-dev libxml2-dev libev-dev libevent-dev libjansson-dev \
61+
libc-ares-dev libjemalloc-dev libsystemd-dev \
62+
python-is-python3 python3-dev python3-setuptools
63+
64+
ENV DEBIAN_FRONTEND=noninteractive
65+
# Add the Debian sid repository
66+
RUN echo 'deb http://deb.debian.org/debian sid main' >> /etc/apt/sources.list \
67+
&& echo 'deb http://deb.debian.org/debian-debug sid-debug main' >> /etc/apt/sources.list
68+
69+
RUN apt-get update \
70+
&& apt-get install -y --no-install-recommends \
71+
openssl libssl-dev openssl-dbgsym libssl3t64-dbgsym \
72+
&& openssl version
73+
74+
# Configure OpenSSL for FIPS-compliant cipher suites if $ENABLE_FIPS_MODE
75+
RUN if [ "$ENABLE_FIPS_MODE" = "true" ]; then \
76+
echo "=== FIPS MODE ENABLED - Configuring OpenSSL ===" && \
77+
cat /etc/ssl/openssl.cnf && \
78+
echo "" >> /etc/ssl/openssl.cnf && \
79+
echo "openssl_conf = openssl_init" >> /etc/ssl/openssl.cnf && \
80+
echo "[openssl_init]" >> /etc/ssl/openssl.cnf && \
81+
echo "ssl_conf = ssl_sect" >> /etc/ssl/openssl.cnf && \
82+
echo "[ssl_sect]" >> /etc/ssl/openssl.cnf && \
83+
echo "system_default = system_default_sect" >> /etc/ssl/openssl.cnf && \
84+
echo "[system_default_sect]" >> /etc/ssl/openssl.cnf && \
85+
echo "CipherString = $OPENSSL_CIPHER_STRING" >> /etc/ssl/openssl.cnf && \
86+
echo "Groups = $OPENSSL_GROUPS" >> /etc/ssl/openssl.cnf && \
87+
echo "=== FIPS Configuration Applied ===" && \
88+
tail -15 /etc/ssl/openssl.cnf; \
89+
else \
90+
echo "=== FIPS MODE DISABLED ==="; \
91+
fi
92+
93+
# If nghttp2 build fail just ignore it
94+
ENV NGHTTP2_VERSION=1.58.0
95+
RUN cd /tmp \
96+
&& curl -L "https://github.com/nghttp2/nghttp2/releases/download/v${NGHTTP2_VERSION}/nghttp2-${NGHTTP2_VERSION}.tar.gz" -o "nghttp2-${NGHTTP2_VERSION}.tar.gz" \
97+
&& tar -zxvf "nghttp2-${NGHTTP2_VERSION}.tar.gz" \
98+
&& cd /tmp/nghttp2-$NGHTTP2_VERSION \
99+
&& ./configure \
100+
&& make \
101+
&& make install || true
102+
103+
# Install docker client
104+
ENV DOCKER_VERSION=17.09.0-ce
105+
RUN cd /tmp \
106+
&& curl "https://download.docker.com/linux/static/stable/${CPUNAME}/docker-${DOCKER_VERSION}.tgz" -o docker.tgz \
107+
&& tar xvzf docker.tgz \
108+
&& cp docker/docker /usr/bin \
109+
&& rm -rf docker.tgz docker
110+
111+
# Install perfcollect
112+
ADD https://raw.githubusercontent.com/microsoft/perfview/main/src/perfcollect/perfcollect /usr/bin/perfcollect
113+
RUN chmod +x /usr/bin/perfcollect
114+
RUN /usr/bin/perfcollect install
115+
116+
COPY --from=build-env /app /app
117+
118+
ENTRYPOINT [ "/app/crank-agent" ]

‎src/BenchmarksApps/TLS/crank/agent/Dockerfile.AzureLinux3‎

Lines changed: 107 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,107 @@
1+
FROM mcr.microsoft.com/dotnet/sdk:8.0-azurelinux3.0 AS build-env
2+
3+
COPY . .
4+
5+
ENV DOTNET_SKIP_FIRST_TIME_EXPERIENCE=1
6+
ENV DOTNET_CLI_TELEMETRY_OPTOUT=1
7+
8+
# Build self contained
9+
RUN dotnet publish -c Release src/Microsoft.Crank.Agent --output /app --framework net8.0
10+
11+
# Build runtime image
12+
# FROM mcr.microsoft.com/dotnet/aspnet:8.0
13+
# Use SDK image as it is required for the dotnet tools
14+
FROM mcr.microsoft.com/dotnet/sdk:8.0-azurelinux3.0
15+
16+
ARG CPUNAME=x86_64
17+
ARG ENABLE_FIPS_MODE=false
18+
ARG OPENSSL_CIPHER_STRING=TLS_AES_256_GCM_SHA384:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256
19+
ARG OPENSSL_GROUPS=P-384:P-256:P-521
20+
21+
# Install dotnet-symbols
22+
RUN dotnet tool install -g dotnet-symbol
23+
ENV PATH="${PATH}:/root/.dotnet/tools"
24+
25+
# Install dependencies
26+
RUN tdnf update -y \
27+
&& tdnf install -y \
28+
git \
29+
procps-ng \
30+
curl \
31+
wget \
32+
libcgroup \
33+
libcgroup-tools \
34+
# dotnet performance repo microbenchmark dependencies
35+
libgdiplus \
36+
# libmsquic requirements
37+
gnupg2 \
38+
# NativeAOT requirements
39+
clang \
40+
zlib-devel \
41+
krb5-devel \
42+
# .NET 9.0 requirement
43+
glibc
44+
45+
# Install HTTP/3 support
46+
RUN tdnf install -y libmsquic
47+
48+
# Build and install h2load. Required as there isn't a way to distribute h2load as a single file to download
49+
RUN tdnf install -y \
50+
gcc-c++ make binutils autoconf automake libtool pkg-config \
51+
zlib-devel cunit-devel libxml2-devel libev-devel libevent-devel jansson-devel \
52+
c-ares-devel jemalloc-devel systemd-devel \
53+
python3-devel python3-setuptools
54+
55+
# ENV OPENSSL_VERSION=3.3.3 Version pinning does not work the same way in AL3 as it does in Debian/Ubuntu. Cannot use * in version, so we will use the latest version available in the repository.
56+
RUN tdnf install -y \
57+
openssl openssl-devel \
58+
&& tdnf clean all
59+
60+
# Configure OpenSSL for FIPS-compliant cipher suites if $ENABLE_FIPS_MODE
61+
RUN if [ "$ENABLE_FIPS_MODE" = "true" ]; then \
62+
echo "=== FIPS MODE ENABLED - Configuring OpenSSL ===" && \
63+
cat /etc/pki/tls/openssl.cnf && \
64+
echo "" >> /etc/pki/tls/openssl.cnf && \
65+
echo "openssl_conf = openssl_init" >> /etc/pki/tls/openssl.cnf && \
66+
echo "[openssl_init]" >> /etc/pki/tls/openssl.cnf && \
67+
echo "ssl_conf = ssl_sect" >> /etc/pki/tls/openssl.cnf && \
68+
echo "[ssl_sect]" >> /etc/pki/tls/openssl.cnf && \
69+
echo "system_default = system_default_sect" >> /etc/pki/tls/openssl.cnf && \
70+
echo "[system_default_sect]" >> /etc/pki/tls/openssl.cnf && \
71+
echo "CipherString = $OPENSSL_CIPHER_STRING" >> /etc/pki/tls/openssl.cnf && \
72+
echo "Groups = $OPENSSL_GROUPS" >> /etc/pki/tls/openssl.cnf && \
73+
echo "=== FIPS Configuration Applied ===" && \
74+
tail -15 /etc/pki/tls/openssl.cnf; \
75+
else \
76+
echo "=== FIPS MODE DISABLED ==="; \
77+
fi
78+
79+
# If nghttp2 build fail just ignore it
80+
ENV NGHTTP2_VERSION=1.58.0
81+
RUN tdnf install -y \
82+
glibc-devel gawk kernel-headers
83+
84+
RUN cd /tmp \
85+
&& curl -L "https://github.com/nghttp2/nghttp2/releases/download/v${NGHTTP2_VERSION}/nghttp2-${NGHTTP2_VERSION}.tar.gz" -o "nghttp2-${NGHTTP2_VERSION}.tar.gz" \
86+
&& tar -zxvf "nghttp2-${NGHTTP2_VERSION}.tar.gz" \
87+
&& cd /tmp/nghttp2-$NGHTTP2_VERSION \
88+
&& ./configure \
89+
&& make \
90+
&& make install || true
91+
92+
# Install docker client
93+
ENV DOCKER_VERSION=17.09.0-ce
94+
RUN cd /tmp \
95+
&& curl "https://download.docker.com/linux/static/stable/${CPUNAME}/docker-${DOCKER_VERSION}.tgz" -o docker.tgz \
96+
&& tar xvzf docker.tgz \
97+
&& cp docker/docker /usr/bin \
98+
&& rm -rf docker.tgz docker
99+
100+
# Install perfcollect
101+
ADD https://raw.githubusercontent.com/microsoft/perfview/main/src/perfcollect/perfcollect /usr/bin/perfcollect
102+
RUN chmod +x /usr/bin/perfcollect
103+
RUN /usr/bin/perfcollect install
104+
105+
COPY --from=build-env /app /app
106+
107+
ENTRYPOINT [ "/app/crank-agent" ]

‎src/BenchmarksApps/TLS/crank/agent/README.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Crank Agent image for TLS tests
2+
3+
...
4+
5+
### Crontab configuration
6+
7+
Machines are configured to perform a git pull, get latest changes, rebuild crank and restart it once in a while. On linux machines it is happening via `cron`.
8+
In order to be able to use Dockerfile from this folder (`/crank/agent`), one should change the crontab.
9+
10+
To lookup crontab configured:
11+
```
12+
sudo crontab -l
13+
```
14+
15+
You can use crontab like this:
16+
```
17+
0 0 * * * cd /home/dotnetperfuser/src/crank/docker/agent; ./stop.sh; docker rm -f $(docker ps -a -q --filter "label=benchmarks"); docker system prune --all --force --volumes; git checkout -f main; git reset --hard; git pull; cd /home/dotnetperfuser/src/Benchmarks; git checkout -f main; git reset --hard; git pull; cp -rf /home/dotnetperfuser/src/Benchmarks/src/BenchmarksApps/TLS/crank/agent/* /home/dotnetperfuser/src/crank/docker/agent/; cd /home/dotnetperfuser/src/crank/docker/agent; ./build.sh <arguments>; ./run.sh <arguments>
18+
```
19+
20+
Cron tab does the following:
21+
1) fetches the latest dotnet/crank
22+
2) fetches latest aspnetcore/Benchmarks
23+
3) copies dockerfile from aspnetcore/Benchmarks into dotnet/crank
24+
4) builds and runs the crank agent using custom Dockerfile
25+

0 commit comments

Comments
 (0)