Skip to content

Commit a50ef82

Browse files
jlb6740jlb6740
committed
Initial commit for WasmScore"
0 parents  commit a50ef82

7 files changed

Lines changed: 553 additions & 0 deletions

File tree

‎Dockerfile‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
FROM ubuntu:20.04
2+
ARG ARCH
3+
ENV LD_LIBRARY_PATH=/usr/local/lib
4+
ENV PATH=/usr/local/bin:$PATH
5+
CMD ["/bin/bash"]
6+
ENV DEBIAN_FRONTEND="noninteractive" TZ="America"
7+
ARG RUST_VERSION="nightly-2023-04-01"
8+
ARG WASMTIME_REPO="https://github.com/bytecodealliance/wasmtime/"
9+
ARG WASMTIME_COMMIT="b306368" #v0.38.0
10+
ARG SIGHTGLASS_REPO="https://github.com/jlb6740/sightglass.git"
11+
ARG SIGHTGLASS_BRANCH="add-native-engine"
12+
ARG SIGHTGLASS_COMMIT="35cccc2"
13+
14+
# Get some prerequisites
15+
RUN apt-get update \
16+
&& apt-get install -y --no-install-recommends \
17+
apt-utils build-essential gpg-agent \
18+
curl ca-certificates wget software-properties-common \
19+
psmisc lsof git nano zlib1g-dev libedit-dev time yasm \
20+
libssl-dev pkg-config
21+
22+
# Bionic does not carry a recent enough cmake needed for wamr but upgrading
23+
# to Focal causes other build issues so the straight forward solution is to just
24+
# install a version of cmake that is recent enough from a separate repository
25+
RUN wget -qO - https://apt.kitware.com/keys/kitware-archive-latest.asc | apt-key add -
26+
RUN apt-add-repository 'deb https://apt.kitware.com/ubuntu/ focal main'
27+
RUN apt-get update && apt-get install --yes cmake
28+
29+
# Install wabt
30+
WORKDIR /opt
31+
RUN git clone --recurse-submodules https://github.com/WebAssembly/wabt.git \
32+
&& cd wabt \
33+
&& mkdir build \
34+
&& cd build \
35+
&& cmake .. \
36+
&& cmake --build . \
37+
&& make
38+
ENV PATH=$PATH:/opt/wabt/bin/
39+
40+
# Install rust
41+
RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs -sSf | sh -s -- --default-toolchain ${RUST_VERSION} -y
42+
ENV PATH=/root/.cargo/bin:${PATH}
43+
44+
# Install clang
45+
RUN apt-get install -y --no-install-recommends clang
46+
47+
# Install python
48+
RUN apt-get install -y --no-install-recommends python3.8 libpython3.8 python3-distutils python3-pip
49+
RUN python3 -m pip install termgraph \
50+
&& python3 -m pip install pandas \
51+
&& python3 -m pip install termcolor \
52+
&& python3 -m pip install pyyaml
53+
54+
# Install docker
55+
RUN apt-get update && apt-get -y install ca-certificates curl gnupg lsb-release
56+
RUN curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
57+
RUN echo \
58+
"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
59+
$(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null
60+
RUN apt-get update && apt-get -y install docker-ce docker-ce-cli containerd.io
61+
62+
# Install sightglass
63+
WORKDIR /
64+
RUN git clone --recurse-submodules ${SIGHTGLASS_REPO} sightglass
65+
WORKDIR /sightglass
66+
RUN git checkout ${SIGHTGLASS_COMMIT} -b ${SIGHTGLASS_COMMIT}
67+
COPY add_time_metric.diff /sightglass/add_time_metric.diff
68+
RUN git apply add_time_metric.diff
69+
RUN cargo build --release
70+
RUN mkdir results
71+
72+
# Build wasmtime engine for sightglass
73+
WORKDIR /
74+
RUN git clone --single-branch --recurse-submodule ${WASMTIME_REPO} wasmtime
75+
WORKDIR /wasmtime
76+
RUN git checkout ${WASMTIME_COMMIT} -b ${WASMTIME_COMMIT}
77+
RUN cargo build -p wasmtime-bench-api --release
78+
RUN cp target/release/libwasmtime_bench_api.so /sightglass/engines/wasmtime/libengine.so
79+
80+
# Build native engine for sightglass
81+
WORKDIR /sightglass/engines/native/libengine
82+
RUN cargo build --release
83+
RUN cp target/release/libnative_bench_api.so ../libengine.so
84+
85+
# Copy driver/helpers into the image
86+
WORKDIR /
87+
COPY wasmscore.py /sightglass/wasmscore.py
88+
COPY wasmscore.sh /

‎README.md‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
# WasmScore
2+
3+
This benchmark is designed to provide a simple, convenient and portable view of the performance of WebAssembly outside the browser on the underlying platform it is run on. It uses a containerized suite of real codes and micros leveraged directly from [Sightglass](https://github.com/bytecodealliance/sightglass) to benchmark the platform and provide a “WasmScore” that is based on a formula that aggregates execution time. In addition, the driver for the benchmark serves as an easy-to-use tool for executing any user determined assortment of Sightglass benchmarks supported by the driver.
4+
5+
One of the most important and challenging aspect of benchmarking is deciding how to interpret the results; should you consider the results to be good or bad? To decide, you really need context on what it is you are trying to achieve, and this often starts with a baseline used to serve as a point of comparison. That baseline could be the execution of that same original source but before some transformation was applied when lowered to WebAssembly, or that baseline could be a modified configuration of the runtime that executes the WebAssembly. In the case of WasmScore, a novel aspect is that for every Wasm real code and micro that is run, WasmScore also executes the native versions of those code compiled from the same high-level source used to generate the Wasm. In this way WasmScore provides a comparison point for the Wasm performance which will ideally be the theoretical upper for the performance of WebAssembly. This feature allows a user to quickly gauge the performance impact of using Wasm instead of using a native compile of the same code when run on that particular platform. It allows developers to see opportunity to improve compilers, or to improve Wasm runtimes, or improve the Wasm spec, or to suggest other solutions (such as Wasi) to address gaps.
6+
7+
Another important feature of WasmScore is simplicity and convenience. Specifically, the user is not expected to have to build the benchmark where they might have to deal with installing or updating dependencies. The user is also not expected contend interpreting the need for turning on or off a myriad of flags and features; to get a platforms WasmScore the user simply runs wasmscore.sh inside the container. Still, while it is meant for the user to simply pull a containerized image and then run the benchmark on the desired platform without worrying, WasmScore can of course be built and then run either within or outside (TODO) the containerized environment. In either case is intended for the compile of all codes to properly utilizes underlying hardware features. To that end, the ideal use case and indeed the target use case for WasmScore is for a quick, simple and consistent cross platform view of Wasm performance. The benchmark especially wants to target usecases and applications that are emerging for Wasm in standalone client and cloud environments. WasmScore is intended to be run on X86-64 and AArch64 Linux platforms.
8+
9+
10+
## Usage
11+
12+
### Use Prebuilt Images
13+
14+
Download and run the latest prebuilt benchmark image:
15+
16+
**X86-64:**
17+
```
18+
docker pull ghcr.io/jlb6740/wasmscore/wasmscore_x86_64:latest
19+
```
20+
```
21+
docker run -it ghcr.io/jlb6740/wasmscore/wasmscore_x86_64:latest /bin/bash /wasmscore.sh
22+
```
23+
**AArch64:**
24+
```
25+
docker pull ghcr.io/jlb6740/wasmscore/wasmscore_aarch64:latest
26+
```
27+
```
28+
docker run -it ghcr.io/jlb6740/wasmscore/wasmscore_aarch64:latest /bin/bash /wasmscore.sh
29+
```
30+
31+
### Build and Run Yourself
32+
33+
To build:
34+
```
35+
./build.sh
36+
```
37+
To run from this local build:
38+
```
39+
docker run -ti wasmscore /bin/bash wasmscore.sh --help
40+
```
41+
42+
To build containerless:
43+
> Not yet supported
44+
45+
### Other Useful Commands
46+
47+
For a detached setup that allows for copying files to the image or entering the container (being mindful of the container name), use the following commands:
48+
```
49+
docker run -ti -d wasmscore /bin/bash
50+
```
51+
```
52+
wasmscore_container_id=$(docker ps | grep -m 1 wasmscore | awk '{ print $1 }')
53+
```
54+
```
55+
docker cp <file> ${wasmscore_container_id}:
56+
```
57+
or
58+
```
59+
docker exec -ti ${wasmscore_container_id} /bin/bash
60+
```
61+
## Example Screenshots
62+
63+
64+

‎add_time_metric.diff‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
diff --git a/crates/recorder/src/measure/cycles.rs b/crates/recorder/src/measure/cycles.rs
2+
index 7b8c7d6..64c37d3 100644
3+
--- a/crates/recorder/src/measure/cycles.rs
4+
+++ b/crates/recorder/src/measure/cycles.rs
5+
@@ -12,7 +12,7 @@ lazy_static! {
6+
// CPU frequency, which adds ~5 seconds on start up time per
7+
// benchmarking process, and we only care about cycles anyways (less
8+
// affected by CPU frequency monitors).
9+
- let config = Config::default().wall_time(false);
10+
+ let config = Config::default();
11+
Precision::new(config).unwrap()
12+
};
13+
}
14+
@@ -39,5 +39,6 @@ impl Measure for CycleMeasure {
15+
let elapsed = end - self.0.take().expect("must call start before end");
16+
17+
measurements.add(phase, "cycles".into(), elapsed.ticks());
18+
+ measurements.add(phase, "nanoseconds".into(), elapsed.as_ns(precision));
19+
}
20+
}

‎build.sh‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
#!/bin/bash
2+
. "$(dirname ${BASH_SOURCE:-$0})/config.inc"
3+
IMAGE_NAME="wasmscore"
4+
ARCH=$(uname -m)
5+
echo "Building ${IMAGE_NAME} for $ARCH based on wasmtime@${WASMTIME_BUILD}"
6+
7+
# Create Docker Image
8+
docker build -t ${IMAGE_NAME} --build-arg ARCH=$(uname -m) .
9+
10+
docker tag ${IMAGE_NAME} ${IMAGE_NAME}_${ARCH}:latest
11+
docker tag ${IMAGE_NAME} ${IMAGE_NAME}_${ARCH}:${IMAGE_VER}
12+
13+
echo ""
14+
echo "To run from this local build use command:"
15+
echo "> docker run -ti ${IMAGE_NAME} /bin/bash wasmscore.sh --help"
16+
echo ""
17+
echo "To stop and rm all ${IMAGE_NAME} containers:"
18+
echo "> docker rm \$(docker stop \$(docker ps -a -q --filter ancestor=${IMAGE_NAME}:latest --format="{{.ID}}"))"
19+
echo ""
20+
echo "For a detached setup that allows for copying files to the image or"
21+
echo "entering the container, use the following commands:"
22+
echo "> docker run -ti -d ${IMAGE_NAME} /bin/bash"
23+
echo "> wasmscore_container_id=\$(docker ps | grep -m 1 ${IMAGE_NAME} | awk '{ print \$1 }')"
24+
echo "> docker cp <file> \${wasmscore_container_id}:"
25+
echo "or"
26+
echo "> docker exec -ti \${wasmscore_container_id} /bin/bash"
27+
echo ""

‎config.inc‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# Global variables used by scripts for docker building and launching
2+
IMAGE_VER="v0.0.1"
3+
WASMTIME_BUILD="b306368" # v0.38.0

0 commit comments

Comments
 (0)