poorer performance than a VM created by libvirt

[giuseppe]

I see significant differences when I run make -j $(nproc) on the crun source code inside a krun VM compared to what I get on a VM created through libvirt.

On the libvirt VM I get:

        Command being timed: "make -j 16"
        User time (seconds): 23.80
        System time (seconds): 2.71
        Percent of CPU this job got: 438%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:06.05
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 101480
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 0
        Minor (reclaiming a frame) page faults: 1314536
        Voluntary context switches: 10232
        Involuntary context switches: 10566
        Swaps: 0
        File system inputs: 0
        File system outputs: 124864
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

while with krun (used through podman --runtime krun):

        Command being timed: "make -j 16"
        User time (seconds): 33.93
        System time (seconds): 16.65
        Percent of CPU this job got: 450%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:11.22
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 98064
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 71133
        Minor (reclaiming a frame) page faults: 1699802
        Voluntary context switches: 618947
        Involuntary context switches: 36435
        Swaps: 0
        File system inputs: 0
        File system outputs: 120488
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

Moving the source code on a tmpfs does not affect the Elapsed time.

[mtjhrc]

Could you try running with stdout/stderr redirected to /dev/null? (To see the impact of virtio-console)

[giuseppe]

\time -v sh -c 'make -j 16 > /dev/null 2>&1l'
        Command being timed: "sh -c make -j 16 > /dev/null 2>&1"
        User time (seconds): 31.25
        System time (seconds): 15.72
        Percent of CPU this job got: 471%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:09.97
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 97700
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 73115
        Minor (reclaiming a frame) page faults: 1713754
        Voluntary context switches: 612334
        Involuntary context switches: 45885
        Swaps: 0
        File system inputs: 0
        File system outputs: 120488
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

[hegerdes]

I'm just getting started with krun, so maybe I'm completely off here, but what kernel do you use for krun?
Is it libkrunfw?

It has this in its known limitations

To save memory, the embedded kernel is configured with CONFIG_NR_CPUS=8, which limits the maximum number of supported CPUs to 8. If this kernel runs in a VM with more CPUs, only the first 8 will be initialized and used.

And I see that you are using 16 cores/threads via -j 16. I would suspect that libvirt has no such limitation. Is there time comparable if both use -j 8?

[giuseppe]

I'm just getting started with krun, so maybe I'm completely off here, but what kernel do you use for krun? Is it libkrunfw?

yes

It has this in its known limitations

To save memory, the embedded kernel is configured with CONFIG_NR_CPUS=8, which limits the maximum number of supported CPUs to 8. If this kernel runs in a VM with more CPUs, only the first 8 will be initialized and used.

And I see that you are using 16 cores/threads via -j 16. I would suspect that libvirt has no such limitation. Is there time comparable if both use -j 8?

libvirt KVM

# \time -v sh -c 'make -j 8 > /dev/null 2>&1'
	Command being timed: "sh -c make -j 8 > /dev/null 2>&1"
	User time (seconds): 27.04
	System time (seconds): 3.45
	Percent of CPU this job got: 499%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:06.10
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 102008
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 2
	Minor (reclaiming a frame) page faults: 1309153
	Voluntary context switches: 10893
	Involuntary context switches: 4795
	Swaps: 0
	File system inputs: 0
	File system outputs: 124192
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

krun

# \time -v sh -c 'make -j 8 > /dev/null 2>&1'
	Command being timed: "sh -c make -j 8 > /dev/null 2>&1"
	User time (seconds): 29.03
	System time (seconds): 11.97
	Percent of CPU this job got: 371%
	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:11.03
	Average shared text size (kbytes): 0
	Average unshared data size (kbytes): 0
	Average stack size (kbytes): 0
	Average total size (kbytes): 0
	Maximum resident set size (kbytes): 96860
	Average resident set size (kbytes): 0
	Major (requiring I/O) page faults: 73661
	Minor (reclaiming a frame) page faults: 1567333
	Voluntary context switches: 489385
	Involuntary context switches: 15966
	Swaps: 0
	File system inputs: 0
	File system outputs: 178767
	Socket messages sent: 0
	Socket messages received: 0
	Signals delivered: 0
	Page size (bytes): 4096
	Exit status: 0

[slp]

I've set up two instances, one with QEMU/KVM and the other with libkrun/KVM, but both with virtio-blk as root and the same kernel. With them, I'm getting roughly the same numbers on both:

• QEMU/KVM

        Command being timed: "make -j16"
        User time (seconds): 24.01
        System time (seconds): 5.18
        Percent of CPU this job got: 475%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:06.13
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 95052
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 3
        Minor (reclaiming a frame) page faults: 1472442
        Voluntary context switches: 12983
        Involuntary context switches: 2158
        Swaps: 0
        File system inputs: 0
        File system outputs: 122280
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

• libkrun/KVM

        Command being timed: "make -j16"
        User time (seconds): 23.48
        System time (seconds): 6.29
        Percent of CPU this job got: 490%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:06.06
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 94380
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 276
        Minor (reclaiming a frame) page faults: 1470644
        Voluntary context switches: 13889
        Involuntary context switches: 2211
        Swaps: 0
        File system inputs: 79504
        File system outputs: 122280
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0

This makes me think the differences you're seeing are probably due to some differences in the VM configuration, with the main suspect being using virtio-fs as root (which is probably still getting used even if the build happens in /tmp).