breuner/elbencho repository overview

⁠elbencho

A distributed benchmark for file, object & block storage with support for GPUs

For the full description, see elbencho's readme and changelog on github: https://github.com/breuner/elbencho⁠

⁠Usage Examples

The default dockerfile defines the path to the elbencho executable as an entrypoint. This means you don't need to specify the name or path of the elbencho binary and just directly provide the elbencho parameters to the docker run command.

⁠Hello World

To just confirm that fetching of the container works, try showing elbencho's built-in help:

docker run --rm -it breuner/elbencho --help

⁠Simple Benchmark

Use docker's -v HOST_PATH:CONTAINER_PATH option to make your test directory available inside the container. E.g. if your benchmark files should be in /data and you want to create a single 10GiB file:

docker run -v /data:/data --rm -it breuner/elbencho /data/mytestfile -s 10G -w

⁠Distributed Mode

If you want to use elbencho's distributed mode for coordinated throughput tests from multiple nodes, the easiest way is to use Docker's --net=host option to make the elbencho service inside the container available under the IP address of the host on which it is running. Also, you will want to prevent the service from going into background (because otherwise Docker will think it can stop the container instance) and instead detach the container to run in the background (elbencho --foreground & docker run -d).

Run this command on all nodes that should participate in your distributed storage benchmarks:

docker run -v /data:/data --net=host --rm -dit --init breuner/elbencho --service --foreground

Now you're ready to control the service instances, again by using the --net=host option for the master instance. Here we write a single shared 10GiB file from all service instances:

docker run --net=host --rm -it breuner/elbencho /data/mytestfile -s 10G -w --hosts HOST1,HOST2,...

When you're done with your distributed benchmarks and want to stop the service instances, use elbencho's --quit option to terminate them:

docker run --net=host --rm -it breuner/elbencho --quit --hosts HOST1,HOST2,...

⁠Docker Image Flavors & Tags

The default image (tagged latest) is based on Ubuntu 26.04. It refers to the latest stable release version of elbencho. The image tags master-* refer to builds based on the GitHub master branch.

Ubuntu and RHEL-clone based images contain the full .deb / .rpm package installation with the corresponding additional tools. To use e.g. one of the contained elbencho tools, simply specify the tool name as an alternative entrypoint:

docker run --rm -it --entrypoint elbencho-scan-path breuner/elbencho --help

The Alpine Linux based image (tagged master-alpine) is optimized for minimum size and only contains the elbencho main executable, no other tools.

⁠GPUs & GPUDirect Storage (GDS)

To test GPU storage access performance through Nvidia CUDA or GPUDirect Storage (GDS/cuFile), use the multi-arch Ubuntu container with CUDA installed (tag master-ubuntu-cuda-multiarch).

Here is an example to write and read 128 large file via GDS, using 256 threads and all available GPUs in the host:

nvidia-docker run --privileged -v /data:/data --rm -it breuner/elbencho:master-ubuntu-cuda-multiarch "/data/mylargefile[1..128]" -w -r -t 256 -s 12g -b 4m --direct --gpuids all --gds

If you don't have nvidia-docker, you can alternatively use docker run --gpus all ....

⁠S3 Support

S3 support is included in the images tagged latest, master-alpine, master-ubuntu-cuda-multiarch and in the images with stable version number tags.

⁠NVMe-oF Initiator Support

NVMe-oF initiator support through SPDK is included in the image tagged master-ubuntu2604.

⁠Local Image Builds

To build docker images from your local elbencho git clone (e.g. because you modified the sources or want to build a particular previous version), you can find the corresponding dockerfiles in the build_helpers/docker subdir with a .local extension. To build a Ubuntu image from your local sources, run this command:

docker build -t elbencho-local -f build_helpers/docker/Dockerfile.ubuntu2604.local .

⁠ARM64 & Multi-Platform

The image tags latest and master-ubuntu-cuda-multiarch support amd64 (aka x86_64) and arm64/v8 (aka aarch64) platforms like Nvidia Grace CPUs. (Other image tags might be available for only one of these platforms.)

⁠FIPS mode

The default image based on Ubuntu is not FIPS compliant, because the default Ubuntu repositories don't contain a FIPS compliant libcrypto. When trying to run an Ubuntu-based image with S3 support on a system with FIPS mode enabled, elbencho will error out during S3 library initialization with a message like "FIPS mode is not supported for the libcrypto".

In this case, consider using a different image or (if allowed by your policies) consider disabling FIPS mode inside the container by masking the fips_enabled file for the container runtime like this:

echo "0" > /tmp/fake_fips_mode

docker run -v /tmp/fake_fips_mode:/proc/sys/crypto/fips_enabled:ro ...