Framework for benchmarking vector search engines.
8.9K
A Rust-based benchmarking tool for vector databases. Measures upload throughput, search QPS, latency percentiles, and recall for various vector search engines.
| Engine | Client Library | Protocol |
|---|---|---|
| Redis (RediSearch) | redis 0.27 | Redis protocol |
| VectorSets | redis 0.27 | Redis protocol |
| Elasticsearch | elasticsearch 8.15 | HTTP/REST |
| OpenSearch | opensearch 2.3 | HTTP/REST |
| Qdrant | qdrant-client 1.17 | gRPC |
| PgVector | postgres 0.19 + pgvector 0.4 | PostgreSQL |
| Weaviate | tonic 0.12 / prost 0.13 (gRPC) + reqwest (REST) | gRPC (search) + HTTP/REST (schema) |
| Milvus | reqwest (REST API v2) | HTTP/REST |
| MongoDB (Atlas Search) | mongodb 3 (sync) | MongoDB protocol |
| Valkey (Valkey Search) | redis 0.27 | RESP protocol |
| Turbopuffer | turbopuffer-client 0.0.4 | HTTP/REST (cloud) |
# Show help
docker run --rm redis/vector-db-benchmark --help
# List available datasets
docker run --rm redis/vector-db-benchmark --describe datasets
# List available engine configurations
docker run --rm redis/vector-db-benchmark --describe engines
# Start Redis
docker run -d --name redis -p 6379:6379 redis:8.8.0
# Run benchmark against localhost Redis
docker run --rm --network host \
-v $(pwd)/datasets:/code/datasets \
-v $(pwd)/results:/code/results \
redis/vector-db-benchmark \
--host localhost --engines 'redis*' --datasets 'glove-25-angular'
| Flag | Description | Default |
|---|---|---|
--engines <PATTERN> | Engine patterns to run (supports wildcards) | * |
--datasets <PATTERN> | Dataset patterns to run (supports wildcards) | * |
--host <HOST> | Redis / engine host | localhost |
--skip-upload | Skip the upload phase | — |
--skip-search | Skip the search phase | — |
--skip-if-exists | Skip if results already exist | — |
--parallels <N,N,...> | Filter by parallel thread counts | — |
--ef-runtime <N,N,...> | Filter by ef runtime values | — |
--update-search-ratio <U:S> | Mixed benchmark: interleave U updates per S searches | — |
--describe <TYPE> | Describe datasets or engines | — |
-v, --verbose | Verbose output for --describe | — |
--exit-on-error | Stop on first error | — |
--timeout <SECS> | Timeout in seconds | 86400 |
Interleave vector updates with searches to measure write impact on search performance:
# 1 update per 10 searches against Redis
docker run --rm --network host \
-v $(pwd)/datasets:/code/datasets \
-v $(pwd)/results:/code/results \
redis/vector-db-benchmark \
--host localhost --engines redis-docker-test --datasets random-100 \
--update-search-ratio 1:10
The ratio format is U:S (e.g., 1:10 = 1 update per 10 searches per cycle). Supported engines: Redis, VectorSets, Valkey. Results include separate search and update metrics (RPS, latencies). Omitting the flag runs standard search-only benchmarks.
| Container Path | Purpose |
|---|---|
/code/datasets | Dataset files (HDF5, NPY, JSONL) |
/code/results | Benchmark result JSON files |
The image includes the random-100 dataset (228KB) for quick smoke tests. For larger datasets, mount your local datasets/ directory.
services:
redis:
image: redis:8.8.0
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 1s
timeout: 3s
retries: 10
benchmark:
image: redis/vector-db-benchmark:latest
depends_on:
redis:
condition: service_healthy
volumes:
- ./datasets:/code/datasets
- ./results:/code/results
command:
- "--engines"
- "redis*"
- "--datasets"
- "glove-25-angular"
- "--host"
- "redis"
Content type
Image
Digest
sha256:54a44f35c…
Size
44.2 MB
Last updated
2 months ago
docker pull redis/vector-db-benchmarkPulls:
21
Last week