Converted from poolside/Laguna-XS-2.1 to MLX format, quantized to 4 bits (group size 64, 4.503 bpw effective).
laguna architecture yet — there's an open PR:
mlx-lm#1223.</think> tag at the start of responses, which
isn't that common. It won't affect anything tho.Measured with oMLX's benchmark harness on a Macbook Pro M5 Max 128GB 40 GPU (single request, 128 generated tokens):
| prompt | gen tok/s | prefill tok/s | TTFT ms | peak GB |
|---|---|---|---|---|
| 1k | 126.0 | 2797 | 367 | 18.2 |
| 4k | 121.2 | 4052 | 1011 | 18.8 |
| 8k | 116.6 | 3785 | 2165 | 18.9 |
| 16k | 109.1 | 3122 | 5248 | 19.2 |
| 32k | 91.3 | 2462 | 13312 | 19.8 |
| Variant | bpw | Disk | gen tok/s (1k → 32k) |
|---|---|---|---|
| bf16 | 16 | 62 GB | 70.6 → 58.7 |
| 8bit | 8.500 | 33 GB | 95.4 → 76.7 |
| 6bit | 6.501 | 25 GB | 102.9 → 80.9 |
| 5bit | 5.502 | 21 GB | 115.9 → 87.7 |
| 4bit (this repo) | 4.503 | 18 GB | 126.0 → 91.3 |
| 3bit | 3.503 | 14 GB | 137.2 → 98.8 |
uvx --from mlx-vlm mlx_vlm.generate --model mlx-community/Laguna-XS-2.1-4bit --prompt "..." --max-tokens 300
OpenMDW-1.1, inherited from the base model.
Content type
Unrecognized
Digest
sha256:2c0e6b81f…
Size
17.5 GB
Last updated
about 1 month ago
docker pull ai/laguna-xs-2.1Pulls:
68
Last week