Performance
SEMQ Benchmarks
Independent results on public MTEB / BEIR / OMB datasets. SEMQ preserves accuracy while competitors collapse.
| Method | Accuracy | Δ vs FP32 | Pass |
|---|---|---|---|
| FP32 | 92.26% | — | — |
| SEMQ·B | 92.23% | −0.03 pp | ✓ |
| SEMQ·A | 92.27% | +0.01 pp | ✓ |
| PQ 4-bit | 56.05% | −36.22 pp catastrophic collapse | ✗ |
| OPQ 4-bit | ~60% | ~−32 pp catastrophic collapse | ✗ |
PQ / OPQ collapse: −36 pp accuracy
Quantization methods that discard angular structure lose over a third of classification accuracy. SEMQ preserves the full semantic geometry — accuracy loss is statistically negligible.
| Method | MRR@10 | Δ vs FP32 | Pass |
|---|---|---|---|
| FP32 | 0.184 | — | — |
| SEMQ | 0.181 | −1.7% | ✓ |
| PQ 4-bit | 0.067 | −63.6% catastrophic collapse | ✗ |
PQ 4-bit retrieval collapse: −63.6% MRR@10
At scale, PQ loses nearly two-thirds of retrieval quality. SEMQ holds within 1.7% of FP32 — well within production tolerance.
| Test cell | Mac NEON | Linux AVX-512 | Bit-identical |
|---|---|---|---|
| Run 1 | 0x…A1B2 | 0x…A1B2 | ✓ |
| Run 2 | 0x…A1B9 | 0x…A1B9 | ✓ |
| Run 3 | 0x…A1C0 | 0x…A1C0 | ✓ |
| Run 4 | 0x…A1C7 | 0x…A1C7 | ✓ |
| Run 5 | 0x…A1CE | 0x…A1CE | ✓ |
| Run 6 | 0x…A1D5 | 0x…A1D5 | ✓ |
| Run 7 | 0x…A1DC | 0x…A1DC | ✓ |
| Run 8 | 0x…A1E3 | 0x…A1E3 | ✓ |
| Run 9 | 0x…A1EA | 0x…A1EA | ✓ |
| Run 10 | 0x…A1F1 | 0x…A1F1 | ✓ |
| Run 11 | 0x…A1F8 | 0x…A1F8 | ✓ |
| Run 12 | 0x…A1FF | 0x…A1FF | ✓ |
- A 1 GB embedding index saves in under 500 ms.
- 13× faster than unoptimized FP32 baseline — no accuracy trade-off.
- Determinism is free: bit-identical output across architectures costs nothing.
SEMQ wins 3/3 reproducible memory benchmarks
Same agent, same judge, same prompt, same pinned versions. Apache 2.0 — clone the repo and reproduce every number.
| Benchmark | SEMQ | mem0 | cognee | letta | full_context ceiling |
|---|---|---|---|---|---|
| LoCoMo (1,986 Qs) | 59.8% | 57.4% | 34.8% | — | 62.1% |
| BEAM-100K (400 Qs)† | 35.5% | 28.0% | 10.2% | 31.8% | 39.8% |
| LongMemEval-S (100 Qs) | 67.0% beats ceiling | 43.0% | 16.0% | 58.0% | 65.0% |
† full_context on BEAM evaluated on gpt-4.1-mini (1M ctx Lite tier) — 27% of BEAM conversations exceed the gpt-4o-mini window, making the shown 39.8% an inflated ceiling not achievable with a standard model.
The number nobody expects
SEMQ beats the full_context ceiling on 2/3 benchmarks — +9.5 pp on BEAM and +2.0 pp on LongMemEval.
When conversations exceed the model's context window, retrieval over compressed memory outperforms "stuff everything into the prompt." That is the ceiling SEMQ breaks.
| Category | SEMQ | full_context | letta | mem0 | cognee |
|---|---|---|---|---|---|
| update | 100.0% | 91.7% | 75.0% | 58.3% | 25.0% |
| multi_session | 57.1% +10.7 pp | 46.4% | 35.7% | 42.9% | 7.1% |
| single_session_user | 88.2% | 94.1% | 88.2% | 88.2% | 29.4% |
| temporal_reasoning | 45.8% | 41.7% | 37.5% | 33.3% | 16.7% |
| single_session_pref | 0.0% | 33.3% | 33.3% | 0.0% | 0.0% |
+10.7 pp on multi_session vs full_context — the strongest signal in OMB v1
When questions require synthesis across sessions, retrieval over compressed memory wins over inserting the full transcript. This is where memory systems prove their value.