Hello LinkedIn Fam, welcome to Optimisations Wednesday.
Today's headline is a 25% increase in throughput when compared to Ollama.
How did I do this?
- Smarter & deeper MTP (Qwen's MTP sidecar looks really impressive and I've found acceptance to be quite high)
- Switched to MTPLX (wasn't significatly faster it's just built specifically for qwen on MLX so provided an easier entry point for my optimisations)
- Patched MTPLX with:
- Smarter MTP draft depth (going deeper when confidence is higher)
- Reduced required synchronisation between the CPU & GPU
The smarter MTP draft depth enabled 4.3 tokens commited per verification run vs 3.2 with standard MTPLX.