AI models don't know what they want to say.
An LLM outputs probabilities for the next token, which can be a word or part of one. It might give "last" 68% and "old" 32%.
When we sample, we roll a weighted die and pick one, and the model carries on from whatever we picked.
So the same prompt can give you two completely different answers, and you never see the one you didn't get.
What if instead, when the model was unsure, you forked and created two paths instead of choosing one?
You can't fork on every hesitation as some are just the model expressing uncertainty about grammar or punctuation. So we looked at entropy in the model's output probabilities, then compared the continuations to decide if a decision was worth a fork.
Here's what that looks like for the prompt "Give me a one sentence story."
The code is on GitHub.