I trained an MNIST classifier and got 98% on the test set, which tells you how it does on other people's handwriting. I wanted to know how it did on mine. So I built a canvas for it.
mnist-interactive is a Python tool that opens a 28×28 grid, the same shape as an MNIST image, and runs your model on it every time you change a pixel. Left click draws, right click erases, and the prediction and confidence sit above the canvas and update as you go.
import mnist_interactive.numberCreatorWindow as mi
import tensorflow as tf
import tkinter as tk
model = tf.keras.models.load_model("sampleModel.keras")
root = tk.Tk()
mi.NumberCreatorWindow(root, model=model, blur=0.3)
root.mainloop()
That's the whole thing. It's on TestPyPI:
pip install -i https://test.pypi.org/simple/ mnist-interactive
The blur#
The first version drew hard-edged pixels and the model was terrible at reading them. MNIST digits aren't hard-edged. They were scanned pen strokes, downsampled, so every stroke has a soft grey halo. A model trained on that has never seen a crisp 1-or-0 pixel and doesn't know what to do with one.
So the canvas blurs. Each stroke bleeds into the neighbouring cells by an amount you set (blur=0.3 in the example). Once the input looked like the training data the predictions started making sense. Most of what I learned from this project is in that one parameter: the model was fine, my input was out of distribution.
The activations window#
The part I find most useful is a second window that shows the neuron activation values for the current drawing. You can watch which units light up as a 3 becomes an 8, or draw something ambiguous and see the model hesitate. It turns "the model is 60% sure" into something you can actually look at.
Other models#
Out of the box it expects a Keras model that takes a 28×28 array and returns ten probabilities. If yours doesn't, you pass a conversion function for the input and an output function for the result, so it works with PyTorch models or anything else with a different shape without changing the tool.
What it's for#
Teaching, mostly, and the first ten minutes after training a model when you want to poke it rather than read a confusion matrix. It's small and it does one thing. Source is on GitHub.